Todd
258
docs/archive/SNAPCHAT_IMPLEMENTATION_SUMMARY.md
Normal file
258
docs/archive/SNAPCHAT_IMPLEMENTATION_SUMMARY.md
Normal file
@@ -0,0 +1,258 @@
|
||||
# Snapchat Downloader Implementation Summary
|
||||
|
||||
## Overview
|
||||
Successfully implemented a complete Snapchat downloader module for the media-downloader system, based on the ImgInn module architecture. The module downloads Snapchat stories via the StoryClon e proxy (https://s.storyclone.com/u/<user>/).
|
||||
|
||||
## Files Created
|
||||
|
||||
### 1. Core Module
|
||||
**File**: `/opt/media-downloader/modules/snapchat_module.py`
|
||||
- Main SnapchatDownloader class
|
||||
- Browser automation with Playwright
|
||||
- FastDL-compatible file naming
|
||||
- Cookie management
|
||||
- Cloudflare challenge handling
|
||||
- Database integration
|
||||
- Timestamp updating (file system + EXIF)
|
||||
- Story extraction and downloading
|
||||
|
||||
### 2. Subprocess Wrapper
|
||||
**File**: `/opt/media-downloader/snapchat_subprocess_wrapper.py`
|
||||
- Isolates Snapchat operations in separate process
|
||||
- Avoids asyncio event loop conflicts
|
||||
- JSON-based configuration input/output
|
||||
- Stderr logging for clean stdout
|
||||
|
||||
### 3. Database Adapter
|
||||
**File**: `/opt/media-downloader/modules/unified_database.py` (modified)
|
||||
- Added SnapchatDatabaseAdapter class
|
||||
- Tracks downloads by URL and metadata
|
||||
- Platform: 'snapchat'
|
||||
- Content type: 'story'
|
||||
|
||||
### 4. Main Integration
|
||||
**File**: `/opt/media-downloader/media-downloader.py` (modified)
|
||||
- Imported SnapchatDownloader module
|
||||
- Added initialization in _init_modules()
|
||||
- Added interval configuration (check_interval_hours)
|
||||
- Created _download_snapchat_content() method
|
||||
- Created download_snapchat() method
|
||||
- Integrated into run() method (download all platforms)
|
||||
- Added command-line argument support: --platform snapchat
|
||||
- Added scheduler filtering support
|
||||
|
||||
### 5. Configuration Example
|
||||
**File**: `/opt/media-downloader/config/snapchat_example.json`
|
||||
- Sample configuration structure
|
||||
- All available settings documented
|
||||
- Ready to copy into main settings.json
|
||||
|
||||
### 6. Documentation
|
||||
**File**: `/opt/media-downloader/SNAPCHAT_README.md`
|
||||
- Complete usage guide
|
||||
- Setup instructions
|
||||
- Configuration options explained
|
||||
- Troubleshooting section
|
||||
- Architecture overview
|
||||
|
||||
## Key Features Implemented
|
||||
|
||||
### ✅ Complete Feature Set
|
||||
1. **Browser Automation**: Playwright-based Chromium automation
|
||||
2. **Proxy Support**: Uses StoryClon e (s.storyclone.com) proxy
|
||||
3. **Story Downloads**: Extracts and downloads all available stories
|
||||
4. **FastDL Naming**: Compatible filename format (user_date_mediaid.ext)
|
||||
5. **Database Tracking**: Full integration with unified database
|
||||
6. **Duplicate Prevention**: Checks database before downloading
|
||||
7. **Timestamp Accuracy**: Updates file system and EXIF timestamps
|
||||
8. **Cookie Persistence**: Saves/loads cookies for faster runs
|
||||
9. **Cloudflare Bypass**: Optional 2captcha integration
|
||||
10. **File Organization**: Automatic moving to destination
|
||||
11. **Subprocess Isolation**: Prevents event loop conflicts
|
||||
12. **Logging**: Comprehensive logging with callback support
|
||||
13. **Error Handling**: Robust error handling and recovery
|
||||
14. **Scheduler Integration**: Supports scheduled downloads
|
||||
15. **Batch Processing**: Supports multiple users
|
||||
|
||||
### ✅ Architecture Alignment
|
||||
- Follows ImgInn module pattern exactly
|
||||
- Uses same subprocess wrapper approach
|
||||
- Integrates with move_module for file management
|
||||
- Uses unified_database for tracking
|
||||
- Compatible with scheduler system
|
||||
- Supports Pushover notifications via move_module
|
||||
- Works with Immich scanning
|
||||
|
||||
## Configuration Structure
|
||||
|
||||
```json
|
||||
{
|
||||
"snapchat": {
|
||||
"enabled": true,
|
||||
"check_interval_hours": 6,
|
||||
"twocaptcha_api_key": "",
|
||||
"cookie_file": "/opt/media-downloader/cookies/snapchat_cookies.json",
|
||||
"usernames": ["user1", "user2"],
|
||||
"stories": {
|
||||
"enabled": true,
|
||||
"days_back": 7,
|
||||
"max_downloads": 50,
|
||||
"temp_dir": "temp/snapchat/stories",
|
||||
"destination_path": "/path/to/media/library/Snapchat"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Download from all platforms (includes Snapchat):
|
||||
```bash
|
||||
cd /opt/media-downloader
|
||||
./venv/bin/python media-downloader.py --platform all
|
||||
```
|
||||
|
||||
### Download only Snapchat:
|
||||
```bash
|
||||
./venv/bin/python media-downloader.py --platform snapchat
|
||||
```
|
||||
|
||||
### Run with scheduler:
|
||||
```bash
|
||||
./venv/bin/python media-downloader.py --scheduler
|
||||
```
|
||||
|
||||
### Test standalone module:
|
||||
```bash
|
||||
./venv/bin/python modules/snapchat_module.py username_to_test
|
||||
```
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Modified Files
|
||||
1. **media-downloader.py**:
|
||||
- Line 47: Import SnapchatDownloader
|
||||
- Line 423-436: Module initialization
|
||||
- Line 511-513: Interval configuration
|
||||
- Line 1187-1325: Download methods
|
||||
- Line 1959-1962: Integration in run()
|
||||
- Line 1998: Command-line choices
|
||||
- Line 2179-2181, 2283-2285: Scheduler filtering
|
||||
- Line 2511-2512: Command-line handler
|
||||
|
||||
2. **unified_database.py**:
|
||||
- Line 1300-1325: SnapchatDatabaseAdapter class
|
||||
|
||||
## File Naming Convention
|
||||
|
||||
**Format**: `{username}_{YYYYMMDD_HHMMSS}_{media_id}.{ext}`
|
||||
|
||||
**Example**: `johndoe_20250123_143022_abc123def456789.jpg`
|
||||
|
||||
**Components**:
|
||||
- username: Snapchat username (lowercase)
|
||||
- YYYYMMDD: Date the story was posted (or current date)
|
||||
- HHMMSS: Time the story was posted (or current time)
|
||||
- media_id: Unique identifier from the media URL
|
||||
- ext: File extension (.jpg, .mp4, etc.)
|
||||
|
||||
## Database Schema
|
||||
|
||||
Stories are recorded in the unified database:
|
||||
- **platform**: 'snapchat'
|
||||
- **source**: username
|
||||
- **content_type**: 'story'
|
||||
- **url**: Original media URL
|
||||
- **filename**: Final filename
|
||||
- **post_date**: Story date/time
|
||||
- **metadata**: JSON with media_id and other info
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
### Before First Run:
|
||||
- [ ] Add configuration to settings.json
|
||||
- [ ] Set enabled: true
|
||||
- [ ] Add at least one username
|
||||
- [ ] Set destination_path
|
||||
- [ ] Configure download_settings.move_to_destination: true
|
||||
- [ ] Ensure Xvfb is running (./run-with-xvfb.sh)
|
||||
|
||||
### Test Execution:
|
||||
- [ ] Test standalone module: `./venv/bin/python modules/snapchat_module.py username`
|
||||
- [ ] Test via main script: `./venv/bin/python media-downloader.py --platform snapchat`
|
||||
- [ ] Verify files downloaded to temp directory
|
||||
- [ ] Verify files moved to destination
|
||||
- [ ] Check database has records
|
||||
- [ ] Verify no duplicate downloads on re-run
|
||||
- [ ] Check logs for errors
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **StoryClon e Dependency**: Relies on s.storyclone.com being available
|
||||
2. **Stories Only**: Only downloads stories, not direct posts/snaps
|
||||
3. **24-Hour Expiry**: Stories expire after 24 hours on Snapchat
|
||||
4. **Cloudflare**: May require 2captcha API key for Cloudflare challenges
|
||||
5. **Date Accuracy**: Story dates may not always be accurate (uses current date if unavailable)
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
1. Support additional Snapchat proxy services
|
||||
2. Parallel processing of multiple users
|
||||
3. Story caption/metadata extraction
|
||||
4. Automatic retry on failures
|
||||
5. Quality selection (if available)
|
||||
6. Video thumbnail generation
|
||||
7. Story highlights download
|
||||
|
||||
## Comparison with ImgInn Module
|
||||
|
||||
| Feature | ImgInn | Snapchat | Status |
|
||||
|---------|--------|----------|--------|
|
||||
| Posts | ✅ | ❌ | N/A for Snapchat |
|
||||
| Stories | ✅ | ✅ | ✅ Implemented |
|
||||
| Browser Automation | ✅ | ✅ | ✅ Implemented |
|
||||
| Subprocess Isolation | ✅ | ✅ | ✅ Implemented |
|
||||
| Database Tracking | ✅ | ✅ | ✅ Implemented |
|
||||
| Cookie Persistence | ✅ | ✅ | ✅ Implemented |
|
||||
| 2captcha Support | ✅ | ✅ | ✅ Implemented |
|
||||
| Phrase Search | ✅ | ❌ | N/A for stories |
|
||||
| FastDL Naming | ✅ | ✅ | ✅ Implemented |
|
||||
| Timestamp Updates | ✅ | ✅ | ✅ Implemented |
|
||||
|
||||
## Success Criteria
|
||||
|
||||
✅ All criteria met:
|
||||
1. ✅ Module follows ImgInn architecture pattern
|
||||
2. ✅ Uses StoryClon e proxy (s.storyclone.com/u/<user>/)
|
||||
3. ✅ Downloads Snapchat stories
|
||||
4. ✅ FastDL-compatible file naming
|
||||
5. ✅ Integrated with unified database
|
||||
6. ✅ Subprocess isolation implemented
|
||||
7. ✅ Command-line support added
|
||||
8. ✅ Scheduler integration complete
|
||||
9. ✅ Configuration example created
|
||||
10. ✅ Documentation written
|
||||
|
||||
## Next Steps for User
|
||||
|
||||
1. **Configure**: Add Snapchat config to settings.json
|
||||
2. **Enable**: Set snapchat.enabled: true
|
||||
3. **Add Users**: Add Snapchat usernames to download from
|
||||
4. **Test**: Run `./venv/bin/python media-downloader.py --platform snapchat`
|
||||
5. **Schedule**: Enable scheduler for automatic downloads
|
||||
6. **Monitor**: Check logs and database for successful downloads
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check SNAPCHAT_README.md for troubleshooting
|
||||
2. Review logs in /opt/media-downloader/logs/
|
||||
3. Test standalone module for detailed output
|
||||
4. Check database entries: `sqlite3 database/media_downloader.db "SELECT * FROM downloads WHERE platform='snapchat';"`
|
||||
|
||||
---
|
||||
|
||||
**Implementation Date**: 2025-10-23
|
||||
**Based On**: ImgInn module architecture
|
||||
**Status**: ✅ Complete and ready for testing
|
||||
Reference in New Issue
Block a user