259 lines
8.4 KiB
Markdown
259 lines
8.4 KiB
Markdown
# Snapchat Downloader Implementation Summary
|
|
|
|
## Overview
|
|
Successfully implemented a complete Snapchat downloader module for the media-downloader system, based on the ImgInn module architecture. The module downloads Snapchat stories via the StoryClon e proxy (https://s.storyclone.com/u/<user>/).
|
|
|
|
## Files Created
|
|
|
|
### 1. Core Module
|
|
**File**: `/opt/media-downloader/modules/snapchat_module.py`
|
|
- Main SnapchatDownloader class
|
|
- Browser automation with Playwright
|
|
- FastDL-compatible file naming
|
|
- Cookie management
|
|
- Cloudflare challenge handling
|
|
- Database integration
|
|
- Timestamp updating (file system + EXIF)
|
|
- Story extraction and downloading
|
|
|
|
### 2. Subprocess Wrapper
|
|
**File**: `/opt/media-downloader/snapchat_subprocess_wrapper.py`
|
|
- Isolates Snapchat operations in separate process
|
|
- Avoids asyncio event loop conflicts
|
|
- JSON-based configuration input/output
|
|
- Stderr logging for clean stdout
|
|
|
|
### 3. Database Adapter
|
|
**File**: `/opt/media-downloader/modules/unified_database.py` (modified)
|
|
- Added SnapchatDatabaseAdapter class
|
|
- Tracks downloads by URL and metadata
|
|
- Platform: 'snapchat'
|
|
- Content type: 'story'
|
|
|
|
### 4. Main Integration
|
|
**File**: `/opt/media-downloader/media-downloader.py` (modified)
|
|
- Imported SnapchatDownloader module
|
|
- Added initialization in _init_modules()
|
|
- Added interval configuration (check_interval_hours)
|
|
- Created _download_snapchat_content() method
|
|
- Created download_snapchat() method
|
|
- Integrated into run() method (download all platforms)
|
|
- Added command-line argument support: --platform snapchat
|
|
- Added scheduler filtering support
|
|
|
|
### 5. Configuration Example
|
|
**File**: `/opt/media-downloader/config/snapchat_example.json`
|
|
- Sample configuration structure
|
|
- All available settings documented
|
|
- Ready to copy into main settings.json
|
|
|
|
### 6. Documentation
|
|
**File**: `/opt/media-downloader/SNAPCHAT_README.md`
|
|
- Complete usage guide
|
|
- Setup instructions
|
|
- Configuration options explained
|
|
- Troubleshooting section
|
|
- Architecture overview
|
|
|
|
## Key Features Implemented
|
|
|
|
### ✅ Complete Feature Set
|
|
1. **Browser Automation**: Playwright-based Chromium automation
|
|
2. **Proxy Support**: Uses StoryClon e (s.storyclone.com) proxy
|
|
3. **Story Downloads**: Extracts and downloads all available stories
|
|
4. **FastDL Naming**: Compatible filename format (user_date_mediaid.ext)
|
|
5. **Database Tracking**: Full integration with unified database
|
|
6. **Duplicate Prevention**: Checks database before downloading
|
|
7. **Timestamp Accuracy**: Updates file system and EXIF timestamps
|
|
8. **Cookie Persistence**: Saves/loads cookies for faster runs
|
|
9. **Cloudflare Bypass**: Optional 2captcha integration
|
|
10. **File Organization**: Automatic moving to destination
|
|
11. **Subprocess Isolation**: Prevents event loop conflicts
|
|
12. **Logging**: Comprehensive logging with callback support
|
|
13. **Error Handling**: Robust error handling and recovery
|
|
14. **Scheduler Integration**: Supports scheduled downloads
|
|
15. **Batch Processing**: Supports multiple users
|
|
|
|
### ✅ Architecture Alignment
|
|
- Follows ImgInn module pattern exactly
|
|
- Uses same subprocess wrapper approach
|
|
- Integrates with move_module for file management
|
|
- Uses unified_database for tracking
|
|
- Compatible with scheduler system
|
|
- Supports Pushover notifications via move_module
|
|
- Works with Immich scanning
|
|
|
|
## Configuration Structure
|
|
|
|
```json
|
|
{
|
|
"snapchat": {
|
|
"enabled": true,
|
|
"check_interval_hours": 6,
|
|
"twocaptcha_api_key": "",
|
|
"cookie_file": "/opt/media-downloader/cookies/snapchat_cookies.json",
|
|
"usernames": ["user1", "user2"],
|
|
"stories": {
|
|
"enabled": true,
|
|
"days_back": 7,
|
|
"max_downloads": 50,
|
|
"temp_dir": "temp/snapchat/stories",
|
|
"destination_path": "/path/to/media/library/Snapchat"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
## Usage Examples
|
|
|
|
### Download from all platforms (includes Snapchat):
|
|
```bash
|
|
cd /opt/media-downloader
|
|
./venv/bin/python media-downloader.py --platform all
|
|
```
|
|
|
|
### Download only Snapchat:
|
|
```bash
|
|
./venv/bin/python media-downloader.py --platform snapchat
|
|
```
|
|
|
|
### Run with scheduler:
|
|
```bash
|
|
./venv/bin/python media-downloader.py --scheduler
|
|
```
|
|
|
|
### Test standalone module:
|
|
```bash
|
|
./venv/bin/python modules/snapchat_module.py username_to_test
|
|
```
|
|
|
|
## Integration Points
|
|
|
|
### Modified Files
|
|
1. **media-downloader.py**:
|
|
- Line 47: Import SnapchatDownloader
|
|
- Line 423-436: Module initialization
|
|
- Line 511-513: Interval configuration
|
|
- Line 1187-1325: Download methods
|
|
- Line 1959-1962: Integration in run()
|
|
- Line 1998: Command-line choices
|
|
- Line 2179-2181, 2283-2285: Scheduler filtering
|
|
- Line 2511-2512: Command-line handler
|
|
|
|
2. **unified_database.py**:
|
|
- Line 1300-1325: SnapchatDatabaseAdapter class
|
|
|
|
## File Naming Convention
|
|
|
|
**Format**: `{username}_{YYYYMMDD_HHMMSS}_{media_id}.{ext}`
|
|
|
|
**Example**: `johndoe_20250123_143022_abc123def456789.jpg`
|
|
|
|
**Components**:
|
|
- username: Snapchat username (lowercase)
|
|
- YYYYMMDD: Date the story was posted (or current date)
|
|
- HHMMSS: Time the story was posted (or current time)
|
|
- media_id: Unique identifier from the media URL
|
|
- ext: File extension (.jpg, .mp4, etc.)
|
|
|
|
## Database Schema
|
|
|
|
Stories are recorded in the unified database:
|
|
- **platform**: 'snapchat'
|
|
- **source**: username
|
|
- **content_type**: 'story'
|
|
- **url**: Original media URL
|
|
- **filename**: Final filename
|
|
- **post_date**: Story date/time
|
|
- **metadata**: JSON with media_id and other info
|
|
|
|
## Testing Checklist
|
|
|
|
### Before First Run:
|
|
- [ ] Add configuration to settings.json
|
|
- [ ] Set enabled: true
|
|
- [ ] Add at least one username
|
|
- [ ] Set destination_path
|
|
- [ ] Configure download_settings.move_to_destination: true
|
|
- [ ] Ensure Xvfb is running (./run-with-xvfb.sh)
|
|
|
|
### Test Execution:
|
|
- [ ] Test standalone module: `./venv/bin/python modules/snapchat_module.py username`
|
|
- [ ] Test via main script: `./venv/bin/python media-downloader.py --platform snapchat`
|
|
- [ ] Verify files downloaded to temp directory
|
|
- [ ] Verify files moved to destination
|
|
- [ ] Check database has records
|
|
- [ ] Verify no duplicate downloads on re-run
|
|
- [ ] Check logs for errors
|
|
|
|
## Known Limitations
|
|
|
|
1. **StoryClon e Dependency**: Relies on s.storyclone.com being available
|
|
2. **Stories Only**: Only downloads stories, not direct posts/snaps
|
|
3. **24-Hour Expiry**: Stories expire after 24 hours on Snapchat
|
|
4. **Cloudflare**: May require 2captcha API key for Cloudflare challenges
|
|
5. **Date Accuracy**: Story dates may not always be accurate (uses current date if unavailable)
|
|
|
|
## Future Enhancements
|
|
|
|
Potential improvements:
|
|
1. Support additional Snapchat proxy services
|
|
2. Parallel processing of multiple users
|
|
3. Story caption/metadata extraction
|
|
4. Automatic retry on failures
|
|
5. Quality selection (if available)
|
|
6. Video thumbnail generation
|
|
7. Story highlights download
|
|
|
|
## Comparison with ImgInn Module
|
|
|
|
| Feature | ImgInn | Snapchat | Status |
|
|
|---------|--------|----------|--------|
|
|
| Posts | ✅ | ❌ | N/A for Snapchat |
|
|
| Stories | ✅ | ✅ | ✅ Implemented |
|
|
| Browser Automation | ✅ | ✅ | ✅ Implemented |
|
|
| Subprocess Isolation | ✅ | ✅ | ✅ Implemented |
|
|
| Database Tracking | ✅ | ✅ | ✅ Implemented |
|
|
| Cookie Persistence | ✅ | ✅ | ✅ Implemented |
|
|
| 2captcha Support | ✅ | ✅ | ✅ Implemented |
|
|
| Phrase Search | ✅ | ❌ | N/A for stories |
|
|
| FastDL Naming | ✅ | ✅ | ✅ Implemented |
|
|
| Timestamp Updates | ✅ | ✅ | ✅ Implemented |
|
|
|
|
## Success Criteria
|
|
|
|
✅ All criteria met:
|
|
1. ✅ Module follows ImgInn architecture pattern
|
|
2. ✅ Uses StoryClon e proxy (s.storyclone.com/u/<user>/)
|
|
3. ✅ Downloads Snapchat stories
|
|
4. ✅ FastDL-compatible file naming
|
|
5. ✅ Integrated with unified database
|
|
6. ✅ Subprocess isolation implemented
|
|
7. ✅ Command-line support added
|
|
8. ✅ Scheduler integration complete
|
|
9. ✅ Configuration example created
|
|
10. ✅ Documentation written
|
|
|
|
## Next Steps for User
|
|
|
|
1. **Configure**: Add Snapchat config to settings.json
|
|
2. **Enable**: Set snapchat.enabled: true
|
|
3. **Add Users**: Add Snapchat usernames to download from
|
|
4. **Test**: Run `./venv/bin/python media-downloader.py --platform snapchat`
|
|
5. **Schedule**: Enable scheduler for automatic downloads
|
|
6. **Monitor**: Check logs and database for successful downloads
|
|
|
|
## Support
|
|
|
|
For issues or questions:
|
|
1. Check SNAPCHAT_README.md for troubleshooting
|
|
2. Review logs in /opt/media-downloader/logs/
|
|
3. Test standalone module for detailed output
|
|
4. Check database entries: `sqlite3 database/media_downloader.db "SELECT * FROM downloads WHERE platform='snapchat';"`
|
|
|
|
---
|
|
|
|
**Implementation Date**: 2025-10-23
|
|
**Based On**: ImgInn module architecture
|
|
**Status**: ✅ Complete and ready for testing
|