Todd
368
docs/DEPENDENCY_UPDATES.md
Normal file
368
docs/DEPENDENCY_UPDATES.md
Normal file
@@ -0,0 +1,368 @@
|
||||
# Automatic Dependency Updates
|
||||
|
||||
## Overview
|
||||
|
||||
The Dependency Updater automatically checks for and installs updates for critical components once per day when running in scheduler mode. This ensures FlareSolverr, Playwright browsers, and yt-dlp stay current without manual intervention.
|
||||
|
||||
## Why Auto-Updates?
|
||||
|
||||
**Critical dependencies that require frequent updates:**
|
||||
|
||||
1. **FlareSolverr** - Cloudflare bypass technology
|
||||
- Cloudflare frequently updates their bot detection
|
||||
- FlareSolverr updates to counter new blocks
|
||||
- Outdated version = downloads fail with Cloudflare errors
|
||||
|
||||
2. **yt-dlp** - Video download engine (TikTok, etc.)
|
||||
- TikTok/YouTube change their APIs constantly
|
||||
- yt-dlp releases updates almost daily
|
||||
- Outdated version = TikTok downloads fail
|
||||
|
||||
3. **Playwright Browsers** - Chromium/Firefox automation
|
||||
- Browser updates include security fixes
|
||||
- Anti-detection improvements
|
||||
- Outdated browsers are easier to detect
|
||||
|
||||
## How It Works
|
||||
|
||||
### Automatic Check Schedule
|
||||
|
||||
- **Runs**: Once every 24 hours (configurable)
|
||||
- **Mode**: Scheduler only (not manual runs)
|
||||
- **Time**: Checks every minute, but internal cooldown prevents spam
|
||||
- **Location**: Integrated into scheduler loop
|
||||
|
||||
### Update Process
|
||||
|
||||
```
|
||||
Scheduler Running
|
||||
↓
|
||||
Every 60 seconds:
|
||||
↓
|
||||
Check if 24 hours passed since last update check
|
||||
↓ Yes
|
||||
Update Components:
|
||||
1. FlareSolverr (docker pull + restart)
|
||||
2. Playwright (chromium + firefox)
|
||||
3. yt-dlp (pip upgrade)
|
||||
↓
|
||||
Log Results
|
||||
↓
|
||||
Send Notification (if updates installed)
|
||||
↓
|
||||
Save State with Timestamp
|
||||
↓
|
||||
Resume Scheduler
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Located in `config/settings.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"dependency_updates": {
|
||||
"enabled": true,
|
||||
"check_interval_hours": 24,
|
||||
"auto_install": true,
|
||||
"components": {
|
||||
"flaresolverr": {
|
||||
"enabled": true,
|
||||
"notify_on_update": true
|
||||
},
|
||||
"playwright": {
|
||||
"enabled": true,
|
||||
"notify_on_update": false
|
||||
},
|
||||
"yt_dlp": {
|
||||
"enabled": true,
|
||||
"notify_on_update": false
|
||||
}
|
||||
},
|
||||
"pushover": {
|
||||
"enabled": true,
|
||||
"priority": -1,
|
||||
"sound": "magic"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
**Main Settings:**
|
||||
- `enabled` (boolean) - Master switch for auto-updates (default: true)
|
||||
- `check_interval_hours` (integer) - Hours between update checks (default: 24)
|
||||
- `auto_install` (boolean) - Automatically install updates (default: true)
|
||||
|
||||
**Component Settings:**
|
||||
- `enabled` (boolean) - Enable updates for this component
|
||||
- `notify_on_update` (boolean) - Send Pushover notification when updated
|
||||
|
||||
**Pushover Settings:**
|
||||
- `enabled` (boolean) - Enable update notifications
|
||||
- `priority` (integer) - Notification priority (-2 to 2, -1 = low)
|
||||
- `sound` (string) - Notification sound (default: "magic")
|
||||
|
||||
## Update Components
|
||||
|
||||
### 1. FlareSolverr (Docker Container)
|
||||
|
||||
**Why**: Cloudflare constantly updates bot detection; FlareSolverr must keep pace
|
||||
|
||||
**Update Process:**
|
||||
```bash
|
||||
1. docker pull ghcr.io/flaresolverr/flaresolverr:latest
|
||||
2. If new image downloaded:
|
||||
a. docker stop flaresolverr
|
||||
b. docker rm flaresolverr
|
||||
c. docker run -d --name flaresolverr -p 8191:8191 ...
|
||||
3. Container running with latest version
|
||||
```
|
||||
|
||||
**Notification**: ✅ Enabled by default (important update)
|
||||
|
||||
**Downtime**: ~5 seconds during container restart
|
||||
|
||||
### 2. Playwright Browsers (Chromium + Firefox)
|
||||
|
||||
**Why**: Browser updates include anti-detection improvements and security fixes
|
||||
|
||||
**Update Process:**
|
||||
```bash
|
||||
1. python3 -m playwright install chromium
|
||||
2. python3 -m playwright install firefox
|
||||
3. Browsers updated in /opt/media-downloader/.playwright/
|
||||
```
|
||||
|
||||
**Notification**: ❌ Disabled by default (routine update)
|
||||
|
||||
**Downtime**: None (browsers updated while not in use)
|
||||
|
||||
### 3. yt-dlp (Python Package)
|
||||
|
||||
**Why**: TikTok/YouTube change APIs constantly; yt-dlp updates almost daily
|
||||
|
||||
**Update Process:**
|
||||
```bash
|
||||
1. pip3 install --upgrade yt-dlp
|
||||
2. Latest version installed system-wide
|
||||
```
|
||||
|
||||
**Notification**: ❌ Disabled by default (very frequent)
|
||||
|
||||
**Downtime**: None
|
||||
|
||||
## Notification Examples
|
||||
|
||||
**FlareSolverr Update:**
|
||||
```
|
||||
🔄 Dependencies Updated
|
||||
|
||||
FlareSolverr has been updated to the latest version.
|
||||
|
||||
Updated at: Oct 29, 3:15 AM
|
||||
```
|
||||
|
||||
**Multiple Updates:**
|
||||
```
|
||||
🔄 Dependencies Updated
|
||||
|
||||
The following components have been updated:
|
||||
|
||||
• FlareSolverr
|
||||
• Playwright Browsers
|
||||
• yt-dlp
|
||||
|
||||
Updated at: Oct 29, 3:15 AM
|
||||
```
|
||||
|
||||
## State Tracking
|
||||
|
||||
State stored in `/opt/media-downloader/database/dependency_updates.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"last_check": "2025-10-29T03:15:00",
|
||||
"components": {
|
||||
"flaresolverr": {
|
||||
"last_update": "2025-10-29T03:15:00",
|
||||
"last_check": "2025-10-29T03:15:00",
|
||||
"status": "updated"
|
||||
},
|
||||
"playwright": {
|
||||
"last_update": "2025-10-28T03:15:00",
|
||||
"last_check": "2025-10-29T03:15:00",
|
||||
"status": "current"
|
||||
},
|
||||
"yt_dlp": {
|
||||
"last_update": "2025-10-29T03:15:00",
|
||||
"last_check": "2025-10-29T03:15:00",
|
||||
"status": "updated"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
### Manual Update Check
|
||||
|
||||
```python
|
||||
from modules.dependency_updater import DependencyUpdater
|
||||
from modules.pushover_notifier import create_notifier_from_config
|
||||
import json
|
||||
|
||||
# Load config
|
||||
with open('/opt/media-downloader/config/settings.json') as f:
|
||||
config = json.load(f)
|
||||
|
||||
# Initialize updater
|
||||
notifier = create_notifier_from_config(config)
|
||||
updater = DependencyUpdater(
|
||||
config=config.get('dependency_updates', {}),
|
||||
pushover_notifier=notifier,
|
||||
scheduler_mode=True
|
||||
)
|
||||
|
||||
# Force update check (ignores 24h cooldown)
|
||||
results = updater.force_update_check()
|
||||
|
||||
print("Update Results:")
|
||||
for component, updated in results.items():
|
||||
status = "✓ Updated" if updated else "Already current"
|
||||
print(f" {component}: {status}")
|
||||
```
|
||||
|
||||
### Check Last Update Time
|
||||
|
||||
```bash
|
||||
cat /opt/media-downloader/database/dependency_updates.json | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Monitor Updates in Logs
|
||||
|
||||
```bash
|
||||
tail -f /opt/media-downloader/logs/*.log | grep -i "dependency\|update"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Updates not running:**
|
||||
- Check `dependency_updates.enabled` is `true`
|
||||
- Verify running in scheduler mode (not manual)
|
||||
- Check last_check timestamp in state file
|
||||
- Ensure 24 hours have passed since last check
|
||||
|
||||
**FlareSolverr update fails:**
|
||||
- Check Docker is running: `docker ps`
|
||||
- Check internet connection
|
||||
- Check Docker Hub access: `docker pull ghcr.io/flaresolverr/flaresolverr:latest`
|
||||
- Review error in logs
|
||||
|
||||
**Playwright update fails:**
|
||||
- Check disk space: `df -h`
|
||||
- Check Python environment
|
||||
- Manual update: `python3 -m playwright install chromium firefox`
|
||||
|
||||
**yt-dlp update fails:**
|
||||
- Check pip permissions
|
||||
- Manual update: `pip3 install --upgrade yt-dlp`
|
||||
- Check internet connection
|
||||
|
||||
**Too many notifications:**
|
||||
- Disable per-component: `notify_on_update: false`
|
||||
- Disable all notifications: `pushover.enabled: false`
|
||||
- Keep enabled only for critical (FlareSolverr)
|
||||
|
||||
**Want to disable auto-updates:**
|
||||
```json
|
||||
{
|
||||
"dependency_updates": {
|
||||
"enabled": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Want to disable specific component:**
|
||||
```json
|
||||
{
|
||||
"dependency_updates": {
|
||||
"components": {
|
||||
"yt_dlp": {
|
||||
"enabled": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Manual Updates
|
||||
|
||||
If you prefer manual updates, disable auto-updates and run:
|
||||
|
||||
```bash
|
||||
# Update FlareSolverr
|
||||
docker pull ghcr.io/flaresolverr/flaresolverr:latest
|
||||
docker stop flaresolverr && docker rm flaresolverr
|
||||
docker run -d --name flaresolverr -p 8191:8191 -e LOG_LEVEL=info --restart unless-stopped ghcr.io/flaresolverr/flaresolverr:latest
|
||||
|
||||
# Update Playwright
|
||||
cd /opt/media-downloader
|
||||
python3 -m playwright install chromium firefox
|
||||
|
||||
# Update yt-dlp
|
||||
pip3 install --upgrade yt-dlp
|
||||
```
|
||||
|
||||
## Logs
|
||||
|
||||
Update activity logged with `[DependencyUpdater]` tag:
|
||||
|
||||
```
|
||||
2025-10-29 03:15:00 [DependencyUpdater] [INFO] Checking for dependency updates...
|
||||
2025-10-29 03:15:05 [DependencyUpdater] [INFO] Checking FlareSolverr for updates...
|
||||
2025-10-29 03:15:10 [DependencyUpdater] [INFO] ✓ FlareSolverr updated and restarted successfully
|
||||
2025-10-29 03:15:15 [DependencyUpdater] [INFO] Checking Playwright browsers for updates...
|
||||
2025-10-29 03:15:45 [DependencyUpdater] [INFO] Playwright browsers already up to date
|
||||
2025-10-29 03:15:46 [DependencyUpdater] [INFO] Checking yt-dlp for updates...
|
||||
2025-10-29 03:15:50 [DependencyUpdater] [INFO] ✓ yt-dlp updated successfully
|
||||
2025-10-29 03:15:51 [DependencyUpdater] [INFO] Sent update notification for: FlareSolverr, yt-dlp
|
||||
```
|
||||
|
||||
## Benefits
|
||||
|
||||
✅ **Zero Maintenance** - Updates install automatically
|
||||
✅ **Always Current** - Critical dependencies stay up to date
|
||||
✅ **Prevents Failures** - Outdated FlareSolverr/yt-dlp cause download failures
|
||||
✅ **Non-Intrusive** - Low-priority notifications, doesn't interrupt workflow
|
||||
✅ **Reliable** - Handles failures gracefully, won't crash scheduler
|
||||
✅ **Configurable** - Enable/disable per component or globally
|
||||
|
||||
## Security Considerations
|
||||
|
||||
**Automatic updates are safe:**
|
||||
- Only updates from official sources (Docker Hub, PyPI)
|
||||
- Uses official image tags (`:latest`)
|
||||
- No code execution from untrusted sources
|
||||
- Same update process as manual updates
|
||||
|
||||
**Risk Mitigation:**
|
||||
- Version backups taken before major changes
|
||||
- Logs all update activity
|
||||
- Can disable if stability is critical
|
||||
- Can rollback FlareSolverr to specific version
|
||||
|
||||
**Recommended for most users:**
|
||||
- ✅ Enable for production (scheduler mode)
|
||||
- ✅ Keeps services working when APIs change
|
||||
- ✅ Minimal risk, high benefit
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- Update rollback if service fails after update
|
||||
- Pinning specific versions
|
||||
- Update schedule (time of day)
|
||||
- Pre-update testing
|
||||
- Update changelog notifications
|
||||
- Critical security update alerts
|
||||
Reference in New Issue
Block a user