Todd
331
docs/DOWNLOADER_MONITORING.md
Normal file
331
docs/DOWNLOADER_MONITORING.md
Normal file
@@ -0,0 +1,331 @@
|
||||
# Downloader Monitoring System
|
||||
|
||||
## Overview
|
||||
|
||||
The Downloader Monitoring System tracks the health of all downloader modules and sends push notifications when a downloader has been consistently failing for a specified time period (default: 3 hours).
|
||||
|
||||
## Features
|
||||
|
||||
✅ **Per-Downloader Tracking** - Monitors each downloader independently:
|
||||
- fastdl (Instagram web scraper)
|
||||
- imginn (Instagram alternative scraper)
|
||||
- toolzu (Instagram high-res scraper)
|
||||
- instagram (Instaloader API)
|
||||
- snapchat (Direct Playwright scraper)
|
||||
- tiktok (yt-dlp)
|
||||
- forums (XenForo/vBulletin scrapers)
|
||||
- coppermine (Coppermine Photo Gallery scraper)
|
||||
|
||||
✅ **Smart Alerting** - Only alerts once per issue (no spam)
|
||||
|
||||
✅ **Pushover Notifications** - Sends high-priority push notifications
|
||||
|
||||
✅ **Configurable Thresholds** - Customize failure windows and minimum failures
|
||||
|
||||
✅ **Automatic Cleanup** - Removes old monitoring logs automatically
|
||||
|
||||
## How It Works
|
||||
|
||||
### 1. Download Tracking
|
||||
Every download attempt is logged to the `download_monitor` table:
|
||||
```sql
|
||||
INSERT INTO download_monitor (
|
||||
downloader, -- 'fastdl', 'snapchat', etc.
|
||||
username, -- User being downloaded
|
||||
timestamp, -- When the attempt occurred
|
||||
success, -- 1 = success, 0 = failure
|
||||
file_count, -- Number of files downloaded
|
||||
error_message, -- Error details if failed
|
||||
alert_sent -- Whether alert was sent
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Failure Detection
|
||||
When a download fails, the system:
|
||||
1. Checks the last N attempts within the time window
|
||||
2. Counts consecutive failures
|
||||
3. If failures ≥ threshold → Send alert
|
||||
4. Marks the failure as alerted (prevents duplicate notifications)
|
||||
|
||||
### 3. Push Notifications
|
||||
Alert format:
|
||||
```
|
||||
🚨 FastDL Failing
|
||||
|
||||
Downloader has been failing for 3+ hours
|
||||
|
||||
Username: evalongoria
|
||||
Consecutive Failures: 3
|
||||
Last Success: 6 hours ago
|
||||
Latest Error: "Cloudflare challenge"
|
||||
|
||||
Check logs for details.
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Database Settings
|
||||
Configuration is stored in the `settings` table:
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"failure_window_hours": 3,
|
||||
"min_consecutive_failures": 2,
|
||||
"pushover": {
|
||||
"enabled": true,
|
||||
"priority": 1
|
||||
},
|
||||
"downloaders": {
|
||||
"fastdl": true,
|
||||
"imginn": true,
|
||||
"toolzu": true,
|
||||
"instagram": true,
|
||||
"snapchat": true,
|
||||
"tiktok": true,
|
||||
"forums": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `enabled` | `true` | Enable/disable monitoring system |
|
||||
| `failure_window_hours` | `3` | How many hours to look back |
|
||||
| `min_consecutive_failures` | `2` | Minimum failures to trigger alert |
|
||||
| `pushover.enabled` | `true` | Enable Pushover notifications |
|
||||
| `pushover.priority` | `1` | Notification priority (1 = high) |
|
||||
| `downloaders.*` | `true` | Enable/disable per-downloader monitoring |
|
||||
|
||||
### Updating Configuration
|
||||
Via Web UI (coming soon) or database:
|
||||
```sql
|
||||
UPDATE settings
|
||||
SET value = json_set(value, '$.failure_window_hours', 6)
|
||||
WHERE key = 'monitoring';
|
||||
```
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Get Monitoring Status
|
||||
```http
|
||||
GET /api/monitoring/status?hours=24
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"downloaders": [
|
||||
{
|
||||
"downloader": "fastdl",
|
||||
"total_attempts": 10,
|
||||
"successful": 8,
|
||||
"failed": 2,
|
||||
"total_files": 45,
|
||||
"success_rate": 80.0,
|
||||
"last_success": "2025-11-19T06:00:00",
|
||||
"last_attempt": "2025-11-19T09:00:00"
|
||||
}
|
||||
],
|
||||
"window_hours": 24
|
||||
}
|
||||
```
|
||||
|
||||
### Get Monitoring History
|
||||
```http
|
||||
GET /api/monitoring/history?downloader=fastdl&limit=100
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"history": [
|
||||
{
|
||||
"id": 1,
|
||||
"downloader": "fastdl",
|
||||
"username": "evalongoria",
|
||||
"timestamp": "2025-11-19T09:00:00",
|
||||
"success": false,
|
||||
"file_count": 0,
|
||||
"error_message": "Cloudflare challenge",
|
||||
"alert_sent": true
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Clear Old Logs
|
||||
```http
|
||||
DELETE /api/monitoring/history?days=30
|
||||
```
|
||||
|
||||
Removes logs older than 30 days.
|
||||
|
||||
## Database Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE download_monitor (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
downloader TEXT NOT NULL,
|
||||
username TEXT,
|
||||
timestamp TEXT NOT NULL,
|
||||
success INTEGER NOT NULL,
|
||||
file_count INTEGER DEFAULT 0,
|
||||
error_message TEXT,
|
||||
alert_sent INTEGER DEFAULT 0,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX idx_download_monitor_downloader ON download_monitor(downloader);
|
||||
CREATE INDEX idx_download_monitor_timestamp ON download_monitor(timestamp);
|
||||
CREATE INDEX idx_download_monitor_success ON download_monitor(success);
|
||||
```
|
||||
|
||||
## Module Architecture
|
||||
|
||||
### Core Modules
|
||||
|
||||
**`modules/downloader_monitor.py`**
|
||||
- Main monitoring logic
|
||||
- Tracks download attempts
|
||||
- Checks for persistent failures
|
||||
- Sends Pushover alerts
|
||||
- Provides status queries
|
||||
- Cleans up old logs
|
||||
|
||||
**`modules/monitor_wrapper.py`**
|
||||
- Helper functions for integration
|
||||
- `log_download_result()` - Simple logging function
|
||||
- `@monitor_download()` - Decorator (future use)
|
||||
|
||||
### Integration Points
|
||||
|
||||
**Subprocess Wrappers:**
|
||||
- `wrappers/fastdl_subprocess_wrapper.py`
|
||||
- `wrappers/imginn_subprocess_wrapper.py`
|
||||
- `wrappers/toolzu_subprocess_wrapper.py`
|
||||
- `wrappers/snapchat_subprocess_wrapper.py`
|
||||
|
||||
Each wrapper calls:
|
||||
```python
|
||||
from modules.monitor_wrapper import log_download_result
|
||||
|
||||
# After download
|
||||
log_download_result('fastdl', username, count, error=None)
|
||||
|
||||
# On failure
|
||||
log_download_result('fastdl', username, 0, error=str(e))
|
||||
```
|
||||
|
||||
## Example Scenarios
|
||||
|
||||
### Scenario 1: Temporary Failure
|
||||
```
|
||||
09:00 - fastdl: Failed (Cloudflare)
|
||||
12:00 - fastdl: Success (5 files)
|
||||
```
|
||||
**Result:** No alert (recovered before threshold)
|
||||
|
||||
### Scenario 2: Persistent Failure
|
||||
```
|
||||
09:00 - fastdl: Failed (Cloudflare)
|
||||
12:00 - fastdl: Failed (Cloudflare)
|
||||
15:00 - fastdl: Failed (Cloudflare)
|
||||
```
|
||||
**Result:** 🚨 Alert sent at 12:00 (2 consecutive failures within 3 hours)
|
||||
|
||||
### Scenario 3: Multiple Downloaders
|
||||
```
|
||||
09:00 - fastdl: Success (3 files)
|
||||
09:00 - toolzu: Failed (Rate limited)
|
||||
12:00 - fastdl: Success (2 files)
|
||||
12:00 - toolzu: Failed (Rate limited)
|
||||
```
|
||||
**Result:** 🚨 Alert for toolzu only (fastdl working fine)
|
||||
|
||||
## Maintenance
|
||||
|
||||
### View Current Status
|
||||
```bash
|
||||
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
||||
SELECT
|
||||
downloader,
|
||||
COUNT(*) as total,
|
||||
SUM(success) as successful,
|
||||
SUM(CASE WHEN success=0 THEN 1 ELSE 0 END) as failed
|
||||
FROM download_monitor
|
||||
WHERE timestamp > datetime('now', '-24 hours')
|
||||
GROUP BY downloader;
|
||||
"
|
||||
```
|
||||
|
||||
### Manual Cleanup
|
||||
```bash
|
||||
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
||||
DELETE FROM download_monitor
|
||||
WHERE timestamp < datetime('now', '-30 days');
|
||||
"
|
||||
```
|
||||
|
||||
### View Recent Failures
|
||||
```bash
|
||||
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
||||
SELECT downloader, username, timestamp, error_message
|
||||
FROM download_monitor
|
||||
WHERE success = 0
|
||||
ORDER BY timestamp DESC
|
||||
LIMIT 10;
|
||||
"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### No Alerts Being Sent
|
||||
1. Check Pushover configuration:
|
||||
```sql
|
||||
SELECT value FROM settings WHERE key = 'pushover';
|
||||
```
|
||||
2. Verify monitoring is enabled:
|
||||
```sql
|
||||
SELECT value FROM settings WHERE key = 'monitoring';
|
||||
```
|
||||
3. Check logs:
|
||||
```bash
|
||||
grep -i "monitor\|alert" /opt/media-downloader/logs/*_api.log
|
||||
```
|
||||
|
||||
### Too Many Alerts
|
||||
Increase thresholds:
|
||||
```sql
|
||||
UPDATE settings
|
||||
SET value = json_set(value, '$.min_consecutive_failures', 5)
|
||||
WHERE key = 'monitoring';
|
||||
```
|
||||
|
||||
### Disable Monitoring for Specific Downloader
|
||||
```sql
|
||||
UPDATE settings
|
||||
SET value = json_set(value, '$.downloaders.fastdl', false)
|
||||
WHERE key = 'monitoring';
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [ ] Web UI dashboard for monitoring
|
||||
- [ ] Historical charts and graphs
|
||||
- [ ] Downloader performance metrics
|
||||
- [ ] Email notifications (in addition to Pushover)
|
||||
- [ ] Webhook support for custom integrations
|
||||
- [ ] Automatic remediation actions
|
||||
|
||||
## Version History
|
||||
|
||||
**v6.36.1** - Initial implementation
|
||||
- Database schema
|
||||
- Monitoring module
|
||||
- Pushover integration
|
||||
- API endpoints
|
||||
- Integration with all downloaders
|
||||
Reference in New Issue
Block a user