332 lines
7.7 KiB
Markdown
332 lines
7.7 KiB
Markdown
# Downloader Monitoring System
|
|
|
|
## Overview
|
|
|
|
The Downloader Monitoring System tracks the health of all downloader modules and sends push notifications when a downloader has been consistently failing for a specified time period (default: 3 hours).
|
|
|
|
## Features
|
|
|
|
✅ **Per-Downloader Tracking** - Monitors each downloader independently:
|
|
- fastdl (Instagram web scraper)
|
|
- imginn (Instagram alternative scraper)
|
|
- toolzu (Instagram high-res scraper)
|
|
- instagram (Instaloader API)
|
|
- snapchat (Direct Playwright scraper)
|
|
- tiktok (yt-dlp)
|
|
- forums (XenForo/vBulletin scrapers)
|
|
- coppermine (Coppermine Photo Gallery scraper)
|
|
|
|
✅ **Smart Alerting** - Only alerts once per issue (no spam)
|
|
|
|
✅ **Pushover Notifications** - Sends high-priority push notifications
|
|
|
|
✅ **Configurable Thresholds** - Customize failure windows and minimum failures
|
|
|
|
✅ **Automatic Cleanup** - Removes old monitoring logs automatically
|
|
|
|
## How It Works
|
|
|
|
### 1. Download Tracking
|
|
Every download attempt is logged to the `download_monitor` table:
|
|
```sql
|
|
INSERT INTO download_monitor (
|
|
downloader, -- 'fastdl', 'snapchat', etc.
|
|
username, -- User being downloaded
|
|
timestamp, -- When the attempt occurred
|
|
success, -- 1 = success, 0 = failure
|
|
file_count, -- Number of files downloaded
|
|
error_message, -- Error details if failed
|
|
alert_sent -- Whether alert was sent
|
|
)
|
|
```
|
|
|
|
### 2. Failure Detection
|
|
When a download fails, the system:
|
|
1. Checks the last N attempts within the time window
|
|
2. Counts consecutive failures
|
|
3. If failures ≥ threshold → Send alert
|
|
4. Marks the failure as alerted (prevents duplicate notifications)
|
|
|
|
### 3. Push Notifications
|
|
Alert format:
|
|
```
|
|
🚨 FastDL Failing
|
|
|
|
Downloader has been failing for 3+ hours
|
|
|
|
Username: evalongoria
|
|
Consecutive Failures: 3
|
|
Last Success: 6 hours ago
|
|
Latest Error: "Cloudflare challenge"
|
|
|
|
Check logs for details.
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Database Settings
|
|
Configuration is stored in the `settings` table:
|
|
```json
|
|
{
|
|
"enabled": true,
|
|
"failure_window_hours": 3,
|
|
"min_consecutive_failures": 2,
|
|
"pushover": {
|
|
"enabled": true,
|
|
"priority": 1
|
|
},
|
|
"downloaders": {
|
|
"fastdl": true,
|
|
"imginn": true,
|
|
"toolzu": true,
|
|
"instagram": true,
|
|
"snapchat": true,
|
|
"tiktok": true,
|
|
"forums": true
|
|
}
|
|
}
|
|
```
|
|
|
|
### Configuration Options
|
|
|
|
| Option | Default | Description |
|
|
|--------|---------|-------------|
|
|
| `enabled` | `true` | Enable/disable monitoring system |
|
|
| `failure_window_hours` | `3` | How many hours to look back |
|
|
| `min_consecutive_failures` | `2` | Minimum failures to trigger alert |
|
|
| `pushover.enabled` | `true` | Enable Pushover notifications |
|
|
| `pushover.priority` | `1` | Notification priority (1 = high) |
|
|
| `downloaders.*` | `true` | Enable/disable per-downloader monitoring |
|
|
|
|
### Updating Configuration
|
|
Via Web UI (coming soon) or database:
|
|
```sql
|
|
UPDATE settings
|
|
SET value = json_set(value, '$.failure_window_hours', 6)
|
|
WHERE key = 'monitoring';
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### Get Monitoring Status
|
|
```http
|
|
GET /api/monitoring/status?hours=24
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"downloaders": [
|
|
{
|
|
"downloader": "fastdl",
|
|
"total_attempts": 10,
|
|
"successful": 8,
|
|
"failed": 2,
|
|
"total_files": 45,
|
|
"success_rate": 80.0,
|
|
"last_success": "2025-11-19T06:00:00",
|
|
"last_attempt": "2025-11-19T09:00:00"
|
|
}
|
|
],
|
|
"window_hours": 24
|
|
}
|
|
```
|
|
|
|
### Get Monitoring History
|
|
```http
|
|
GET /api/monitoring/history?downloader=fastdl&limit=100
|
|
```
|
|
|
|
**Response:**
|
|
```json
|
|
{
|
|
"success": true,
|
|
"history": [
|
|
{
|
|
"id": 1,
|
|
"downloader": "fastdl",
|
|
"username": "evalongoria",
|
|
"timestamp": "2025-11-19T09:00:00",
|
|
"success": false,
|
|
"file_count": 0,
|
|
"error_message": "Cloudflare challenge",
|
|
"alert_sent": true
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
### Clear Old Logs
|
|
```http
|
|
DELETE /api/monitoring/history?days=30
|
|
```
|
|
|
|
Removes logs older than 30 days.
|
|
|
|
## Database Schema
|
|
|
|
```sql
|
|
CREATE TABLE download_monitor (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
downloader TEXT NOT NULL,
|
|
username TEXT,
|
|
timestamp TEXT NOT NULL,
|
|
success INTEGER NOT NULL,
|
|
file_count INTEGER DEFAULT 0,
|
|
error_message TEXT,
|
|
alert_sent INTEGER DEFAULT 0,
|
|
created_at TEXT DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
CREATE INDEX idx_download_monitor_downloader ON download_monitor(downloader);
|
|
CREATE INDEX idx_download_monitor_timestamp ON download_monitor(timestamp);
|
|
CREATE INDEX idx_download_monitor_success ON download_monitor(success);
|
|
```
|
|
|
|
## Module Architecture
|
|
|
|
### Core Modules
|
|
|
|
**`modules/downloader_monitor.py`**
|
|
- Main monitoring logic
|
|
- Tracks download attempts
|
|
- Checks for persistent failures
|
|
- Sends Pushover alerts
|
|
- Provides status queries
|
|
- Cleans up old logs
|
|
|
|
**`modules/monitor_wrapper.py`**
|
|
- Helper functions for integration
|
|
- `log_download_result()` - Simple logging function
|
|
- `@monitor_download()` - Decorator (future use)
|
|
|
|
### Integration Points
|
|
|
|
**Subprocess Wrappers:**
|
|
- `wrappers/fastdl_subprocess_wrapper.py`
|
|
- `wrappers/imginn_subprocess_wrapper.py`
|
|
- `wrappers/toolzu_subprocess_wrapper.py`
|
|
- `wrappers/snapchat_subprocess_wrapper.py`
|
|
|
|
Each wrapper calls:
|
|
```python
|
|
from modules.monitor_wrapper import log_download_result
|
|
|
|
# After download
|
|
log_download_result('fastdl', username, count, error=None)
|
|
|
|
# On failure
|
|
log_download_result('fastdl', username, 0, error=str(e))
|
|
```
|
|
|
|
## Example Scenarios
|
|
|
|
### Scenario 1: Temporary Failure
|
|
```
|
|
09:00 - fastdl: Failed (Cloudflare)
|
|
12:00 - fastdl: Success (5 files)
|
|
```
|
|
**Result:** No alert (recovered before threshold)
|
|
|
|
### Scenario 2: Persistent Failure
|
|
```
|
|
09:00 - fastdl: Failed (Cloudflare)
|
|
12:00 - fastdl: Failed (Cloudflare)
|
|
15:00 - fastdl: Failed (Cloudflare)
|
|
```
|
|
**Result:** 🚨 Alert sent at 12:00 (2 consecutive failures within 3 hours)
|
|
|
|
### Scenario 3: Multiple Downloaders
|
|
```
|
|
09:00 - fastdl: Success (3 files)
|
|
09:00 - toolzu: Failed (Rate limited)
|
|
12:00 - fastdl: Success (2 files)
|
|
12:00 - toolzu: Failed (Rate limited)
|
|
```
|
|
**Result:** 🚨 Alert for toolzu only (fastdl working fine)
|
|
|
|
## Maintenance
|
|
|
|
### View Current Status
|
|
```bash
|
|
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
|
SELECT
|
|
downloader,
|
|
COUNT(*) as total,
|
|
SUM(success) as successful,
|
|
SUM(CASE WHEN success=0 THEN 1 ELSE 0 END) as failed
|
|
FROM download_monitor
|
|
WHERE timestamp > datetime('now', '-24 hours')
|
|
GROUP BY downloader;
|
|
"
|
|
```
|
|
|
|
### Manual Cleanup
|
|
```bash
|
|
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
|
DELETE FROM download_monitor
|
|
WHERE timestamp < datetime('now', '-30 days');
|
|
"
|
|
```
|
|
|
|
### View Recent Failures
|
|
```bash
|
|
sqlite3 /opt/media-downloader/database/media_downloader.db "
|
|
SELECT downloader, username, timestamp, error_message
|
|
FROM download_monitor
|
|
WHERE success = 0
|
|
ORDER BY timestamp DESC
|
|
LIMIT 10;
|
|
"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### No Alerts Being Sent
|
|
1. Check Pushover configuration:
|
|
```sql
|
|
SELECT value FROM settings WHERE key = 'pushover';
|
|
```
|
|
2. Verify monitoring is enabled:
|
|
```sql
|
|
SELECT value FROM settings WHERE key = 'monitoring';
|
|
```
|
|
3. Check logs:
|
|
```bash
|
|
grep -i "monitor\|alert" /opt/media-downloader/logs/*_api.log
|
|
```
|
|
|
|
### Too Many Alerts
|
|
Increase thresholds:
|
|
```sql
|
|
UPDATE settings
|
|
SET value = json_set(value, '$.min_consecutive_failures', 5)
|
|
WHERE key = 'monitoring';
|
|
```
|
|
|
|
### Disable Monitoring for Specific Downloader
|
|
```sql
|
|
UPDATE settings
|
|
SET value = json_set(value, '$.downloaders.fastdl', false)
|
|
WHERE key = 'monitoring';
|
|
```
|
|
|
|
## Future Enhancements
|
|
|
|
- [ ] Web UI dashboard for monitoring
|
|
- [ ] Historical charts and graphs
|
|
- [ ] Downloader performance metrics
|
|
- [ ] Email notifications (in addition to Pushover)
|
|
- [ ] Webhook support for custom integrations
|
|
- [ ] Automatic remediation actions
|
|
|
|
## Version History
|
|
|
|
**v6.36.1** - Initial implementation
|
|
- Database schema
|
|
- Monitoring module
|
|
- Pushover integration
|
|
- API endpoints
|
|
- Integration with all downloaders
|