Files

Todd 0d7b2b1aab Initial commit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-29 22:42:55 -04:00

7.7 KiB

Raw Blame History

Downloader Monitoring System

Overview

The Downloader Monitoring System tracks the health of all downloader modules and sends push notifications when a downloader has been consistently failing for a specified time period (default: 3 hours).

Features

✅ Per-Downloader Tracking - Monitors each downloader independently:

fastdl (Instagram web scraper)
imginn (Instagram alternative scraper)
toolzu (Instagram high-res scraper)
instagram (Instaloader API)
snapchat (Direct Playwright scraper)
tiktok (yt-dlp)
forums (XenForo/vBulletin scrapers)
coppermine (Coppermine Photo Gallery scraper)

✅ Smart Alerting - Only alerts once per issue (no spam)

✅ Pushover Notifications - Sends high-priority push notifications

✅ Configurable Thresholds - Customize failure windows and minimum failures

✅ Automatic Cleanup - Removes old monitoring logs automatically

How It Works

1. Download Tracking

Every download attempt is logged to the download_monitor table:

INSERT INTO download_monitor (
    downloader,      -- 'fastdl', 'snapchat', etc.
    username,        -- User being downloaded
    timestamp,       -- When the attempt occurred
    success,         -- 1 = success, 0 = failure
    file_count,      -- Number of files downloaded
    error_message,   -- Error details if failed
    alert_sent       -- Whether alert was sent
)

2. Failure Detection

When a download fails, the system:

Checks the last N attempts within the time window
Counts consecutive failures
If failures ≥ threshold → Send alert
Marks the failure as alerted (prevents duplicate notifications)

3. Push Notifications

Alert format:

🚨 FastDL Failing

Downloader has been failing for 3+ hours

Username: evalongoria
Consecutive Failures: 3
Last Success: 6 hours ago
Latest Error: "Cloudflare challenge"

Check logs for details.

Configuration

Database Settings

Configuration is stored in the settings table:

{
  "enabled": true,
  "failure_window_hours": 3,
  "min_consecutive_failures": 2,
  "pushover": {
    "enabled": true,
    "priority": 1
  },
  "downloaders": {
    "fastdl": true,
    "imginn": true,
    "toolzu": true,
    "instagram": true,
    "snapchat": true,
    "tiktok": true,
    "forums": true
  }
}

Configuration Options

Option	Default	Description
`enabled`	`true`	Enable/disable monitoring system
`failure_window_hours`	`3`	How many hours to look back
`min_consecutive_failures`	`2`	Minimum failures to trigger alert
`pushover.enabled`	`true`	Enable Pushover notifications
`pushover.priority`	`1`	Notification priority (1 = high)
`downloaders.*`	`true`	Enable/disable per-downloader monitoring

Updating Configuration

Via Web UI (coming soon) or database:

UPDATE settings
SET value = json_set(value, '$.failure_window_hours', 6)
WHERE key = 'monitoring';

API Endpoints

Get Monitoring Status

GET /api/monitoring/status?hours=24

Response:

{
  "success": true,
  "downloaders": [
    {
      "downloader": "fastdl",
      "total_attempts": 10,
      "successful": 8,
      "failed": 2,
      "total_files": 45,
      "success_rate": 80.0,
      "last_success": "2025-11-19T06:00:00",
      "last_attempt": "2025-11-19T09:00:00"
    }
  ],
  "window_hours": 24
}

Get Monitoring History

GET /api/monitoring/history?downloader=fastdl&limit=100

Response:

{
  "success": true,
  "history": [
    {
      "id": 1,
      "downloader": "fastdl",
      "username": "evalongoria",
      "timestamp": "2025-11-19T09:00:00",
      "success": false,
      "file_count": 0,
      "error_message": "Cloudflare challenge",
      "alert_sent": true
    }
  ]
}

Clear Old Logs

DELETE /api/monitoring/history?days=30

Removes logs older than 30 days.

Database Schema

CREATE TABLE download_monitor (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    downloader TEXT NOT NULL,
    username TEXT,
    timestamp TEXT NOT NULL,
    success INTEGER NOT NULL,
    file_count INTEGER DEFAULT 0,
    error_message TEXT,
    alert_sent INTEGER DEFAULT 0,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_download_monitor_downloader ON download_monitor(downloader);
CREATE INDEX idx_download_monitor_timestamp ON download_monitor(timestamp);
CREATE INDEX idx_download_monitor_success ON download_monitor(success);

Module Architecture

Core Modules

modules/downloader_monitor.py

Main monitoring logic
Tracks download attempts
Checks for persistent failures
Sends Pushover alerts
Provides status queries
Cleans up old logs

modules/monitor_wrapper.py

Helper functions for integration
log_download_result() - Simple logging function
@monitor_download() - Decorator (future use)

Integration Points

Subprocess Wrappers:

wrappers/fastdl_subprocess_wrapper.py
wrappers/imginn_subprocess_wrapper.py
wrappers/toolzu_subprocess_wrapper.py
wrappers/snapchat_subprocess_wrapper.py

Each wrapper calls:

from modules.monitor_wrapper import log_download_result

# After download
log_download_result('fastdl', username, count, error=None)

# On failure
log_download_result('fastdl', username, 0, error=str(e))

Example Scenarios

Scenario 1: Temporary Failure

09:00 - fastdl: Failed (Cloudflare)
12:00 - fastdl: Success (5 files)

Result: No alert (recovered before threshold)

Scenario 2: Persistent Failure

09:00 - fastdl: Failed (Cloudflare)
12:00 - fastdl: Failed (Cloudflare)
15:00 - fastdl: Failed (Cloudflare)

Result: 🚨 Alert sent at 12:00 (2 consecutive failures within 3 hours)

Scenario 3: Multiple Downloaders

09:00 - fastdl: Success (3 files)
09:00 - toolzu: Failed (Rate limited)
12:00 - fastdl: Success (2 files)
12:00 - toolzu: Failed (Rate limited)

Result: 🚨 Alert for toolzu only (fastdl working fine)

Maintenance

View Current Status

sqlite3 /opt/media-downloader/database/media_downloader.db "
SELECT
    downloader,
    COUNT(*) as total,
    SUM(success) as successful,
    SUM(CASE WHEN success=0 THEN 1 ELSE 0 END) as failed
FROM download_monitor
WHERE timestamp > datetime('now', '-24 hours')
GROUP BY downloader;
"

Manual Cleanup

sqlite3 /opt/media-downloader/database/media_downloader.db "
DELETE FROM download_monitor
WHERE timestamp < datetime('now', '-30 days');
"

View Recent Failures

sqlite3 /opt/media-downloader/database/media_downloader.db "
SELECT downloader, username, timestamp, error_message
FROM download_monitor
WHERE success = 0
ORDER BY timestamp DESC
LIMIT 10;
"

Troubleshooting

No Alerts Being Sent

Check Pushover configuration:

SELECT value FROM settings WHERE key = 'pushover';

Verify monitoring is enabled:

SELECT value FROM settings WHERE key = 'monitoring';

Check logs:

grep -i "monitor\|alert" /opt/media-downloader/logs/*_api.log

Too Many Alerts

Increase thresholds:

UPDATE settings
SET value = json_set(value, '$.min_consecutive_failures', 5)
WHERE key = 'monitoring';

Disable Monitoring for Specific Downloader

UPDATE settings
SET value = json_set(value, '$.downloaders.fastdl', false)
WHERE key = 'monitoring';

Future Enhancements

Web UI dashboard for monitoring
Historical charts and graphs
Downloader performance metrics
Email notifications (in addition to Pushover)
Webhook support for custom integrations
Automatic remediation actions

Version History

v6.36.1 - Initial implementation

Database schema
Monitoring module
Pushover integration
API endpoints
Integration with all downloaders

7.7 KiB Raw Blame History