# Downloader Monitoring System

## Overview

The Downloader Monitoring System tracks the health of all downloader modules and sends push notifications when a downloader has been consistently failing for a specified time period (default: 3 hours).

## Features

✅ **Per-Downloader Tracking** - Monitors each downloader independently:
- fastdl (Instagram web scraper)
- imginn (Instagram alternative scraper)
- toolzu (Instagram high-res scraper)
- instagram (Instaloader API)
- snapchat (Direct Playwright scraper)
- tiktok (yt-dlp)
- forums (XenForo/vBulletin scrapers)
- coppermine (Coppermine Photo Gallery scraper)

✅ **Smart Alerting** - Only alerts once per issue (no spam)

✅ **Pushover Notifications** - Sends high-priority push notifications

✅ **Configurable Thresholds** - Customize failure windows and minimum failures

✅ **Automatic Cleanup** - Removes old monitoring logs automatically

## How It Works

### 1. Download Tracking
Every download attempt is logged to the `download_monitor` table:
```sql
INSERT INTO download_monitor (
    downloader,      -- 'fastdl', 'snapchat', etc.
    username,        -- User being downloaded
    timestamp,       -- When the attempt occurred
    success,         -- 1 = success, 0 = failure
    file_count,      -- Number of files downloaded
    error_message,   -- Error details if failed
    alert_sent       -- Whether alert was sent
)
```

### 2. Failure Detection
When a download fails, the system:
1. Checks the last N attempts within the time window
2. Counts consecutive failures
3. If failures ≥ threshold → Send alert
4. Marks the failure as alerted (prevents duplicate notifications)

### 3. Push Notifications
Alert format:
```
🚨 FastDL Failing

Downloader has been failing for 3+ hours

Username: evalongoria
Consecutive Failures: 3
Last Success: 6 hours ago
Latest Error: "Cloudflare challenge"

Check logs for details.
```

## Configuration

### Database Settings
Configuration is stored in the `settings` table:
```json
{
  "enabled": true,
  "failure_window_hours": 3,
  "min_consecutive_failures": 2,
  "pushover": {
    "enabled": true,
    "priority": 1
  },
  "downloaders": {
    "fastdl": true,
    "imginn": true,
    "toolzu": true,
    "instagram": true,
    "snapchat": true,
    "tiktok": true,
    "forums": true
  }
}
```

### Configuration Options

| Option | Default | Description |
|--------|---------|-------------|
| `enabled` | `true` | Enable/disable monitoring system |
| `failure_window_hours` | `3` | How many hours to look back |
| `min_consecutive_failures` | `2` | Minimum failures to trigger alert |
| `pushover.enabled` | `true` | Enable Pushover notifications |
| `pushover.priority` | `1` | Notification priority (1 = high) |
| `downloaders.*` | `true` | Enable/disable per-downloader monitoring |

### Updating Configuration
Via Web UI (coming soon) or database:
```sql
UPDATE settings
SET value = json_set(value, '$.failure_window_hours', 6)
WHERE key = 'monitoring';
```

## API Endpoints

### Get Monitoring Status
```http
GET /api/monitoring/status?hours=24
```

**Response:**
```json
{
  "success": true,
  "downloaders": [
    {
      "downloader": "fastdl",
      "total_attempts": 10,
      "successful": 8,
      "failed": 2,
      "total_files": 45,
      "success_rate": 80.0,
      "last_success": "2025-11-19T06:00:00",
      "last_attempt": "2025-11-19T09:00:00"
    }
  ],
  "window_hours": 24
}
```

### Get Monitoring History
```http
GET /api/monitoring/history?downloader=fastdl&limit=100
```

**Response:**
```json
{
  "success": true,
  "history": [
    {
      "id": 1,
      "downloader": "fastdl",
      "username": "evalongoria",
      "timestamp": "2025-11-19T09:00:00",
      "success": false,
      "file_count": 0,
      "error_message": "Cloudflare challenge",
      "alert_sent": true
    }
  ]
}
```

### Clear Old Logs
```http
DELETE /api/monitoring/history?days=30
```

Removes logs older than 30 days.

## Database Schema

```sql
CREATE TABLE download_monitor (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    downloader TEXT NOT NULL,
    username TEXT,
    timestamp TEXT NOT NULL,
    success INTEGER NOT NULL,
    file_count INTEGER DEFAULT 0,
    error_message TEXT,
    alert_sent INTEGER DEFAULT 0,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_download_monitor_downloader ON download_monitor(downloader);
CREATE INDEX idx_download_monitor_timestamp ON download_monitor(timestamp);
CREATE INDEX idx_download_monitor_success ON download_monitor(success);
```

## Module Architecture

### Core Modules

**`modules/downloader_monitor.py`**
- Main monitoring logic
- Tracks download attempts
- Checks for persistent failures
- Sends Pushover alerts
- Provides status queries
- Cleans up old logs

**`modules/monitor_wrapper.py`**
- Helper functions for integration
- `log_download_result()` - Simple logging function
- `@monitor_download()` - Decorator (future use)

### Integration Points

**Subprocess Wrappers:**
- `wrappers/fastdl_subprocess_wrapper.py`
- `wrappers/imginn_subprocess_wrapper.py`
- `wrappers/toolzu_subprocess_wrapper.py`
- `wrappers/snapchat_subprocess_wrapper.py`

Each wrapper calls:
```python
from modules.monitor_wrapper import log_download_result

# After download
log_download_result('fastdl', username, count, error=None)

# On failure
log_download_result('fastdl', username, 0, error=str(e))
```

## Example Scenarios

### Scenario 1: Temporary Failure
```
09:00 - fastdl: Failed (Cloudflare)
12:00 - fastdl: Success (5 files)
```
**Result:** No alert (recovered before threshold)

### Scenario 2: Persistent Failure
```
09:00 - fastdl: Failed (Cloudflare)
12:00 - fastdl: Failed (Cloudflare)
15:00 - fastdl: Failed (Cloudflare)
```
**Result:** 🚨 Alert sent at 12:00 (2 consecutive failures within 3 hours)

### Scenario 3: Multiple Downloaders
```
09:00 - fastdl: Success (3 files)
09:00 - toolzu: Failed (Rate limited)
12:00 - fastdl: Success (2 files)
12:00 - toolzu: Failed (Rate limited)
```
**Result:** 🚨 Alert for toolzu only (fastdl working fine)

## Maintenance

### View Current Status
```bash
sqlite3 /opt/media-downloader/database/media_downloader.db "
SELECT
    downloader,
    COUNT(*) as total,
    SUM(success) as successful,
    SUM(CASE WHEN success=0 THEN 1 ELSE 0 END) as failed
FROM download_monitor
WHERE timestamp > datetime('now', '-24 hours')
GROUP BY downloader;
"
```

### Manual Cleanup
```bash
sqlite3 /opt/media-downloader/database/media_downloader.db "
DELETE FROM download_monitor
WHERE timestamp < datetime('now', '-30 days');
"
```

### View Recent Failures
```bash
sqlite3 /opt/media-downloader/database/media_downloader.db "
SELECT downloader, username, timestamp, error_message
FROM download_monitor
WHERE success = 0
ORDER BY timestamp DESC
LIMIT 10;
"
```

## Troubleshooting

### No Alerts Being Sent
1. Check Pushover configuration:
   ```sql
   SELECT value FROM settings WHERE key = 'pushover';
   ```
2. Verify monitoring is enabled:
   ```sql
   SELECT value FROM settings WHERE key = 'monitoring';
   ```
3. Check logs:
   ```bash
   grep -i "monitor\|alert" /opt/media-downloader/logs/*_api.log
   ```

### Too Many Alerts
Increase thresholds:
```sql
UPDATE settings
SET value = json_set(value, '$.min_consecutive_failures', 5)
WHERE key = 'monitoring';
```

### Disable Monitoring for Specific Downloader
```sql
UPDATE settings
SET value = json_set(value, '$.downloaders.fastdl', false)
WHERE key = 'monitoring';
```

## Future Enhancements

- [ ] Web UI dashboard for monitoring
- [ ] Historical charts and graphs
- [ ] Downloader performance metrics
- [ ] Email notifications (in addition to Pushover)
- [ ] Webhook support for custom integrations
- [ ] Automatic remediation actions

## Version History

**v6.36.1** - Initial implementation
- Database schema
- Monitoring module
- Pushover integration
- API endpoints
- Integration with all downloaders