# Media Downloader - GUI Design & Implementation Plan

**Version:** 1.0
**Date:** October 25, 2025
**Status:** Planning Phase

---

## Table of Contents

1. [Executive Summary](#executive-summary)
2. [Current System Analysis](#current-system-analysis)
3. [GUI Architecture Options](#gui-architecture-options)
4. [Recommended Approach](#recommended-approach)
5. [Technology Stack](#technology-stack)
6. [Implementation Phases](#implementation-phases)
7. [Feature Roadmap](#feature-roadmap)
8. [API Specification](#api-specification)
9. [UI/UX Design](#uiux-design)
10. [Database Integration](#database-integration)
11. [Real-time Updates](#real-time-updates)
12. [Security Considerations](#security-considerations)
13. [Development Timeline](#development-timeline)

---

## Executive Summary

The Media Downloader GUI project aims to create a modern, user-friendly web interface for managing automated media downloads from multiple platforms (Instagram, TikTok, Snapchat, Forums). The GUI will be modeled after the proven **backup-central** architecture, using Node.js/Express backend with vanilla JavaScript frontend.

### Key Goals:
- **Maintain existing Python backend** - Preserve all battle-tested scraping logic
- **Modern web interface** - Real-time updates, responsive design, dark/light themes
- **Easy management** - Visual account configuration, manual triggers, scheduler control
- **Enterprise-grade** - Similar to backup-central's polished UI and reliability

---

## Current System Analysis

### Existing Architecture

```
media-downloader.py (Python Orchestrator)
├── Unified Database (SQLite with WAL mode)
│   ├── downloads table (1,183+ records)
│   ├── forum_threads, forum_posts
│   ├── scheduler_state, download_queue
│   └── File hash deduplication (NEW)
│
├── Platform Modules (16 modules)
│   ├── instaloader_module.py (Instagram via API)
│   ├── fastdl_module.py (Instagram web scraper)
│   ├── imginn_module.py (Instagram alternative)
│   ├── toolzu_module.py (High-res Instagram 1920x1440)
│   ├── snapchat_scraper.py (direct Playwright scraper)
│   ├── tiktok_module.py (yt-dlp wrapper)
│   └── forum_downloader.py (7 forum types)
│
├── Subprocess Wrappers (Playwright automation)
│   ├── fastdl_subprocess_wrapper.py
│   ├── imginn_subprocess_wrapper.py
│   ├── toolzu_subprocess_wrapper.py
│   ├── snapchat_subprocess_wrapper.py
│   └── forum_subprocess_wrapper.py
│
├── Support Systems
│   ├── scheduler.py (randomized intervals, persistent state)
│   ├── move_module.py (file operations + deduplication)
│   ├── pushover_notifier.py (push notifications)
│   ├── download_manager.py (multi-threaded downloads)
│   └── unified_database.py (connection pooling, WAL mode)
│
└── Configuration
    └── config/settings.json (100+ parameters)
```

### Current Capabilities

**Supported Platforms:**
- Instagram (4 methods: InstaLoader, FastDL, ImgInn, Toolzu)
- TikTok (via yt-dlp)
- Snapchat Stories
- Forums (XenForo, vBulletin, phpBB, Discourse, IPB, MyBB, SMF)

**Advanced Features:**
- Quality upgrade merging (FastDL + Toolzu)
- File hash deduplication (SHA256-based)
- Timestamp preservation (EXIF metadata)
- Randomized scheduler intervals
- Pushover notifications with thumbnails
- Immich photo library integration
- Cookie-based authentication
- 2captcha CAPTCHA solving
- Browser automation (Playwright)

**Statistics:**
- 19,100+ lines of production Python code
- 1,183+ downloads tracked
- 213 files with SHA256 hashes
- 30 duplicate groups detected
- 8 database tables with 17 indexes

---

## GUI Architecture Options

### Option 1: Hybrid Approach ⭐ **RECOMMENDED**

**Architecture:**
```
┌─────────────────────────────────────┐
│   Node.js Web GUI                   │
│   - Express.js API server           │
│   - Vanilla JS frontend             │
│   - Real-time WebSocket updates     │
│   - Chart.js analytics              │
└──────────────┬──────────────────────┘
               │ REST API + WebSocket
               ▼
┌─────────────────────────────────────┐
│   Existing Python Backend           │
│   - All platform downloaders        │
│   - Database layer                  │
│   - Scheduler                       │
│   - Browser automation              │
└─────────────────────────────────────┘
```

**Pros:**
✅ Preserves all battle-tested scraping logic
✅ Modern, responsive web UI
✅ Lower risk, faster development (4-8 weeks)
✅ Python ecosystem better for web scraping
✅ Can develop frontend and API simultaneously

**Cons:**
⚠️ Two codebases to maintain (Node.js + Python)
⚠️ Inter-process communication overhead

---

### Option 2: Full Node.js Rewrite

**Architecture:**
```
┌─────────────────────────────────────┐
│   Full Node.js/TypeScript Stack     │
│   - Express/Fastify API             │
│   - React/Next.js frontend          │
│   - Playwright Node.js bindings     │
│   - Prisma ORM                      │
└─────────────────────────────────────┘
```

**Pros:**
✅ Unified JavaScript/TypeScript codebase
✅ Modern tooling, better IDE support
✅ Easier for full-stack JS developers

**Cons:**
❌ 3-6 months minimum development time
❌ Need to reimplement all platform scraping
❌ Risk of losing subtle platform-specific fixes
❌ No instaloader equivalent in Node.js
❌ Complex authentication flows need rediscovery

**Verdict:** Only consider if planning long-term open-source project with JavaScript contributors.

---

### Option 3: Simple Dashboard (Quickest)

**Architecture:**
```
Node.js Dashboard (read-only)
├── Reads SQLite database directly
├── Displays stats, history, schedules
├── Tails Python logs
└── No control features (view-only)
```

**Timeline:** 1-2 weeks
**Use Case:** Quick visibility without control features

---

## Recommended Approach

### **Hybrid Architecture with Backup-Central Design Pattern**

After analyzing `/opt/backup-central`, we recommend adopting its proven architecture:

**Backend Stack:**
- Express.js (HTTP server)
- WebSocket (ws package) for real-time updates
- SQLite3 (reuse existing unified database)
- Winston (structured logging)
- node-cron (scheduler coordination)
- Helmet + Compression (security & performance)

**Frontend Stack:**
- **Vanilla JavaScript** (no React/Vue - faster, simpler)
- Chart.js (analytics visualizations)
- Font Awesome (icons)
- Inter font (modern typography)
- Mobile-responsive CSS
- Dark/Light theme support

**Why Backup-Central's Approach:**
1. Proven in production
2. Simple to understand and maintain
3. Fast loading (no framework overhead)
4. Real-time updates work flawlessly
5. Beautiful, modern UI without complexity

---

## Technology Stack

### Backend (Node.js)

```json
{
  "dependencies": {
    "express": "^4.18.2",
    "ws": "^8.14.2",
    "sqlite3": "^5.1.7",
    "winston": "^3.18.3",
    "node-cron": "^4.2.1",
    "compression": "^1.8.1",
    "helmet": "^8.1.0",
    "dotenv": "^17.2.3",
    "express-session": "^1.18.2",
    "jsonwebtoken": "^9.0.2"
  }
}
```

### Frontend (Vanilla JS)

```html
<!-- Libraries -->
<script src="chart.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.0/css/all.min.css">
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700">
```

### Python Integration

```javascript
// Subprocess execution for Python backend
const { spawn } = require('child_process');

function triggerDownload(platform, username) {
  return spawn('python3', [
    'media-downloader.py',
    '--platform', platform,
    '--username', username
  ]);
}
```

---

## Implementation Phases

### **Phase 1: Backend API Foundation** (Week 1-2)

**Deliverables:**
```
media-downloader-gui/
├── server.js (Express + WebSocket)
├── .env.example
├── package.json
└── lib/
    ├── db-helper.js (SQLite wrapper)
    ├── python-bridge.js (subprocess manager)
    ├── logger.js (Winston)
    └── api-v1/
        ├── downloads.js
        ├── accounts.js
        ├── stats.js
        ├── scheduler.js
        └── config.js
```

**API Endpoints:**
- `GET /api/downloads` - Query download history
- `GET /api/downloads/recent` - Last 100 downloads
- `POST /api/downloads/trigger` - Manual download trigger
- `GET /api/accounts` - List all configured accounts
- `POST /api/accounts` - Add new account
- `PUT /api/accounts/:id` - Update account
- `DELETE /api/accounts/:id` - Remove account
- `GET /api/stats` - Platform statistics
- `GET /api/scheduler/status` - Scheduler state
- `POST /api/scheduler/start` - Start scheduler
- `POST /api/scheduler/stop` - Stop scheduler
- `GET /api/config` - Read configuration
- `PUT /api/config` - Update configuration
- `GET /api/logs` - Tail Python logs
- `WS /api/live` - Real-time updates

---

### **Phase 2: Core Frontend UI** (Week 3-4)

**Dashboard Layout:**
```
┌─────────────────────────────────────────────────────┐
│  Header: Media Downloader | [Theme] [Profile] [⚙️]  │
├─────────────────────────────────────────────────────┤
│  Platform Cards                                     │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐            │
│  │Instagram │ │ TikTok   │ │ Snapchat │            │
│  │ 523 DL   │ │  87 DL   │ │  142 DL  │            │
│  │ ▶️ Trigger│ │ ▶️ Trigger│ │ ▶️ Trigger│            │
│  └──────────┘ └──────────┘ └──────────┘            │
├─────────────────────────────────────────────────────┤
│  Recent Downloads (Live Feed)                       │
│  🟢 evalongoria_20251025... (Instagram/evalongoria) │
│  🟢 20251025_TikTok... (TikTok/evalongoria)        │
│  ⚠️  Duplicate skipped: photo.jpg (hash match)     │
├─────────────────────────────────────────────────────┤
│  Statistics (Chart.js)                              │
│  📊 Downloads per Platform | 📈 Timeline Graph      │
└─────────────────────────────────────────────────────┘
```

**Components:**
1. **Dashboard** (`public/index.html`)
   - Platform overview cards
   - Live download feed (WebSocket)
   - Quick stats

2. **Accounts Manager** (`public/accounts.html`)
   - Add/Edit/Delete Instagram usernames
   - Add/Edit/Delete TikTok accounts
   - Add/Edit/Delete Forum configurations
   - Per-account interval settings

3. **Download History** (`public/history.html`)
   - Searchable table
   - Filter by platform/source/date
   - Thumbnail previews
   - Duplicate indicators

4. **Scheduler Control** (`public/scheduler.html`)
   - Enable/Disable scheduler
   - View next run times
   - Adjust global intervals
   - Force run specific tasks

5. **Configuration Editor** (`public/config.html`)
   - JSON editor with validation
   - Platform-specific settings
   - Notification configuration
   - Immich integration settings

6. **Logs Viewer** (`public/logs.html`)
   - Tail Python application logs
   - Filter by level (DEBUG/INFO/WARNING/ERROR)
   - Search functionality
   - Auto-scroll toggle

---

### **Phase 3: Advanced Features** (Week 5-6)

**Real-time Features:**
```javascript
// WebSocket message types
{
  type: 'download_start',
  platform: 'instagram',
  username: 'evalongoria',
  content_type: 'story'
}

{
  type: 'download_complete',
  platform: 'instagram',
  filename: 'evalongoria_20251025_123456.jpg',
  file_size: 245678,
  duplicate: false
}

{
  type: 'duplicate_detected',
  filename: 'photo.jpg',
  existing_file: 'photo_original.jpg',
  platform: 'instagram'
}

{
  type: 'scheduler_update',
  task_id: 'instagram:evalongoria',
  next_run: '2025-10-25T23:00:00Z'
}
```

**Features:**
- Live download progress bars
- Duplicate detection alerts
- Scheduler countdown timers
- Platform health indicators
- Download speed metrics

---

### **Phase 4: Polish & Deploy** (Week 7-8)

**Final Touches:**
- Mobile-responsive design
- Dark mode implementation
- Keyboard shortcuts
- Toast notifications (success/error)
- Loading skeletons
- Error boundary handling
- Performance optimization
- Security hardening
- Documentation
- Deployment scripts

---

## Feature Roadmap

### **MVP Features** (Phase 1-2)

✅ View download history
✅ See platform statistics
✅ Manual download triggers
✅ Account management (CRUD)
✅ Real-time download feed
✅ Dark/Light theme
✅ Mobile responsive

### **Enhanced Features** (Phase 3)

🔄 Scheduler control (start/stop/adjust)
🔄 Configuration editor
🔄 Logs viewer
🔄 Advanced search/filtering
🔄 Duplicate management UI
🔄 Download queue management

### **Future Features** (Phase 4+)

📋 Batch operations (delete/retry multiple)
📋 Download rules engine (auto-skip based on criteria)
📋 Analytics dashboard (trends, insights)
📋 Export/Import configurations
📋 Webhook integrations
📋 Multi-user support with authentication
📋 API key management
📋 Browser screenshot viewer (see Playwright automation)
📋 Cookie editor (manage authentication)

---

## API Specification

### REST API Endpoints

#### Downloads

**GET /api/downloads**
```javascript
// Query downloads with filters
GET /api/downloads?platform=instagram&limit=50&offset=0

Response:
{
  "total": 1183,
  "downloads": [
    {
      "id": 1,
      "url": "https://...",
      "url_hash": "sha256...",
      "platform": "instagram",
      "source": "evalongoria",
      "content_type": "story",
      "filename": "evalongoria_20251025_123456.jpg",
      "file_path": "/opt/immich/md/social media/instagram/...",
      "file_size": 245678,
      "file_hash": "sha256...",
      "post_date": "2025-10-25T12:34:56Z",
      "download_date": "2025-10-25T12:35:00Z",
      "status": "completed",
      "metadata": {}
    }
  ]
}
```

**POST /api/downloads/trigger**
```javascript
// Trigger manual download
POST /api/downloads/trigger
{
  "platform": "instagram",
  "username": "evalongoria",
  "content_types": ["stories", "posts"]
}

Response:
{
  "status": "started",
  "job_id": "instagram_evalongoria_1729900000",
  "message": "Download started in background"
}
```

#### Accounts

**GET /api/accounts**
```javascript
GET /api/accounts?platform=instagram

Response:
{
  "instagram": [
    {
      "username": "evalongoria",
      "enabled": true,
      "check_interval_hours": 6,
      "content_types": {
        "posts": true,
        "stories": true,
        "reels": false
      }
    }
  ],
  "tiktok": [...],
  "snapchat": [...]
}
```

**POST /api/accounts**
```javascript
POST /api/accounts
{
  "platform": "instagram",
  "username": "newuser",
  "check_interval_hours": 12,
  "content_types": {
    "posts": true,
    "stories": false
  }
}

Response:
{
  "success": true,
  "account": { ... }
}
```

#### Statistics

**GET /api/stats**
```javascript
GET /api/stats

Response:
{
  "platforms": {
    "instagram": {
      "total": 523,
      "completed": 520,
      "failed": 3,
      "duplicates": 15,
      "total_size": 1234567890
    },
    "tiktok": { ... },
    "snapchat": { ... }
  },
  "recent_activity": {
    "last_24h": 45,
    "last_7d": 312
  }
}
```

#### Scheduler

**GET /api/scheduler/status**
```javascript
GET /api/scheduler/status

Response:
{
  "running": true,
  "tasks": [
    {
      "task_id": "instagram:evalongoria",
      "last_run": "2025-10-25T12:00:00Z",
      "next_run": "2025-10-25T18:00:00Z",
      "interval_hours": 6,
      "status": "active"
    }
  ]
}
```

#### Configuration

**GET /api/config**
```javascript
GET /api/config

Response:
{
  "instagram": { ... },
  "tiktok": { ... },
  "pushover": { ... },
  "immich": { ... }
}
```

**PUT /api/config**
```javascript
PUT /api/config
{
  "instagram": {
    "enabled": true,
    "check_interval_hours": 8
  }
}

Response:
{
  "success": true,
  "config": { ... }
}
```

### WebSocket Events

**Client → Server:**
```javascript
// Subscribe to live updates
{
  "action": "subscribe",
  "channels": ["downloads", "scheduler", "duplicates"]
}
```

**Server → Client:**
```javascript
// Download started
{
  "type": "download_start",
  "timestamp": "2025-10-25T12:34:56Z",
  "platform": "instagram",
  "username": "evalongoria"
}

// Download completed
{
  "type": "download_complete",
  "timestamp": "2025-10-25T12:35:00Z",
  "platform": "instagram",
  "filename": "evalongoria_20251025_123456.jpg",
  "file_size": 245678,
  "duplicate": false
}

// Duplicate detected
{
  "type": "duplicate_detected",
  "timestamp": "2025-10-25T12:35:05Z",
  "filename": "photo.jpg",
  "existing_file": {
    "filename": "photo_original.jpg",
    "platform": "instagram",
    "source": "evalongoria"
  }
}
```

---

## UI/UX Design

### Design System (Inspired by Backup-Central)

**Colors:**
```css
:root {
  /* Light Theme */
  --primary-color: #2563eb;
  --secondary-color: #64748b;
  --success-color: #10b981;
  --warning-color: #f59e0b;
  --error-color: #ef4444;
  --bg-color: #f8fafc;
  --card-bg: #ffffff;
  --text-color: #1e293b;
  --border-color: #e2e8f0;
}

[data-theme="dark"] {
  /* Dark Theme */
  --primary-color: #3b82f6;
  --bg-color: #0f172a;
  --card-bg: #1e293b;
  --text-color: #f1f5f9;
  --border-color: #334155;
}
```

**Typography:**
```css
font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
```

**Components:**
- Cards with subtle shadows
- Rounded corners (8px border-radius)
- Smooth transitions (0.3s ease)
- Gradient accents on hover
- Loading skeletons
- Toast notifications (top-right)

---

## Database Integration

### Database Access Strategy

**Read Operations (Node.js):**
```javascript
// Direct SQLite reads for fast queries
const db = require('better-sqlite3')('/opt/media-downloader/database/media_downloader.db');

const downloads = db.prepare(`
  SELECT * FROM downloads
  WHERE platform = ?
  ORDER BY download_date DESC
  LIMIT ?
`).all('instagram', 50);
```

**Write Operations (Python):**
```javascript
// Route through Python backend for consistency
const { spawn } = require('child_process');

function addAccount(platform, username) {
  // Update config.json
  // Trigger Python process to reload config
}
```

**Why This Approach:**
- Python maintains database writes (consistency)
- Node.js reads for fast UI queries
- No duplicate database logic
- Leverages existing connection pooling

---

## Real-time Updates

### WebSocket Architecture

**Server-Side (Node.js):**
```javascript
const WebSocket = require('ws');
const wss = new WebSocket.Server({ server });

// Broadcast to all connected clients
function broadcast(message) {
  wss.clients.forEach(client => {
    if (client.readyState === WebSocket.OPEN) {
      client.send(JSON.stringify(message));
    }
  });
}

// Watch Python logs for events
const { spawn } = require('child_process');
const pythonProcess = spawn('python3', ['media-downloader.py', '--daemon']);

pythonProcess.stdout.on('data', (data) => {
  // Parse log output and broadcast events
  const event = parseLogEvent(data.toString());
  if (event) broadcast(event);
});
```

**Client-Side (JavaScript):**
```javascript
const ws = new WebSocket('ws://localhost:3000/api/live');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);

  switch(data.type) {
    case 'download_complete':
      addToDownloadFeed(data);
      updateStats();
      showToast(`Downloaded ${data.filename}`, 'success');
      break;

    case 'duplicate_detected':
      showToast(`Duplicate skipped: ${data.filename}`, 'warning');
      break;
  }
};
```

---

## Security Considerations

### Authentication (Optional for Single-User)

**Simple Auth:**
- Environment variable password
- Session-based auth (express-session)
- No registration needed

**Enhanced Auth (Future):**
- TOTP/2FA (speakeasy)
- Passkeys (WebAuthn)
- JWT tokens
- Per-user configurations

### API Security

```javascript
// Helmet for security headers
app.use(helmet());

// CORS configuration
app.use(cors({
  origin: process.env.ALLOWED_ORIGINS?.split(',') || '*',
  credentials: true
}));

// Rate limiting
const rateLimit = require('express-rate-limit');
const limiter = rateLimit({
  windowMs: 15 * 60 * 1000, // 15 minutes
  max: 100 // limit each IP to 100 requests per windowMs
});
app.use('/api/', limiter);
```

### Environment Variables

```bash
# .env
NODE_ENV=production
PORT=3000
SESSION_SECRET=random_secret_key
PYTHON_PATH=/opt/media-downloader/venv/bin/python3
DATABASE_PATH=/opt/media-downloader/database/media_downloader.db
CONFIG_PATH=/opt/media-downloader/config/settings.json
```

---

## Development Timeline

### **Estimated Timeline: 8 Weeks**

**Week 1-2: Backend API**
- Express server setup
- Database integration
- Python subprocess bridge
- Basic API endpoints
- WebSocket setup

**Week 3-4: Core Frontend**
- Dashboard layout
- Platform cards
- Download feed
- Account management UI
- Basic stats

**Week 5-6: Advanced Features**
- Real-time updates
- Scheduler control
- Config editor
- Logs viewer
- Search/filtering

**Week 7-8: Polish**
- Mobile responsive
- Dark mode
- Error handling
- Testing
- Documentation
- Deployment

---

## Next Steps

### Immediate Actions:

1. **✅ File Hash Deduplication** - COMPLETED
   - Added SHA256 hashing to unified_database.py
   - Implemented automatic duplicate detection in move_module.py
   - Created utilities for backfilling and managing hashes
   - Scanned 213 existing files and found 30 duplicate groups

2. **✅ Directory Cleanup** - COMPLETED
   - Moved test files to `tests/` directory
   - Moved one-time scripts to `archive/`
   - Organized utilities in `utilities/` directory
   - Removed obsolete documentation

3. **📋 Begin GUI Development**
   - Initialize Node.js project
   - Set up Express server
   - Create basic API endpoints
   - Build dashboard prototype

---

## References

- **Backup-Central:** `/opt/backup-central` - Reference implementation
- **Python Backend:** `/opt/media-downloader/media-downloader.py`
- **Database Schema:** `/opt/media-downloader/modules/unified_database.py`
- **Existing Docs:** `/opt/media-downloader/archive/` (old GUI plans)

---

## Appendix

### Directory Structure After Cleanup

```
/opt/media-downloader/
├── media-downloader.py (main application)
├── setup.py (installation script)
├── INSTALL.md (installation guide)
├── GUI_DESIGN_PLAN.md (this document)
├── requirements.txt
├── config/
│   └── settings.json
├── database/
│   ├── media_downloader.db
│   └── scheduler_state.db
├── modules/ (16 Python modules)
│   ├── unified_database.py
│   ├── scheduler.py
│   ├── move_module.py
│   ├── instaloader_module.py
│   ├── fastdl_module.py
│   ├── imginn_module.py
│   ├── toolzu_module.py
│   ├── snapchat_module.py
│   ├── tiktok_module.py
│   ├── forum_downloader.py
│   └── ... (10 more modules)
├── utilities/
│   ├── backfill_file_hashes.py
│   ├── cleanup_database_filenames.py
│   └── scan_and_hash_files.py
├── archive/ (old docs, one-time scripts)
│   ├── HIGH_RES_DOWNLOAD.md
│   ├── SNAPCHAT_*.md
│   ├── TOOLZU-TIMESTAMPS.md
│   ├── WEB_GUI_*.md (4 old GUI docs)
│   ├── cleanup_last_week.py
│   ├── merge-quality-upgrade.py
│   ├── reset_database.py
│   └── debug_snapchat.py
├── tests/ (7 test scripts)
│   ├── test_all_notifications.py
│   ├── test_pushover.py
│   └── ... (5 more tests)
├── subprocess wrappers/ (5 wrappers)
│   ├── fastdl_subprocess_wrapper.py
│   ├── imginn_subprocess_wrapper.py
│   ├── toolzu_subprocess_wrapper.py
│   ├── snapchat_subprocess_wrapper.py
│   └── forum_subprocess_wrapper.py
├── venv/ (Python virtual environment)
├── logs/ (application logs)
├── temp/ (temporary download directories)
└── ... (other directories)
```

---

**End of Document**

For questions or updates, refer to this document as the single source of truth for GUI development planning.