Files

Todd 0d7b2b1aab Initial commit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-29 22:42:55 -04:00

36 KiB

Raw Blame History

Media Downloader Web GUI - Full Development Plan

Executive Summary

Transform the CLI media downloader into a modern web application with real-time monitoring, queue management, and intuitive configuration. Preserve all existing functionality while adding professional-grade UI/UX.

1. Technology Stack

Backend

Framework: FastAPI 0.104+ (async, auto-docs, WebSocket support)
Task Queue: Celery + Redis (background job processing)
Database: SQLite (existing) + Redis (cache/sessions)
WebSocket: FastAPI native WebSocket for real-time updates
Authentication: JWT tokens + secure session management
API Documentation: Auto-generated Swagger/OpenAPI

Frontend

Framework: Vue.js 3 with Composition API
UI Library: Vuetify 3 (Material Design components)
State Management: Pinia (Vue's official state manager)
HTTP Client: Axios
Real-time: WebSocket with auto-reconnect
Build Tool: Vite (fast, modern)
Charts: Chart.js or Apache ECharts

DevOps

Containerization: Docker + Docker Compose
Process Manager: Supervisor (for workers)
Reverse Proxy: Nginx (production)
Logging: Structured JSON logs + Log rotation

2. System Architecture

┌─────────────────────────────────────────────────────────┐
│                     Web Browser                         │
│  ┌──────────────────────────────────────────────────┐   │
│  │   Vue.js Frontend (Port 8080)                    │   │
│  │  - Dashboard    - Queue Manager                  │   │
│  │  - Scheduler    - Settings                       │   │
│  │  - Logs         - History                        │   │
│  └──────────────────────────────────────────────────┘   │
└─────────────────┬───────────────────────────────────────┘
                  │ HTTP/WebSocket
                  ▼
┌─────────────────────────────────────────────────────────┐
│              FastAPI Backend (Port 8000)                │
│  ┌──────────────────────────────────────────────────┐   │
│  │  REST API Endpoints                              │   │
│  │  /api/v1/downloads  /api/v1/queue               │   │
│  │  /api/v1/scheduler  /api/v1/settings            │   │
│  │  /api/v1/platforms  /api/v1/logs                │   │
│  └──────────────────────────────────────────────────┘   │
│  ┌──────────────────────────────────────────────────┐   │
│  │  WebSocket Endpoints                             │   │
│  │  /ws/downloads (real-time progress)              │   │
│  │  /ws/logs      (live log streaming)              │   │
│  └──────────────────────────────────────────────────┘   │
└─────────┬───────────────────────────────────┬───────────┘
          │                                   │
          ▼                                   ▼
┌──────────────────────┐         ┌────────────────────────┐
│   SQLite Database    │         │   Redis (Cache/Queue)  │
│  media_downloader.db │         │  - Task queue          │
│  - Downloads         │         │  - Session store       │
│  - Queue             │         │  - Real-time events    │
│  - Scheduler         │         │  - Rate limit tracking │
└──────────────────────┘         └────────────────────────┘
          ▲                                   ▲
          │                                   │
          └───────────┬───────────────────────┘
                      │
        ┌─────────────▼──────────────────────────────┐
        │      Celery Workers (Background Tasks)      │
        │  ┌─────────────────────────────────────┐   │
        │  │  Worker 1: Instagram Downloads      │   │
        │  │  Worker 2: TikTok Downloads         │   │
        │  │  Worker 3: Forum Downloads          │   │
        │  │  Worker 4: Scheduler Tasks          │   │
        │  └─────────────────────────────────────┘   │
        │  Uses existing modules:                     │
        │  - fastdl_module.py                         │
        │  - toolzu_module.py                         │
        │  - tiktok_module.py                         │
        │  - forum_downloader.py                      │
        └─────────────────────────────────────────────┘

3. Project Structure

/opt/media-downloader/
├── backend/
│   ├── api/
│   │   ├── __init__.py
│   │   ├── main.py                 # FastAPI app entry point
│   │   ├── dependencies.py         # Dependency injection
│   │   ├── auth.py                 # JWT authentication
│   │   └── routes/
│   │       ├── __init__.py
│   │       ├── downloads.py        # Download CRUD endpoints
│   │       ├── queue.py            # Queue management
│   │       ├── scheduler.py        # Scheduler config
│   │       ├── platforms.py        # Platform-specific endpoints
│   │       ├── settings.py         # App configuration
│   │       ├── logs.py             # Log viewer
│   │       ├── stats.py            # Statistics/analytics
│   │       └── websocket.py        # WebSocket handlers
│   ├── models/
│   │   ├── __init__.py
│   │   ├── download.py             # Pydantic models
│   │   ├── queue.py
│   │   ├── scheduler.py
│   │   └── user.py
│   ├── services/
│   │   ├── __init__.py
│   │   ├── download_service.py     # Business logic
│   │   ├── queue_service.py
│   │   ├── scheduler_service.py
│   │   └── platform_service.py
│   ├── workers/
│   │   ├── __init__.py
│   │   ├── celery_app.py           # Celery configuration
│   │   ├── download_worker.py      # Background download tasks
│   │   └── scheduler_worker.py     # Scheduled task executor
│   ├── core/
│   │   ├── __init__.py
│   │   ├── config.py               # App configuration
│   │   ├── database.py             # DB connection management
│   │   └── websocket_manager.py    # WebSocket connection pool
│   └── requirements.txt
│
├── frontend/
│   ├── public/
│   │   ├── index.html
│   │   └── favicon.ico
│   ├── src/
│   │   ├── main.js                 # Vue app entry
│   │   ├── App.vue                 # Root component
│   │   ├── router/
│   │   │   └── index.js            # Vue Router config
│   │   ├── stores/
│   │   │   ├── downloads.js        # Pinia store
│   │   │   ├── queue.js
│   │   │   ├── scheduler.js
│   │   │   └── auth.js
│   │   ├── views/
│   │   │   ├── Dashboard.vue       # Main dashboard
│   │   │   ├── QueueManager.vue    # Queue UI
│   │   │   ├── SchedulerView.vue   # Scheduler config
│   │   │   ├── Settings.vue        # App settings
│   │   │   ├── Logs.vue            # Log viewer
│   │   │   ├── History.vue         # Download history
│   │   │   └── Login.vue           # Authentication
│   │   ├── components/
│   │   │   ├── DownloadCard.vue    # Download item display
│   │   │   ├── QueueItem.vue       # Queue item
│   │   │   ├── ProgressBar.vue     # Progress indicator
│   │   │   ├── PlatformSelector.vue
│   │   │   ├── LogViewer.vue
│   │   │   ├── StatCard.vue
│   │   │   └── ScheduleEditor.vue
│   │   ├── services/
│   │   │   ├── api.js              # Axios HTTP client
│   │   │   └── websocket.js        # WebSocket client
│   │   └── utils/
│   │       ├── formatters.js       # Date/size formatters
│   │       └── validators.js       # Input validation
│   ├── package.json
│   └── vite.config.js
│
├── docker/
│   ├── Dockerfile.backend
│   ├── Dockerfile.frontend
│   ├── Dockerfile.worker
│   └── nginx.conf
│
├── docker-compose.yml              # Full stack orchestration
├── docker-compose.dev.yml          # Development overrides
│
├── modules/                        # Existing downloader modules (kept as-is)
│   ├── fastdl_module.py
│   ├── toolzu_module.py
│   ├── tiktok_module.py
│   ├── forum_downloader.py
│   ├── unified_database.py
│   └── scheduler.py
│
├── database/
│   └── media_downloader.db         # Existing SQLite DB
│
├── downloads/                      # Downloaded media files
├── logs/                           # Application logs
└── config/
    └── settings.json               # App configuration

4. Database Schema Enhancements

Add new tables to existing media_downloader.db:

-- User management (for authentication)
CREATE TABLE users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    username TEXT UNIQUE NOT NULL,
    email TEXT UNIQUE NOT NULL,
    password_hash TEXT NOT NULL,
    role TEXT DEFAULT 'user',  -- 'admin', 'user'
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    last_login DATETIME,
    active BOOLEAN DEFAULT 1
);

-- API tokens
CREATE TABLE api_tokens (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    user_id INTEGER NOT NULL,
    token TEXT UNIQUE NOT NULL,
    name TEXT,
    expires_at DATETIME,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    last_used DATETIME,
    FOREIGN KEY (user_id) REFERENCES users(id)
);

-- Activity log
CREATE TABLE activity_log (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    user_id INTEGER,
    action TEXT NOT NULL,  -- 'download_started', 'queue_modified', etc.
    resource_type TEXT,    -- 'download', 'queue', 'scheduler'
    resource_id TEXT,
    details TEXT,          -- JSON
    ip_address TEXT,
    timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (user_id) REFERENCES users(id)
);

-- Notification settings
CREATE TABLE notifications (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    user_id INTEGER NOT NULL,
    type TEXT NOT NULL,    -- 'email', 'browser', 'webhook'
    events TEXT NOT NULL,  -- JSON array of events to notify
    config TEXT,           -- JSON config (email address, webhook URL)
    enabled BOOLEAN DEFAULT 1,
    FOREIGN KEY (user_id) REFERENCES users(id)
);

-- System settings (key-value store)
CREATE TABLE system_settings (
    key TEXT PRIMARY KEY,
    value TEXT NOT NULL,   -- JSON
    updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

-- Enhance existing download_queue table
ALTER TABLE download_queue ADD COLUMN assigned_worker TEXT;
ALTER TABLE download_queue ADD COLUMN started_at DATETIME;
ALTER TABLE download_queue ADD COLUMN progress INTEGER DEFAULT 0;  -- 0-100
ALTER TABLE download_queue ADD COLUMN estimated_completion DATETIME;

5. API Endpoints Design

Authentication

POST   /api/v1/auth/login           # Login with username/password
POST   /api/v1/auth/logout          # Logout
POST   /api/v1/auth/refresh         # Refresh JWT token
GET    /api/v1/auth/me              # Get current user

Downloads

GET    /api/v1/downloads            # List downloads (with filters)
GET    /api/v1/downloads/{id}       # Get download details
POST   /api/v1/downloads            # Manually add download
DELETE /api/v1/downloads/{id}       # Delete download record
GET    /api/v1/downloads/stats      # Get download statistics
POST   /api/v1/downloads/bulk       # Bulk operations (delete, retry)

Queue Management

GET    /api/v1/queue                # Get download queue
POST   /api/v1/queue                # Add to queue
DELETE /api/v1/queue/{id}           # Remove from queue
PATCH  /api/v1/queue/{id}           # Update queue item (priority, pause)
POST   /api/v1/queue/{id}/retry     # Retry failed download
POST   /api/v1/queue/clear          # Clear completed/failed
GET    /api/v1/queue/stats          # Queue statistics

Scheduler

GET    /api/v1/scheduler/jobs       # List scheduled jobs
POST   /api/v1/scheduler/jobs       # Create new job
GET    /api/v1/scheduler/jobs/{id}  # Get job details
PATCH  /api/v1/scheduler/jobs/{id}  # Update job
DELETE /api/v1/scheduler/jobs/{id}  # Delete job
POST   /api/v1/scheduler/jobs/{id}/run  # Trigger job manually
GET    /api/v1/scheduler/history    # Job execution history

Platforms

GET    /api/v1/platforms            # List all platforms (Instagram, TikTok, etc.)
GET    /api/v1/platforms/{name}     # Get platform details
POST   /api/v1/platforms/{name}/test  # Test platform connection
POST   /api/v1/platforms/{name}/download  # Start platform download

# Platform-specific endpoints
POST   /api/v1/platforms/instagram/posts     # Download Instagram posts
POST   /api/v1/platforms/instagram/stories   # Download Instagram stories
POST   /api/v1/platforms/tiktok/user         # Download TikTok user videos
POST   /api/v1/platforms/forum/thread        # Monitor forum thread

Settings

GET    /api/v1/settings             # Get all settings
PATCH  /api/v1/settings             # Update settings
GET    /api/v1/settings/platforms   # Get platform configs
PATCH  /api/v1/settings/platforms/{name}  # Update platform config

Logs

GET    /api/v1/logs                 # Get logs (paginated, filtered)
GET    /api/v1/logs/download        # Download log file
DELETE /api/v1/logs                 # Clear old logs

Statistics

GET    /api/v1/stats/overview       # Dashboard overview
GET    /api/v1/stats/timeline       # Downloads over time
GET    /api/v1/stats/platforms      # Breakdown by platform
GET    /api/v1/stats/sources        # Breakdown by source (username)
GET    /api/v1/stats/storage        # Storage usage

WebSocket

WS     /ws/downloads                # Real-time download updates
WS     /ws/queue                    # Real-time queue updates
WS     /ws/logs                     # Live log streaming

6. Frontend Views & Features

6.1 Dashboard View

Purpose: At-a-glance overview of system status

Components:

Stats Cards
- Total downloads today/week/month
- Queue size (pending/active/failed)
- Success rate percentage
- Storage used/available
- Next scheduled task
Recent Activity
- Live feed of downloads (last 20)
- Real-time progress bars
- Success/failure indicators
Platform Status
- Instagram, TikTok, Forums
- Green/yellow/red status indicators
- Last successful download time
- Error counts
Quick Actions
- Add to queue button
- Pause/Resume all
- Run scheduler manually

6.2 Queue Manager View

Purpose: Manage download queue with full control

Features:

Queue List
- Filterable by status (pending/active/completed/failed)
- Sortable by priority, date, platform
- Bulk selection checkboxes
- Search by URL/username
Queue Item Cards
- Platform icon + source username
- URL preview
- Priority indicator (High/Medium/Low)
- Status badge
- Progress bar (for active downloads)
- Action buttons:
  - Retry (failed items)
  - Pause/Resume
  - Move up/down (priority)
  - Delete
  - View details
Bulk Actions
- Clear completed
- Retry all failed
- Delete selected
- Change priority
Add to Queue Form
- Platform dropdown
- URL input (with validation)
- Content type selector
- Priority selector
- Schedule for later checkbox

6.3 Scheduler View

Purpose: Configure automated download schedules

Features:

Schedule List
- Active/Paused schedules
- Next run time
- Last run status
- Edit/Delete/Duplicate buttons
Schedule Editor
- Name input
- Platform selector
- Content type (posts/stories/reels/videos)
- Username/source input
- Cron expression editor (with visual helper)
- Frequency presets (Hourly, Daily, Weekly, Custom)
- Date range limits
- Download limits (max items per run)
- Enabled toggle
Schedule History
- Execution log per schedule
- Success/failure counts
- Duration statistics
- Error messages

6.4 Settings View

Purpose: Configure application and platform settings

Tabs:

General Settings

Download directory
Concurrent downloads limit
Retry attempts
Default priority
Storage limits
Auto-cleanup old files

Platform Settings

Instagram (FastDL, Toolzu, ImgInn)
- Enable/disable modules
- API keys
- Quality upgrade settings
- Rate limiting
TikTok
- Download quality
- Watermark removal
Forums
- Monitored forums list
- Check frequency
- Thread expiration

Notification Settings

Email notifications
- SMTP configuration
- Events to notify
Browser notifications
Webhook endpoints

Security Settings

Change password
API tokens management
Session timeout
Two-factor authentication

6.5 History View

Purpose: Browse and search downloaded content

Features:

Gallery View
- Thumbnail grid (for images)
- Platform icons
- Date overlay
List View
- Detailed table
- Sortable columns (date, platform, source, size)
- Filterable
Filters Sidebar
- Date range picker
- Platform checkboxes
- Source/username filter
- Content type filter
- File size range
Search
- Full-text search
- Advanced search (metadata)
Actions
- View file
- Download file
- Copy URL
- Delete (with confirmation)
- Re-download (add to queue)

6.6 Logs View

Purpose: Monitor application logs in real-time

Features:

Log Stream
- Auto-scrolling live feed
- Color-coded by level (DEBUG/INFO/WARNING/ERROR)
- Timestamp column
- Module/component column
- Message column
Filters
- Level filter (show only errors, etc.)
- Module filter (Scheduler, Instagram, etc.)
- Date range
- Search text
Actions
- Pause auto-scroll
- Clear logs
- Download log file
- Copy log entry

7. Real-Time Features (WebSocket)

Download Progress Events

// WebSocket message format
{
  "type": "download_progress",
  "data": {
    "id": 123,
    "url": "https://...",
    "platform": "instagram",
    "source": "evalongoria",
    "status": "downloading",
    "progress": 45,  // percentage
    "speed": "2.5 MB/s",
    "eta": "00:01:23",
    "file_size": 15728640
  }
}

Queue Updates

{
  "type": "queue_updated",
  "data": {
    "action": "added",  // added, removed, completed, failed
    "item": { /* queue item */ }
  }
}

Log Streaming

{
  "type": "log",
  "data": {
    "timestamp": "2025-10-13T10:30:45.123Z",
    "level": "INFO",
    "module": "Instagram",
    "message": "Downloaded post 123456"
  }
}

System Notifications

{
  "type": "notification",
  "data": {
    "level": "success",  // success, warning, error, info
    "title": "Download Complete",
    "message": "Downloaded 15 posts from evalongoria",
    "action": {
      "label": "View",
      "link": "/history?source=evalongoria"
    }
  }
}

8. Implementation Phases

Phase 1: Foundation (Week 1-2)

Goal: Basic infrastructure and authentication

Backend Tasks:

Set up FastAPI project structure
Create database models (Pydantic)
Implement JWT authentication
Create user management endpoints
Set up Redis connection
Basic CRUD for downloads/queue
Set up logging infrastructure

Frontend Tasks:

Initialize Vue.js project with Vite
Set up Vue Router and Pinia
Create base layout component
Implement login page
Create API service layer
Set up authentication flow

Deliverable: Working login and basic download list view

Phase 2: Core Features (Week 3-4)

Goal: Main functionality - queue and downloads

Backend Tasks:

Complete all download endpoints
Complete queue management endpoints
Integrate existing downloader modules
Set up Celery workers
Implement basic WebSocket support
Create download worker tasks

Frontend Tasks:

Build Dashboard view
Build Queue Manager view
Create download/queue components
Implement real-time updates (WebSocket)
Add progress indicators
Create notification system

Deliverable: Functional queue management and download monitoring

Phase 3: Scheduler & Settings (Week 5-6)

Goal: Automated scheduling and configuration

Backend Tasks:

Implement scheduler endpoints
Create scheduler worker
Integrate existing scheduler.py logic
Implement settings endpoints
Add validation and error handling

Frontend Tasks:

Build Scheduler view
Create cron expression editor
Build Settings view
Create platform configuration forms
Add form validation

Deliverable: Complete scheduler configuration and settings management

Phase 4: Advanced Features (Week 7-8)

Goal: History, logs, and analytics

Backend Tasks:

Implement log viewer endpoints
Create statistics/analytics endpoints
Add advanced filtering
Implement bulk operations
Add export functionality

Frontend Tasks:

Build History view with gallery
Build Logs view with streaming
Add statistics charts (Chart.js)
Implement advanced filters
Create export dialogs

Deliverable: Complete application with all views

Phase 5: Polish & Testing (Week 9-10)

Goal: Production readiness

Tasks:

Error handling improvements
Loading states and skeletons
Responsive design (mobile/tablet)
Performance optimization
Security hardening
Write unit tests
Write integration tests
Create Docker setup
Write deployment docs
User documentation

Deliverable: Production-ready application

9. Key Technical Implementations

9.1 Background Task Processing

Celery Task Example:

# backend/workers/download_worker.py
from celery import Celery
from modules.fastdl_module import FastDLDownloader
from backend.core.database import get_db
from backend.core.websocket_manager import broadcast_update

celery_app = Celery('media_downloader')

@celery_app.task(bind=True)
def download_instagram_posts(self, queue_item_id: int, config: dict):
    """Background task to download Instagram posts"""

    # Update queue status to 'downloading'
    update_queue_status(queue_item_id, 'downloading')

    # Broadcast via WebSocket
    broadcast_update({
        'type': 'download_started',
        'data': {'id': queue_item_id}
    })

    try:
        # Use existing FastDL module
        downloader = FastDLDownloader(
            unified_db=get_unified_db(),
            log_callback=lambda module, level, msg: self.update_state(
                state='PROGRESS',
                meta={'message': msg}
            )
        )

        count = downloader.download(
            username=config['username'],
            content_type=config['content_type'],
            output_dir=config['output_dir']
        )

        # Update queue status
        update_queue_status(queue_item_id, 'completed')

        # Broadcast completion
        broadcast_update({
            'type': 'download_completed',
            'data': {
                'id': queue_item_id,
                'count': count
            }
        })

        return {'status': 'success', 'count': count}

    except Exception as e:
        update_queue_status(queue_item_id, 'failed', error=str(e))
        broadcast_update({
            'type': 'download_failed',
            'data': {'id': queue_item_id, 'error': str(e)}
        })
        raise

9.2 Real-Time Updates

WebSocket Manager:

# backend/core/websocket_manager.py
from fastapi import WebSocket
from typing import List, Dict
import json
import asyncio

class ConnectionManager:
    def __init__(self):
        self.active_connections: List[WebSocket] = []
        self.user_connections: Dict[int, List[WebSocket]] = {}

    async def connect(self, websocket: WebSocket, user_id: int):
        await websocket.accept()
        self.active_connections.append(websocket)

        if user_id not in self.user_connections:
            self.user_connections[user_id] = []
        self.user_connections[user_id].append(websocket)

    def disconnect(self, websocket: WebSocket, user_id: int):
        self.active_connections.remove(websocket)
        if user_id in self.user_connections:
            self.user_connections[user_id].remove(websocket)

    async def broadcast(self, message: dict):
        """Broadcast to all connected clients"""
        disconnected = []
        for connection in self.active_connections:
            try:
                await connection.send_json(message)
            except:
                disconnected.append(connection)

        # Clean up disconnected clients
        for conn in disconnected:
            self.active_connections.remove(conn)

    async def send_to_user(self, user_id: int, message: dict):
        """Send message to specific user's connections"""
        if user_id in self.user_connections:
            for connection in self.user_connections[user_id]:
                try:
                    await connection.send_json(message)
                except:
                    pass

manager = ConnectionManager()

# Global function for easy broadcasting
async def broadcast_update(message: dict):
    await manager.broadcast(message)

WebSocket Endpoint:

# backend/api/routes/websocket.py
from fastapi import APIRouter, WebSocket, Depends
from backend.core.websocket_manager import manager
from backend.api.auth import get_current_user_ws

router = APIRouter()

@router.websocket("/ws/downloads")
async def websocket_downloads(
    websocket: WebSocket,
    user_id: int = Depends(get_current_user_ws)
):
    await manager.connect(websocket, user_id)
    try:
        while True:
            # Keep connection alive and handle client messages
            data = await websocket.receive_text()

            # Echo back (for ping/pong)
            if data == "ping":
                await websocket.send_text("pong")

    except Exception:
        manager.disconnect(websocket, user_id)

Frontend WebSocket Client:

// frontend/src/services/websocket.js
class WebSocketClient {
  constructor() {
    this.ws = null;
    this.reconnectTimer = null;
    this.listeners = new Map();
  }

  connect(url, token) {
    this.ws = new WebSocket(`${url}?token=${token}`);

    this.ws.onopen = () => {
      console.log('WebSocket connected');
      this.startHeartbeat();
    };

    this.ws.onmessage = (event) => {
      const message = JSON.parse(event.data);
      this.notifyListeners(message);
    };

    this.ws.onclose = () => {
      console.log('WebSocket disconnected');
      this.scheduleReconnect();
    };

    this.ws.onerror = (error) => {
      console.error('WebSocket error:', error);
    };
  }

  on(type, callback) {
    if (!this.listeners.has(type)) {
      this.listeners.set(type, []);
    }
    this.listeners.get(type).push(callback);
  }

  notifyListeners(message) {
    const { type, data } = message;
    if (this.listeners.has(type)) {
      this.listeners.get(type).forEach(callback => callback(data));
    }
  }

  startHeartbeat() {
    this.heartbeatTimer = setInterval(() => {
      if (this.ws.readyState === WebSocket.OPEN) {
        this.ws.send('ping');
      }
    }, 30000);
  }

  scheduleReconnect() {
    if (this.reconnectTimer) return;
    this.reconnectTimer = setTimeout(() => {
      console.log('Attempting to reconnect...');
      this.connect(this.url, this.token);
      this.reconnectTimer = null;
    }, 5000);
  }

  disconnect() {
    clearInterval(this.heartbeatTimer);
    clearTimeout(this.reconnectTimer);
    if (this.ws) {
      this.ws.close();
    }
  }
}

export default new WebSocketClient();

9.3 Integration with Existing Modules

Service Layer Example:

# backend/services/download_service.py
from pathlib import Path
from backend.core.database import get_unified_db
from modules.fastdl_module import FastDLDownloader
from modules.toolzu_module import ToolzuDownloader
from modules.tiktok_module import TikTokDownloader
from backend.workers.download_worker import download_instagram_posts

class DownloadService:
    def __init__(self):
        self.db = get_unified_db()

    async def start_instagram_download(
        self,
        username: str,
        content_type: str,
        platform: str = 'fastdl',
        max_downloads: int = None
    ):
        """Start Instagram download (queue as Celery task)"""

        # Add to queue in database
        queue_item = self.db.add_to_queue(
            url=f"https://instagram.com/{username}",
            platform='instagram',
            source=username,
            metadata={
                'content_type': content_type,
                'downloader': platform,
                'max_downloads': max_downloads
            }
        )

        # Queue Celery task
        task = download_instagram_posts.delay(
            queue_item_id=queue_item.id,
            config={
                'username': username,
                'content_type': content_type,
                'output_dir': str(Path('downloads') / platform),
                'max_downloads': max_downloads
            }
        )

        return {
            'queue_id': queue_item.id,
            'task_id': task.id,
            'status': 'queued'
        }

10. Docker Deployment

docker-compose.yml:

version: '3.8'

services:
  # Backend API
  backend:
    build:
      context: .
      dockerfile: docker/Dockerfile.backend
    ports:
      - "8000:8000"
    environment:
      - DATABASE_PATH=/app/database/media_downloader.db
      - REDIS_URL=redis://redis:6379/0
      - SECRET_KEY=${SECRET_KEY}
    volumes:
      - ./database:/app/database
      - ./downloads:/app/downloads
      - ./logs:/app/logs
    depends_on:
      - redis
    restart: unless-stopped

  # Celery Workers
  worker:
    build:
      context: .
      dockerfile: docker/Dockerfile.worker
    environment:
      - DATABASE_PATH=/app/database/media_downloader.db
      - REDIS_URL=redis://redis:6379/0
    volumes:
      - ./database:/app/database
      - ./downloads:/app/downloads
      - ./logs:/app/logs
    depends_on:
      - redis
      - backend
    restart: unless-stopped

  # Redis (cache and task queue)
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    restart: unless-stopped

  # Frontend
  frontend:
    build:
      context: .
      dockerfile: docker/Dockerfile.frontend
    ports:
      - "8080:80"
    depends_on:
      - backend
    restart: unless-stopped

  # Nginx Reverse Proxy (optional, for production)
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./docker/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./ssl:/etc/nginx/ssl:ro
    depends_on:
      - backend
      - frontend
    restart: unless-stopped

volumes:
  redis_data:

11. Security Considerations

Authentication
- JWT with short expiration (15 min) + refresh tokens
- Bcrypt password hashing
- Rate limiting on login endpoint
- Optional 2FA
Authorization
- Role-based access (admin/user)
- API key support for automation
- Scope-based permissions
Input Validation
- Pydantic models for all inputs
- URL validation and sanitization
- File upload restrictions
Data Protection
- HTTPS only (in production)
- CORS configuration
- SQL injection prevention (parameterized queries)
- XSS prevention (Vue escaping)
Rate Limiting
- Per-IP rate limits
- Per-user download limits
- Celery task rate limits

12. Testing Strategy

Backend Tests

Unit Tests: pytest for services/models
Integration Tests: API endpoint testing
E2E Tests: Full workflow testing

Frontend Tests

Unit Tests: Vitest for components
Integration Tests: Vue Test Utils
E2E Tests: Playwright/Cypress

Load Testing

Apache JMeter or Locust
Test concurrent downloads
Test WebSocket connections

13. Monitoring & Observability

Logging
- Structured JSON logs
- Log levels (DEBUG/INFO/WARNING/ERROR)
- Log rotation (daily, 7-day retention)
Metrics (optional)
- Prometheus + Grafana
- Download success rate
- Queue depth over time
- Worker utilization
Health Checks
- /health endpoint
- Database connectivity
- Redis connectivity
- Worker status
Alerts
- Email on critical errors
- Webhook on job failures
- Queue backlog alerts

14. Development Timeline Summary

Phase	Duration	Deliverable
Phase 1: Foundation	2 weeks	Auth + basic UI
Phase 2: Core Features	2 weeks	Queue + downloads
Phase 3: Scheduler	2 weeks	Automation + settings
Phase 4: Advanced	2 weeks	History + logs + stats
Phase 5: Polish	2 weeks	Production ready
Total	10 weeks	Full application

15. Post-Launch Features (Future)

Mobile App (React Native)
- iOS/Android native apps
- Push notifications
- Same API backend
Browser Extension
- Quick add from Instagram/TikTok
- Right-click menu integration
Advanced Analytics
- ML-based duplicate detection
- Content recommendations
- Storage optimization suggestions
Multi-tenancy
- Separate workspaces
- Team collaboration
- Shared queues
Cloud Storage Integration
- Google Drive
- Dropbox
- AWS S3
AI Features
- Auto-tagging with AI
- Face recognition
- Content categorization

Conclusion

This is a comprehensive, production-ready architecture that transforms your CLI tool into a modern web application. The design:

Preserves all existing functionality
Reuses existing downloader modules
Adds professional UI/UX
Enables real-time monitoring
Scales for future growth
Maintains simplicity where possible

The 10-week timeline is realistic for a single developer working full-time, or 20 weeks part-time (evenings/weekends).

Recommended Starting Point: Build Phase 1 (authentication + basic UI) first. If you like the direction, continue with Phase 2. This gives you an early proof-of-concept to validate the approach.

Would you like me to start implementing any specific part of this plan?

36 KiB Raw Blame History

Media Downloader Web GUI - Full Development Plan

Executive Summary

1. Technology Stack

Backend

Frontend

DevOps

2. System Architecture

3. Project Structure

4. Database Schema Enhancements

5. API Endpoints Design

Authentication

Downloads

Queue Management

Scheduler

Platforms

Settings

Logs

Statistics

WebSocket

6. Frontend Views & Features

6.1 Dashboard View

6.2 Queue Manager View

6.3 Scheduler View

6.4 Settings View

6.5 History View

6.6 Logs View

7. Real-Time Features (WebSocket)

Download Progress Events

Queue Updates

Log Streaming

System Notifications

8. Implementation Phases

Phase 1: Foundation (Week 1-2)

Phase 2: Core Features (Week 3-4)

Phase 3: Scheduler & Settings (Week 5-6)

Phase 4: Advanced Features (Week 7-8)

Phase 5: Polish & Testing (Week 9-10)

9. Key Technical Implementations

9.1 Background Task Processing

9.2 Real-Time Updates

9.3 Integration with Existing Modules

10. Docker Deployment

11. Security Considerations

12. Testing Strategy

Backend Tests

Frontend Tests

Load Testing

13. Monitoring & Observability

14. Development Timeline Summary

15. Post-Launch Features (Future)

Conclusion

36 KiB

Raw Blame History