1224 lines
36 KiB
Markdown
1224 lines
36 KiB
Markdown
# Media Downloader Web GUI - Full Development Plan
|
|
|
|
## Executive Summary
|
|
|
|
Transform the CLI media downloader into a modern web application with real-time monitoring, queue management, and intuitive configuration. Preserve all existing functionality while adding professional-grade UI/UX.
|
|
|
|
---
|
|
|
|
## 1. Technology Stack
|
|
|
|
### Backend
|
|
- **Framework:** FastAPI 0.104+ (async, auto-docs, WebSocket support)
|
|
- **Task Queue:** Celery + Redis (background job processing)
|
|
- **Database:** SQLite (existing) + Redis (cache/sessions)
|
|
- **WebSocket:** FastAPI native WebSocket for real-time updates
|
|
- **Authentication:** JWT tokens + secure session management
|
|
- **API Documentation:** Auto-generated Swagger/OpenAPI
|
|
|
|
### Frontend
|
|
- **Framework:** Vue.js 3 with Composition API
|
|
- **UI Library:** Vuetify 3 (Material Design components)
|
|
- **State Management:** Pinia (Vue's official state manager)
|
|
- **HTTP Client:** Axios
|
|
- **Real-time:** WebSocket with auto-reconnect
|
|
- **Build Tool:** Vite (fast, modern)
|
|
- **Charts:** Chart.js or Apache ECharts
|
|
|
|
### DevOps
|
|
- **Containerization:** Docker + Docker Compose
|
|
- **Process Manager:** Supervisor (for workers)
|
|
- **Reverse Proxy:** Nginx (production)
|
|
- **Logging:** Structured JSON logs + Log rotation
|
|
|
|
---
|
|
|
|
## 2. System Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Web Browser │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ Vue.js Frontend (Port 8080) │ │
|
|
│ │ - Dashboard - Queue Manager │ │
|
|
│ │ - Scheduler - Settings │ │
|
|
│ │ - Logs - History │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
└─────────────────┬───────────────────────────────────────┘
|
|
│ HTTP/WebSocket
|
|
▼
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ FastAPI Backend (Port 8000) │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ REST API Endpoints │ │
|
|
│ │ /api/v1/downloads /api/v1/queue │ │
|
|
│ │ /api/v1/scheduler /api/v1/settings │ │
|
|
│ │ /api/v1/platforms /api/v1/logs │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
│ ┌──────────────────────────────────────────────────┐ │
|
|
│ │ WebSocket Endpoints │ │
|
|
│ │ /ws/downloads (real-time progress) │ │
|
|
│ │ /ws/logs (live log streaming) │ │
|
|
│ └──────────────────────────────────────────────────┘ │
|
|
└─────────┬───────────────────────────────────┬───────────┘
|
|
│ │
|
|
▼ ▼
|
|
┌──────────────────────┐ ┌────────────────────────┐
|
|
│ SQLite Database │ │ Redis (Cache/Queue) │
|
|
│ media_downloader.db │ │ - Task queue │
|
|
│ - Downloads │ │ - Session store │
|
|
│ - Queue │ │ - Real-time events │
|
|
│ - Scheduler │ │ - Rate limit tracking │
|
|
└──────────────────────┘ └────────────────────────┘
|
|
▲ ▲
|
|
│ │
|
|
└───────────┬───────────────────────┘
|
|
│
|
|
┌─────────────▼──────────────────────────────┐
|
|
│ Celery Workers (Background Tasks) │
|
|
│ ┌─────────────────────────────────────┐ │
|
|
│ │ Worker 1: Instagram Downloads │ │
|
|
│ │ Worker 2: TikTok Downloads │ │
|
|
│ │ Worker 3: Forum Downloads │ │
|
|
│ │ Worker 4: Scheduler Tasks │ │
|
|
│ └─────────────────────────────────────┘ │
|
|
│ Uses existing modules: │
|
|
│ - fastdl_module.py │
|
|
│ - toolzu_module.py │
|
|
│ - tiktok_module.py │
|
|
│ - forum_downloader.py │
|
|
└─────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Project Structure
|
|
|
|
```
|
|
/opt/media-downloader/
|
|
├── backend/
|
|
│ ├── api/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── main.py # FastAPI app entry point
|
|
│ │ ├── dependencies.py # Dependency injection
|
|
│ │ ├── auth.py # JWT authentication
|
|
│ │ └── routes/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── downloads.py # Download CRUD endpoints
|
|
│ │ ├── queue.py # Queue management
|
|
│ │ ├── scheduler.py # Scheduler config
|
|
│ │ ├── platforms.py # Platform-specific endpoints
|
|
│ │ ├── settings.py # App configuration
|
|
│ │ ├── logs.py # Log viewer
|
|
│ │ ├── stats.py # Statistics/analytics
|
|
│ │ └── websocket.py # WebSocket handlers
|
|
│ ├── models/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── download.py # Pydantic models
|
|
│ │ ├── queue.py
|
|
│ │ ├── scheduler.py
|
|
│ │ └── user.py
|
|
│ ├── services/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── download_service.py # Business logic
|
|
│ │ ├── queue_service.py
|
|
│ │ ├── scheduler_service.py
|
|
│ │ └── platform_service.py
|
|
│ ├── workers/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── celery_app.py # Celery configuration
|
|
│ │ ├── download_worker.py # Background download tasks
|
|
│ │ └── scheduler_worker.py # Scheduled task executor
|
|
│ ├── core/
|
|
│ │ ├── __init__.py
|
|
│ │ ├── config.py # App configuration
|
|
│ │ ├── database.py # DB connection management
|
|
│ │ └── websocket_manager.py # WebSocket connection pool
|
|
│ └── requirements.txt
|
|
│
|
|
├── frontend/
|
|
│ ├── public/
|
|
│ │ ├── index.html
|
|
│ │ └── favicon.ico
|
|
│ ├── src/
|
|
│ │ ├── main.js # Vue app entry
|
|
│ │ ├── App.vue # Root component
|
|
│ │ ├── router/
|
|
│ │ │ └── index.js # Vue Router config
|
|
│ │ ├── stores/
|
|
│ │ │ ├── downloads.js # Pinia store
|
|
│ │ │ ├── queue.js
|
|
│ │ │ ├── scheduler.js
|
|
│ │ │ └── auth.js
|
|
│ │ ├── views/
|
|
│ │ │ ├── Dashboard.vue # Main dashboard
|
|
│ │ │ ├── QueueManager.vue # Queue UI
|
|
│ │ │ ├── SchedulerView.vue # Scheduler config
|
|
│ │ │ ├── Settings.vue # App settings
|
|
│ │ │ ├── Logs.vue # Log viewer
|
|
│ │ │ ├── History.vue # Download history
|
|
│ │ │ └── Login.vue # Authentication
|
|
│ │ ├── components/
|
|
│ │ │ ├── DownloadCard.vue # Download item display
|
|
│ │ │ ├── QueueItem.vue # Queue item
|
|
│ │ │ ├── ProgressBar.vue # Progress indicator
|
|
│ │ │ ├── PlatformSelector.vue
|
|
│ │ │ ├── LogViewer.vue
|
|
│ │ │ ├── StatCard.vue
|
|
│ │ │ └── ScheduleEditor.vue
|
|
│ │ ├── services/
|
|
│ │ │ ├── api.js # Axios HTTP client
|
|
│ │ │ └── websocket.js # WebSocket client
|
|
│ │ └── utils/
|
|
│ │ ├── formatters.js # Date/size formatters
|
|
│ │ └── validators.js # Input validation
|
|
│ ├── package.json
|
|
│ └── vite.config.js
|
|
│
|
|
├── docker/
|
|
│ ├── Dockerfile.backend
|
|
│ ├── Dockerfile.frontend
|
|
│ ├── Dockerfile.worker
|
|
│ └── nginx.conf
|
|
│
|
|
├── docker-compose.yml # Full stack orchestration
|
|
├── docker-compose.dev.yml # Development overrides
|
|
│
|
|
├── modules/ # Existing downloader modules (kept as-is)
|
|
│ ├── fastdl_module.py
|
|
│ ├── toolzu_module.py
|
|
│ ├── tiktok_module.py
|
|
│ ├── forum_downloader.py
|
|
│ ├── unified_database.py
|
|
│ └── scheduler.py
|
|
│
|
|
├── database/
|
|
│ └── media_downloader.db # Existing SQLite DB
|
|
│
|
|
├── downloads/ # Downloaded media files
|
|
├── logs/ # Application logs
|
|
└── config/
|
|
└── settings.json # App configuration
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Database Schema Enhancements
|
|
|
|
Add new tables to existing `media_downloader.db`:
|
|
|
|
```sql
|
|
-- User management (for authentication)
|
|
CREATE TABLE users (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
username TEXT UNIQUE NOT NULL,
|
|
email TEXT UNIQUE NOT NULL,
|
|
password_hash TEXT NOT NULL,
|
|
role TEXT DEFAULT 'user', -- 'admin', 'user'
|
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
|
last_login DATETIME,
|
|
active BOOLEAN DEFAULT 1
|
|
);
|
|
|
|
-- API tokens
|
|
CREATE TABLE api_tokens (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
user_id INTEGER NOT NULL,
|
|
token TEXT UNIQUE NOT NULL,
|
|
name TEXT,
|
|
expires_at DATETIME,
|
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
|
last_used DATETIME,
|
|
FOREIGN KEY (user_id) REFERENCES users(id)
|
|
);
|
|
|
|
-- Activity log
|
|
CREATE TABLE activity_log (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
user_id INTEGER,
|
|
action TEXT NOT NULL, -- 'download_started', 'queue_modified', etc.
|
|
resource_type TEXT, -- 'download', 'queue', 'scheduler'
|
|
resource_id TEXT,
|
|
details TEXT, -- JSON
|
|
ip_address TEXT,
|
|
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
|
|
FOREIGN KEY (user_id) REFERENCES users(id)
|
|
);
|
|
|
|
-- Notification settings
|
|
CREATE TABLE notifications (
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
user_id INTEGER NOT NULL,
|
|
type TEXT NOT NULL, -- 'email', 'browser', 'webhook'
|
|
events TEXT NOT NULL, -- JSON array of events to notify
|
|
config TEXT, -- JSON config (email address, webhook URL)
|
|
enabled BOOLEAN DEFAULT 1,
|
|
FOREIGN KEY (user_id) REFERENCES users(id)
|
|
);
|
|
|
|
-- System settings (key-value store)
|
|
CREATE TABLE system_settings (
|
|
key TEXT PRIMARY KEY,
|
|
value TEXT NOT NULL, -- JSON
|
|
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
|
);
|
|
|
|
-- Enhance existing download_queue table
|
|
ALTER TABLE download_queue ADD COLUMN assigned_worker TEXT;
|
|
ALTER TABLE download_queue ADD COLUMN started_at DATETIME;
|
|
ALTER TABLE download_queue ADD COLUMN progress INTEGER DEFAULT 0; -- 0-100
|
|
ALTER TABLE download_queue ADD COLUMN estimated_completion DATETIME;
|
|
```
|
|
|
|
---
|
|
|
|
## 5. API Endpoints Design
|
|
|
|
### Authentication
|
|
```
|
|
POST /api/v1/auth/login # Login with username/password
|
|
POST /api/v1/auth/logout # Logout
|
|
POST /api/v1/auth/refresh # Refresh JWT token
|
|
GET /api/v1/auth/me # Get current user
|
|
```
|
|
|
|
### Downloads
|
|
```
|
|
GET /api/v1/downloads # List downloads (with filters)
|
|
GET /api/v1/downloads/{id} # Get download details
|
|
POST /api/v1/downloads # Manually add download
|
|
DELETE /api/v1/downloads/{id} # Delete download record
|
|
GET /api/v1/downloads/stats # Get download statistics
|
|
POST /api/v1/downloads/bulk # Bulk operations (delete, retry)
|
|
```
|
|
|
|
### Queue Management
|
|
```
|
|
GET /api/v1/queue # Get download queue
|
|
POST /api/v1/queue # Add to queue
|
|
DELETE /api/v1/queue/{id} # Remove from queue
|
|
PATCH /api/v1/queue/{id} # Update queue item (priority, pause)
|
|
POST /api/v1/queue/{id}/retry # Retry failed download
|
|
POST /api/v1/queue/clear # Clear completed/failed
|
|
GET /api/v1/queue/stats # Queue statistics
|
|
```
|
|
|
|
### Scheduler
|
|
```
|
|
GET /api/v1/scheduler/jobs # List scheduled jobs
|
|
POST /api/v1/scheduler/jobs # Create new job
|
|
GET /api/v1/scheduler/jobs/{id} # Get job details
|
|
PATCH /api/v1/scheduler/jobs/{id} # Update job
|
|
DELETE /api/v1/scheduler/jobs/{id} # Delete job
|
|
POST /api/v1/scheduler/jobs/{id}/run # Trigger job manually
|
|
GET /api/v1/scheduler/history # Job execution history
|
|
```
|
|
|
|
### Platforms
|
|
```
|
|
GET /api/v1/platforms # List all platforms (Instagram, TikTok, etc.)
|
|
GET /api/v1/platforms/{name} # Get platform details
|
|
POST /api/v1/platforms/{name}/test # Test platform connection
|
|
POST /api/v1/platforms/{name}/download # Start platform download
|
|
|
|
# Platform-specific endpoints
|
|
POST /api/v1/platforms/instagram/posts # Download Instagram posts
|
|
POST /api/v1/platforms/instagram/stories # Download Instagram stories
|
|
POST /api/v1/platforms/tiktok/user # Download TikTok user videos
|
|
POST /api/v1/platforms/forum/thread # Monitor forum thread
|
|
```
|
|
|
|
### Settings
|
|
```
|
|
GET /api/v1/settings # Get all settings
|
|
PATCH /api/v1/settings # Update settings
|
|
GET /api/v1/settings/platforms # Get platform configs
|
|
PATCH /api/v1/settings/platforms/{name} # Update platform config
|
|
```
|
|
|
|
### Logs
|
|
```
|
|
GET /api/v1/logs # Get logs (paginated, filtered)
|
|
GET /api/v1/logs/download # Download log file
|
|
DELETE /api/v1/logs # Clear old logs
|
|
```
|
|
|
|
### Statistics
|
|
```
|
|
GET /api/v1/stats/overview # Dashboard overview
|
|
GET /api/v1/stats/timeline # Downloads over time
|
|
GET /api/v1/stats/platforms # Breakdown by platform
|
|
GET /api/v1/stats/sources # Breakdown by source (username)
|
|
GET /api/v1/stats/storage # Storage usage
|
|
```
|
|
|
|
### WebSocket
|
|
```
|
|
WS /ws/downloads # Real-time download updates
|
|
WS /ws/queue # Real-time queue updates
|
|
WS /ws/logs # Live log streaming
|
|
```
|
|
|
|
---
|
|
|
|
## 6. Frontend Views & Features
|
|
|
|
### 6.1 Dashboard View
|
|
**Purpose:** At-a-glance overview of system status
|
|
|
|
**Components:**
|
|
- **Stats Cards**
|
|
- Total downloads today/week/month
|
|
- Queue size (pending/active/failed)
|
|
- Success rate percentage
|
|
- Storage used/available
|
|
- Next scheduled task
|
|
|
|
- **Recent Activity**
|
|
- Live feed of downloads (last 20)
|
|
- Real-time progress bars
|
|
- Success/failure indicators
|
|
|
|
- **Platform Status**
|
|
- Instagram, TikTok, Forums
|
|
- Green/yellow/red status indicators
|
|
- Last successful download time
|
|
- Error counts
|
|
|
|
- **Quick Actions**
|
|
- Add to queue button
|
|
- Pause/Resume all
|
|
- Run scheduler manually
|
|
|
|
### 6.2 Queue Manager View
|
|
**Purpose:** Manage download queue with full control
|
|
|
|
**Features:**
|
|
- **Queue List**
|
|
- Filterable by status (pending/active/completed/failed)
|
|
- Sortable by priority, date, platform
|
|
- Bulk selection checkboxes
|
|
- Search by URL/username
|
|
|
|
- **Queue Item Cards**
|
|
- Platform icon + source username
|
|
- URL preview
|
|
- Priority indicator (High/Medium/Low)
|
|
- Status badge
|
|
- Progress bar (for active downloads)
|
|
- Action buttons:
|
|
- Retry (failed items)
|
|
- Pause/Resume
|
|
- Move up/down (priority)
|
|
- Delete
|
|
- View details
|
|
|
|
- **Bulk Actions**
|
|
- Clear completed
|
|
- Retry all failed
|
|
- Delete selected
|
|
- Change priority
|
|
|
|
- **Add to Queue Form**
|
|
- Platform dropdown
|
|
- URL input (with validation)
|
|
- Content type selector
|
|
- Priority selector
|
|
- Schedule for later checkbox
|
|
|
|
### 6.3 Scheduler View
|
|
**Purpose:** Configure automated download schedules
|
|
|
|
**Features:**
|
|
- **Schedule List**
|
|
- Active/Paused schedules
|
|
- Next run time
|
|
- Last run status
|
|
- Edit/Delete/Duplicate buttons
|
|
|
|
- **Schedule Editor**
|
|
- Name input
|
|
- Platform selector
|
|
- Content type (posts/stories/reels/videos)
|
|
- Username/source input
|
|
- Cron expression editor (with visual helper)
|
|
- Frequency presets (Hourly, Daily, Weekly, Custom)
|
|
- Date range limits
|
|
- Download limits (max items per run)
|
|
- Enabled toggle
|
|
|
|
- **Schedule History**
|
|
- Execution log per schedule
|
|
- Success/failure counts
|
|
- Duration statistics
|
|
- Error messages
|
|
|
|
### 6.4 Settings View
|
|
**Purpose:** Configure application and platform settings
|
|
|
|
**Tabs:**
|
|
|
|
**General Settings**
|
|
- Download directory
|
|
- Concurrent downloads limit
|
|
- Retry attempts
|
|
- Default priority
|
|
- Storage limits
|
|
- Auto-cleanup old files
|
|
|
|
**Platform Settings**
|
|
- Instagram (FastDL, Toolzu, ImgInn)
|
|
- Enable/disable modules
|
|
- API keys
|
|
- Quality upgrade settings
|
|
- Rate limiting
|
|
- TikTok
|
|
- Download quality
|
|
- Watermark removal
|
|
- Forums
|
|
- Monitored forums list
|
|
- Check frequency
|
|
- Thread expiration
|
|
|
|
**Notification Settings**
|
|
- Email notifications
|
|
- SMTP configuration
|
|
- Events to notify
|
|
- Browser notifications
|
|
- Webhook endpoints
|
|
|
|
**Security Settings**
|
|
- Change password
|
|
- API tokens management
|
|
- Session timeout
|
|
- Two-factor authentication
|
|
|
|
### 6.5 History View
|
|
**Purpose:** Browse and search downloaded content
|
|
|
|
**Features:**
|
|
- **Gallery View**
|
|
- Thumbnail grid (for images)
|
|
- Platform icons
|
|
- Date overlay
|
|
|
|
- **List View**
|
|
- Detailed table
|
|
- Sortable columns (date, platform, source, size)
|
|
- Filterable
|
|
|
|
- **Filters Sidebar**
|
|
- Date range picker
|
|
- Platform checkboxes
|
|
- Source/username filter
|
|
- Content type filter
|
|
- File size range
|
|
|
|
- **Search**
|
|
- Full-text search
|
|
- Advanced search (metadata)
|
|
|
|
- **Actions**
|
|
- View file
|
|
- Download file
|
|
- Copy URL
|
|
- Delete (with confirmation)
|
|
- Re-download (add to queue)
|
|
|
|
### 6.6 Logs View
|
|
**Purpose:** Monitor application logs in real-time
|
|
|
|
**Features:**
|
|
- **Log Stream**
|
|
- Auto-scrolling live feed
|
|
- Color-coded by level (DEBUG/INFO/WARNING/ERROR)
|
|
- Timestamp column
|
|
- Module/component column
|
|
- Message column
|
|
|
|
- **Filters**
|
|
- Level filter (show only errors, etc.)
|
|
- Module filter (Scheduler, Instagram, etc.)
|
|
- Date range
|
|
- Search text
|
|
|
|
- **Actions**
|
|
- Pause auto-scroll
|
|
- Clear logs
|
|
- Download log file
|
|
- Copy log entry
|
|
|
|
---
|
|
|
|
## 7. Real-Time Features (WebSocket)
|
|
|
|
### Download Progress Events
|
|
```javascript
|
|
// WebSocket message format
|
|
{
|
|
"type": "download_progress",
|
|
"data": {
|
|
"id": 123,
|
|
"url": "https://...",
|
|
"platform": "instagram",
|
|
"source": "evalongoria",
|
|
"status": "downloading",
|
|
"progress": 45, // percentage
|
|
"speed": "2.5 MB/s",
|
|
"eta": "00:01:23",
|
|
"file_size": 15728640
|
|
}
|
|
}
|
|
```
|
|
|
|
### Queue Updates
|
|
```javascript
|
|
{
|
|
"type": "queue_updated",
|
|
"data": {
|
|
"action": "added", // added, removed, completed, failed
|
|
"item": { /* queue item */ }
|
|
}
|
|
}
|
|
```
|
|
|
|
### Log Streaming
|
|
```javascript
|
|
{
|
|
"type": "log",
|
|
"data": {
|
|
"timestamp": "2025-10-13T10:30:45.123Z",
|
|
"level": "INFO",
|
|
"module": "Instagram",
|
|
"message": "Downloaded post 123456"
|
|
}
|
|
}
|
|
```
|
|
|
|
### System Notifications
|
|
```javascript
|
|
{
|
|
"type": "notification",
|
|
"data": {
|
|
"level": "success", // success, warning, error, info
|
|
"title": "Download Complete",
|
|
"message": "Downloaded 15 posts from evalongoria",
|
|
"action": {
|
|
"label": "View",
|
|
"link": "/history?source=evalongoria"
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Implementation Phases
|
|
|
|
### Phase 1: Foundation (Week 1-2)
|
|
**Goal:** Basic infrastructure and authentication
|
|
|
|
**Backend Tasks:**
|
|
- Set up FastAPI project structure
|
|
- Create database models (Pydantic)
|
|
- Implement JWT authentication
|
|
- Create user management endpoints
|
|
- Set up Redis connection
|
|
- Basic CRUD for downloads/queue
|
|
- Set up logging infrastructure
|
|
|
|
**Frontend Tasks:**
|
|
- Initialize Vue.js project with Vite
|
|
- Set up Vue Router and Pinia
|
|
- Create base layout component
|
|
- Implement login page
|
|
- Create API service layer
|
|
- Set up authentication flow
|
|
|
|
**Deliverable:** Working login and basic download list view
|
|
|
|
### Phase 2: Core Features (Week 3-4)
|
|
**Goal:** Main functionality - queue and downloads
|
|
|
|
**Backend Tasks:**
|
|
- Complete all download endpoints
|
|
- Complete queue management endpoints
|
|
- Integrate existing downloader modules
|
|
- Set up Celery workers
|
|
- Implement basic WebSocket support
|
|
- Create download worker tasks
|
|
|
|
**Frontend Tasks:**
|
|
- Build Dashboard view
|
|
- Build Queue Manager view
|
|
- Create download/queue components
|
|
- Implement real-time updates (WebSocket)
|
|
- Add progress indicators
|
|
- Create notification system
|
|
|
|
**Deliverable:** Functional queue management and download monitoring
|
|
|
|
### Phase 3: Scheduler & Settings (Week 5-6)
|
|
**Goal:** Automated scheduling and configuration
|
|
|
|
**Backend Tasks:**
|
|
- Implement scheduler endpoints
|
|
- Create scheduler worker
|
|
- Integrate existing scheduler.py logic
|
|
- Implement settings endpoints
|
|
- Add validation and error handling
|
|
|
|
**Frontend Tasks:**
|
|
- Build Scheduler view
|
|
- Create cron expression editor
|
|
- Build Settings view
|
|
- Create platform configuration forms
|
|
- Add form validation
|
|
|
|
**Deliverable:** Complete scheduler configuration and settings management
|
|
|
|
### Phase 4: Advanced Features (Week 7-8)
|
|
**Goal:** History, logs, and analytics
|
|
|
|
**Backend Tasks:**
|
|
- Implement log viewer endpoints
|
|
- Create statistics/analytics endpoints
|
|
- Add advanced filtering
|
|
- Implement bulk operations
|
|
- Add export functionality
|
|
|
|
**Frontend Tasks:**
|
|
- Build History view with gallery
|
|
- Build Logs view with streaming
|
|
- Add statistics charts (Chart.js)
|
|
- Implement advanced filters
|
|
- Create export dialogs
|
|
|
|
**Deliverable:** Complete application with all views
|
|
|
|
### Phase 5: Polish & Testing (Week 9-10)
|
|
**Goal:** Production readiness
|
|
|
|
**Tasks:**
|
|
- Error handling improvements
|
|
- Loading states and skeletons
|
|
- Responsive design (mobile/tablet)
|
|
- Performance optimization
|
|
- Security hardening
|
|
- Write unit tests
|
|
- Write integration tests
|
|
- Create Docker setup
|
|
- Write deployment docs
|
|
- User documentation
|
|
|
|
**Deliverable:** Production-ready application
|
|
|
|
---
|
|
|
|
## 9. Key Technical Implementations
|
|
|
|
### 9.1 Background Task Processing
|
|
|
|
**Celery Task Example:**
|
|
```python
|
|
# backend/workers/download_worker.py
|
|
from celery import Celery
|
|
from modules.fastdl_module import FastDLDownloader
|
|
from backend.core.database import get_db
|
|
from backend.core.websocket_manager import broadcast_update
|
|
|
|
celery_app = Celery('media_downloader')
|
|
|
|
@celery_app.task(bind=True)
|
|
def download_instagram_posts(self, queue_item_id: int, config: dict):
|
|
"""Background task to download Instagram posts"""
|
|
|
|
# Update queue status to 'downloading'
|
|
update_queue_status(queue_item_id, 'downloading')
|
|
|
|
# Broadcast via WebSocket
|
|
broadcast_update({
|
|
'type': 'download_started',
|
|
'data': {'id': queue_item_id}
|
|
})
|
|
|
|
try:
|
|
# Use existing FastDL module
|
|
downloader = FastDLDownloader(
|
|
unified_db=get_unified_db(),
|
|
log_callback=lambda module, level, msg: self.update_state(
|
|
state='PROGRESS',
|
|
meta={'message': msg}
|
|
)
|
|
)
|
|
|
|
count = downloader.download(
|
|
username=config['username'],
|
|
content_type=config['content_type'],
|
|
output_dir=config['output_dir']
|
|
)
|
|
|
|
# Update queue status
|
|
update_queue_status(queue_item_id, 'completed')
|
|
|
|
# Broadcast completion
|
|
broadcast_update({
|
|
'type': 'download_completed',
|
|
'data': {
|
|
'id': queue_item_id,
|
|
'count': count
|
|
}
|
|
})
|
|
|
|
return {'status': 'success', 'count': count}
|
|
|
|
except Exception as e:
|
|
update_queue_status(queue_item_id, 'failed', error=str(e))
|
|
broadcast_update({
|
|
'type': 'download_failed',
|
|
'data': {'id': queue_item_id, 'error': str(e)}
|
|
})
|
|
raise
|
|
```
|
|
|
|
### 9.2 Real-Time Updates
|
|
|
|
**WebSocket Manager:**
|
|
```python
|
|
# backend/core/websocket_manager.py
|
|
from fastapi import WebSocket
|
|
from typing import List, Dict
|
|
import json
|
|
import asyncio
|
|
|
|
class ConnectionManager:
|
|
def __init__(self):
|
|
self.active_connections: List[WebSocket] = []
|
|
self.user_connections: Dict[int, List[WebSocket]] = {}
|
|
|
|
async def connect(self, websocket: WebSocket, user_id: int):
|
|
await websocket.accept()
|
|
self.active_connections.append(websocket)
|
|
|
|
if user_id not in self.user_connections:
|
|
self.user_connections[user_id] = []
|
|
self.user_connections[user_id].append(websocket)
|
|
|
|
def disconnect(self, websocket: WebSocket, user_id: int):
|
|
self.active_connections.remove(websocket)
|
|
if user_id in self.user_connections:
|
|
self.user_connections[user_id].remove(websocket)
|
|
|
|
async def broadcast(self, message: dict):
|
|
"""Broadcast to all connected clients"""
|
|
disconnected = []
|
|
for connection in self.active_connections:
|
|
try:
|
|
await connection.send_json(message)
|
|
except:
|
|
disconnected.append(connection)
|
|
|
|
# Clean up disconnected clients
|
|
for conn in disconnected:
|
|
self.active_connections.remove(conn)
|
|
|
|
async def send_to_user(self, user_id: int, message: dict):
|
|
"""Send message to specific user's connections"""
|
|
if user_id in self.user_connections:
|
|
for connection in self.user_connections[user_id]:
|
|
try:
|
|
await connection.send_json(message)
|
|
except:
|
|
pass
|
|
|
|
manager = ConnectionManager()
|
|
|
|
# Global function for easy broadcasting
|
|
async def broadcast_update(message: dict):
|
|
await manager.broadcast(message)
|
|
```
|
|
|
|
**WebSocket Endpoint:**
|
|
```python
|
|
# backend/api/routes/websocket.py
|
|
from fastapi import APIRouter, WebSocket, Depends
|
|
from backend.core.websocket_manager import manager
|
|
from backend.api.auth import get_current_user_ws
|
|
|
|
router = APIRouter()
|
|
|
|
@router.websocket("/ws/downloads")
|
|
async def websocket_downloads(
|
|
websocket: WebSocket,
|
|
user_id: int = Depends(get_current_user_ws)
|
|
):
|
|
await manager.connect(websocket, user_id)
|
|
try:
|
|
while True:
|
|
# Keep connection alive and handle client messages
|
|
data = await websocket.receive_text()
|
|
|
|
# Echo back (for ping/pong)
|
|
if data == "ping":
|
|
await websocket.send_text("pong")
|
|
|
|
except Exception:
|
|
manager.disconnect(websocket, user_id)
|
|
```
|
|
|
|
**Frontend WebSocket Client:**
|
|
```javascript
|
|
// frontend/src/services/websocket.js
|
|
class WebSocketClient {
|
|
constructor() {
|
|
this.ws = null;
|
|
this.reconnectTimer = null;
|
|
this.listeners = new Map();
|
|
}
|
|
|
|
connect(url, token) {
|
|
this.ws = new WebSocket(`${url}?token=${token}`);
|
|
|
|
this.ws.onopen = () => {
|
|
console.log('WebSocket connected');
|
|
this.startHeartbeat();
|
|
};
|
|
|
|
this.ws.onmessage = (event) => {
|
|
const message = JSON.parse(event.data);
|
|
this.notifyListeners(message);
|
|
};
|
|
|
|
this.ws.onclose = () => {
|
|
console.log('WebSocket disconnected');
|
|
this.scheduleReconnect();
|
|
};
|
|
|
|
this.ws.onerror = (error) => {
|
|
console.error('WebSocket error:', error);
|
|
};
|
|
}
|
|
|
|
on(type, callback) {
|
|
if (!this.listeners.has(type)) {
|
|
this.listeners.set(type, []);
|
|
}
|
|
this.listeners.get(type).push(callback);
|
|
}
|
|
|
|
notifyListeners(message) {
|
|
const { type, data } = message;
|
|
if (this.listeners.has(type)) {
|
|
this.listeners.get(type).forEach(callback => callback(data));
|
|
}
|
|
}
|
|
|
|
startHeartbeat() {
|
|
this.heartbeatTimer = setInterval(() => {
|
|
if (this.ws.readyState === WebSocket.OPEN) {
|
|
this.ws.send('ping');
|
|
}
|
|
}, 30000);
|
|
}
|
|
|
|
scheduleReconnect() {
|
|
if (this.reconnectTimer) return;
|
|
this.reconnectTimer = setTimeout(() => {
|
|
console.log('Attempting to reconnect...');
|
|
this.connect(this.url, this.token);
|
|
this.reconnectTimer = null;
|
|
}, 5000);
|
|
}
|
|
|
|
disconnect() {
|
|
clearInterval(this.heartbeatTimer);
|
|
clearTimeout(this.reconnectTimer);
|
|
if (this.ws) {
|
|
this.ws.close();
|
|
}
|
|
}
|
|
}
|
|
|
|
export default new WebSocketClient();
|
|
```
|
|
|
|
### 9.3 Integration with Existing Modules
|
|
|
|
**Service Layer Example:**
|
|
```python
|
|
# backend/services/download_service.py
|
|
from pathlib import Path
|
|
from backend.core.database import get_unified_db
|
|
from modules.fastdl_module import FastDLDownloader
|
|
from modules.toolzu_module import ToolzuDownloader
|
|
from modules.tiktok_module import TikTokDownloader
|
|
from backend.workers.download_worker import download_instagram_posts
|
|
|
|
class DownloadService:
|
|
def __init__(self):
|
|
self.db = get_unified_db()
|
|
|
|
async def start_instagram_download(
|
|
self,
|
|
username: str,
|
|
content_type: str,
|
|
platform: str = 'fastdl',
|
|
max_downloads: int = None
|
|
):
|
|
"""Start Instagram download (queue as Celery task)"""
|
|
|
|
# Add to queue in database
|
|
queue_item = self.db.add_to_queue(
|
|
url=f"https://instagram.com/{username}",
|
|
platform='instagram',
|
|
source=username,
|
|
metadata={
|
|
'content_type': content_type,
|
|
'downloader': platform,
|
|
'max_downloads': max_downloads
|
|
}
|
|
)
|
|
|
|
# Queue Celery task
|
|
task = download_instagram_posts.delay(
|
|
queue_item_id=queue_item.id,
|
|
config={
|
|
'username': username,
|
|
'content_type': content_type,
|
|
'output_dir': str(Path('downloads') / platform),
|
|
'max_downloads': max_downloads
|
|
}
|
|
)
|
|
|
|
return {
|
|
'queue_id': queue_item.id,
|
|
'task_id': task.id,
|
|
'status': 'queued'
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 10. Docker Deployment
|
|
|
|
**docker-compose.yml:**
|
|
```yaml
|
|
version: '3.8'
|
|
|
|
services:
|
|
# Backend API
|
|
backend:
|
|
build:
|
|
context: .
|
|
dockerfile: docker/Dockerfile.backend
|
|
ports:
|
|
- "8000:8000"
|
|
environment:
|
|
- DATABASE_PATH=/app/database/media_downloader.db
|
|
- REDIS_URL=redis://redis:6379/0
|
|
- SECRET_KEY=${SECRET_KEY}
|
|
volumes:
|
|
- ./database:/app/database
|
|
- ./downloads:/app/downloads
|
|
- ./logs:/app/logs
|
|
depends_on:
|
|
- redis
|
|
restart: unless-stopped
|
|
|
|
# Celery Workers
|
|
worker:
|
|
build:
|
|
context: .
|
|
dockerfile: docker/Dockerfile.worker
|
|
environment:
|
|
- DATABASE_PATH=/app/database/media_downloader.db
|
|
- REDIS_URL=redis://redis:6379/0
|
|
volumes:
|
|
- ./database:/app/database
|
|
- ./downloads:/app/downloads
|
|
- ./logs:/app/logs
|
|
depends_on:
|
|
- redis
|
|
- backend
|
|
restart: unless-stopped
|
|
|
|
# Redis (cache and task queue)
|
|
redis:
|
|
image: redis:7-alpine
|
|
ports:
|
|
- "6379:6379"
|
|
volumes:
|
|
- redis_data:/data
|
|
restart: unless-stopped
|
|
|
|
# Frontend
|
|
frontend:
|
|
build:
|
|
context: .
|
|
dockerfile: docker/Dockerfile.frontend
|
|
ports:
|
|
- "8080:80"
|
|
depends_on:
|
|
- backend
|
|
restart: unless-stopped
|
|
|
|
# Nginx Reverse Proxy (optional, for production)
|
|
nginx:
|
|
image: nginx:alpine
|
|
ports:
|
|
- "80:80"
|
|
- "443:443"
|
|
volumes:
|
|
- ./docker/nginx.conf:/etc/nginx/nginx.conf:ro
|
|
- ./ssl:/etc/nginx/ssl:ro
|
|
depends_on:
|
|
- backend
|
|
- frontend
|
|
restart: unless-stopped
|
|
|
|
volumes:
|
|
redis_data:
|
|
```
|
|
|
|
---
|
|
|
|
## 11. Security Considerations
|
|
|
|
1. **Authentication**
|
|
- JWT with short expiration (15 min) + refresh tokens
|
|
- Bcrypt password hashing
|
|
- Rate limiting on login endpoint
|
|
- Optional 2FA
|
|
|
|
2. **Authorization**
|
|
- Role-based access (admin/user)
|
|
- API key support for automation
|
|
- Scope-based permissions
|
|
|
|
3. **Input Validation**
|
|
- Pydantic models for all inputs
|
|
- URL validation and sanitization
|
|
- File upload restrictions
|
|
|
|
4. **Data Protection**
|
|
- HTTPS only (in production)
|
|
- CORS configuration
|
|
- SQL injection prevention (parameterized queries)
|
|
- XSS prevention (Vue escaping)
|
|
|
|
5. **Rate Limiting**
|
|
- Per-IP rate limits
|
|
- Per-user download limits
|
|
- Celery task rate limits
|
|
|
|
---
|
|
|
|
## 12. Testing Strategy
|
|
|
|
### Backend Tests
|
|
- **Unit Tests:** pytest for services/models
|
|
- **Integration Tests:** API endpoint testing
|
|
- **E2E Tests:** Full workflow testing
|
|
|
|
### Frontend Tests
|
|
- **Unit Tests:** Vitest for components
|
|
- **Integration Tests:** Vue Test Utils
|
|
- **E2E Tests:** Playwright/Cypress
|
|
|
|
### Load Testing
|
|
- Apache JMeter or Locust
|
|
- Test concurrent downloads
|
|
- Test WebSocket connections
|
|
|
|
---
|
|
|
|
## 13. Monitoring & Observability
|
|
|
|
1. **Logging**
|
|
- Structured JSON logs
|
|
- Log levels (DEBUG/INFO/WARNING/ERROR)
|
|
- Log rotation (daily, 7-day retention)
|
|
|
|
2. **Metrics** (optional)
|
|
- Prometheus + Grafana
|
|
- Download success rate
|
|
- Queue depth over time
|
|
- Worker utilization
|
|
|
|
3. **Health Checks**
|
|
- `/health` endpoint
|
|
- Database connectivity
|
|
- Redis connectivity
|
|
- Worker status
|
|
|
|
4. **Alerts**
|
|
- Email on critical errors
|
|
- Webhook on job failures
|
|
- Queue backlog alerts
|
|
|
|
---
|
|
|
|
## 14. Development Timeline Summary
|
|
|
|
| Phase | Duration | Deliverable |
|
|
|-------|----------|-------------|
|
|
| Phase 1: Foundation | 2 weeks | Auth + basic UI |
|
|
| Phase 2: Core Features | 2 weeks | Queue + downloads |
|
|
| Phase 3: Scheduler | 2 weeks | Automation + settings |
|
|
| Phase 4: Advanced | 2 weeks | History + logs + stats |
|
|
| Phase 5: Polish | 2 weeks | Production ready |
|
|
| **Total** | **10 weeks** | **Full application** |
|
|
|
|
---
|
|
|
|
## 15. Post-Launch Features (Future)
|
|
|
|
1. **Mobile App** (React Native)
|
|
- iOS/Android native apps
|
|
- Push notifications
|
|
- Same API backend
|
|
|
|
2. **Browser Extension**
|
|
- Quick add from Instagram/TikTok
|
|
- Right-click menu integration
|
|
|
|
3. **Advanced Analytics**
|
|
- ML-based duplicate detection
|
|
- Content recommendations
|
|
- Storage optimization suggestions
|
|
|
|
4. **Multi-tenancy**
|
|
- Separate workspaces
|
|
- Team collaboration
|
|
- Shared queues
|
|
|
|
5. **Cloud Storage Integration**
|
|
- Google Drive
|
|
- Dropbox
|
|
- AWS S3
|
|
|
|
6. **AI Features**
|
|
- Auto-tagging with AI
|
|
- Face recognition
|
|
- Content categorization
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
This is a comprehensive, production-ready architecture that transforms your CLI tool into a modern web application. The design:
|
|
|
|
- **Preserves** all existing functionality
|
|
- **Reuses** existing downloader modules
|
|
- **Adds** professional UI/UX
|
|
- **Enables** real-time monitoring
|
|
- **Scales** for future growth
|
|
- **Maintains** simplicity where possible
|
|
|
|
The 10-week timeline is realistic for a single developer working full-time, or 20 weeks part-time (evenings/weekends).
|
|
|
|
**Recommended Starting Point:**
|
|
Build Phase 1 (authentication + basic UI) first. If you like the direction, continue with Phase 2. This gives you an early proof-of-concept to validate the approach.
|
|
|
|
Would you like me to start implementing any specific part of this plan?
|