Initial commit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 22:42:55 -04:00
commit 0d7b2b1aab
389 changed files with 280296 additions and 0 deletions
--- a/docs/archive/CODE_REVIEW_2026-01-16.md
+++ b/docs/archive/CODE_REVIEW_2026-01-16.md
@@ -0,0 +1,287 @@
+# Code Review: Media Downloader
+**Date:** 2026-01-16
+**Reviewer:** Claude (Opus 4.5)
+
+---
+
+## Summary: Current State
+
+| Category | Previous | Current | Status |
+|----------|----------|---------|--------|
+| Silent exception catches (backend) | 30+ problematic | All justified/intentional | RESOLVED |
+| SQL f-string interpolation | 8 instances flagged | All verified safe (constants only) | RESOLVED |
+| Path validation duplication | 8+ instances | Centralized in `core/utils.py` | RESOLVED |
+| `@handle_exceptions` coverage | Mixed | 87% covered, 30 endpoints missing | PARTIAL |
+| TypeScript `as any` | 65+ | 53 instances | IMPROVED |
+| Bare except handlers (modules) | 120+ | 31 remaining | SIGNIFICANTLY IMPROVED |
+| Direct sqlite3.connect() | 28 calls | 28 calls | NO CHANGE |
+| Shared components created | None | FilterBar, useMediaFiltering hook | CREATED BUT NOT USED |
+
+---
+
+## FIXED ISSUES
+
+### Backend Routers
+1. **Silent exception catches** - All remaining `except Exception: pass` patterns are now intentional with proper comments explaining fallback behavior
+2. **SQL interpolation** - MEDIA_FILTERS is confirmed as a constant string, no SQL injection risk
+3. **Path validation** - Centralized to `core/utils.py:55-103`, all routers use shared `validate_file_path()`
+4. **Thumbnail generation** - Properly centralized with imports from `core.utils`
+5. **Rate limiting** - Well-designed with appropriate limits per operation type
+
+### Python Modules
+1. **Bare exception handlers** - Reduced from 120+ to 31 (scheduler.py completely fixed)
+
+---
+
+## PARTIALLY FIXED / REMAINING ISSUES
+
+### Backend: Missing `@handle_exceptions` Decorator (30 endpoints)
+
+| Router | Missing Count | Lines |
+|--------|---------------|-------|
+| `appearances.py` | **25 endpoints** | All endpoints (lines 219-3007) |
+| `dashboard.py` | **3 endpoints** | Lines 17, 231, 254 |
+| `video_queue.py` | **1 endpoint** | Line 820 (stream endpoint) |
+| `files.py` | **1 endpoint** | Line 21 (thumbnail) |
+
+**Impact**: Unhandled exceptions will cause 500 errors instead of proper error responses.
+
+### Backend: Response Format Inconsistency (Still Present)
+
+| Router | Key Used | Should Be |
+|--------|----------|-----------|
+| `media.py:1483` | `"media"` | `"results"` |
+| `video_queue.py:369` | `"items"` | `"results"` |
+| `semantic.py:96` | `"count"` | `"total"` |
+
+### Frontend: Shared Components Created But Not Integrated
+
+**Created but unused:**
+- `FilterBar.tsx` (389 lines) - comprehensive reusable filter component
+- `useMediaFiltering.ts` hook (225 lines) - with useTransition/useDeferredValue optimizations
+
+**Pages still duplicating filter logic:**
+- Media.tsx, Review.tsx, Downloads.tsx, RecycleBin.tsx all have 10-15 duplicate filter state variables
+
+### Frontend: Giant Components Unchanged
+
+| File | Lines | Status |
+|------|-------|--------|
+| `Configuration.tsx` | **8,576** | Still massive, 32 `as any` assertions |
+| `InternetDiscovery.tsx` | 2,389 | Unchanged |
+| `Dashboard.tsx` | 2,182 | Unchanged |
+| `VideoDownloader.tsx` | 1,699 | Unchanged |
+
+### Frontend: Modal Duplication Persists
+
+Still duplicated across Media.tsx, Review.tsx, Downloads.tsx:
+- Move Modal
+- Add Reference Modal
+- Date Edit Modal
+
+---
+
+## NOT FIXED
+
+### Python Modules: Direct sqlite3.connect() Calls (28 total)
+
+| Module | Count | Lines |
+|--------|-------|-------|
+| `thumbnail_cache_builder.py` | 11 | 58, 200, 231, 259, 272, 356, 472, 521-522, 548-549 |
+| `forum_downloader.py` | 4 | 1180, 1183, 1185, 1188 |
+| `download_manager.py` | 4 | 132, 177, 775, 890 |
+| `easynews_monitor.py` | 3 | 82, 88, 344 |
+| `scheduler.py` | 6 | 105, 177, 217, 273, 307, 1952 (uses `closing()`) |
+
+**Problem**: These bypass `unified_database.py` connection pooling and write locks.
+
+### Python Modules: Remaining Bare Exception Handlers (31)
+
+| Module | Count | Issue |
+|--------|-------|-------|
+| `forum_downloader.py` | 26 | Silent failures in download loops, no logging |
+| `download_manager.py` | 2 | Returns fallback values silently |
+| `easynews_monitor.py` | 2 | Returns None/0 silently |
+| `thumbnail_cache_builder.py` | 1 | Cleanup only (minor) |
+
+---
+
+## Priority Fix List
+
+### P0 - Critical (Backend)
+1. Add `@handle_exceptions` to all 25 endpoints in `appearances.py`
+2. Add `@handle_exceptions` to all 3 endpoints in `dashboard.py`
+3. Add `@handle_exceptions` to `files.py` and `video_queue.py` stream endpoint
+
+### P1 - High (Modules)
+4. Add logging to 26 bare exception handlers in `forum_downloader.py`
+5. Migrate `download_manager.py` to use `unified_database.py`
+
+### P2 - Medium (Frontend)
+6. Integrate `FilterBar.tsx` into Media, Review, Downloads, RecycleBin pages
+7. Integrate `useMediaFiltering` hook
+8. Extract Configuration.tsx into sub-components
+
+### P3 - Low
+9. Standardize response pagination keys
+10. Migrate remaining modules to unified_database context managers
+
+---
+
+## Modernization Options
+
+### Option 1: UI Framework Modernization
+**Current**: Custom Tailwind CSS components
+**Upgrade to**: shadcn/ui - Modern, accessible, customizable component library built on Radix UI primitives
+**Benefits**: Consistent design system, accessibility built-in, dark mode support, reduces duplicate modal/form code
+
+### Option 2: State Management
+**Current**: Multiple `useState` calls (20+ per page), manual data fetching
+**Upgrade to**:
+- TanStack Query (already partially used): Expand usage for all data fetching
+- Zustand or Jotai: For global UI state (currently scattered across components)
+**Benefits**: Automatic caching, background refetching, optimistic updates
+
+### Option 3: API Layer
+**Current**: 2500+ line `api.ts` with manual fetch calls
+**Upgrade to**:
+- tRPC: End-to-end typesafe APIs (requires backend changes)
+- React Query + OpenAPI codegen: Auto-generate TypeScript client from FastAPI's OpenAPI spec
+**Benefits**: Eliminates `as any` assertions, compile-time API contract validation
+
+### Option 4: Component Architecture
+**Current**: Monolithic page components (Configuration.tsx: 8,576 lines)
+**Upgrade to**:
+- Split into feature-based modules
+- Extract reusable components: `DateEditModal`, `ConfirmDialog`, `BatchProgressModal`, `EmptyState`
+- Use compound component pattern for complex UIs
+
+### Option 5: Backend Patterns
+**Current**: Mixed patterns across routers
+**Standardize**:
+- Use Pydantic response models everywhere (enables automatic OpenAPI docs)
+- Centralized rate limiting configuration
+- Unified error handling middleware
+- Request ID injection for all logs
+
+### Option 6: Real-time Updates
+**Current**: WebSocket with manual reconnection (fixed 5s delay)
+**Upgrade to**:
+- Exponential backoff with jitter for reconnection
+- Server-Sent Events (SSE) for simpler one-way updates
+- Consider Socket.IO for robust connection handling
+
+---
+
+## Infrastructure Note
+
+The infrastructure for modernization exists:
+- **FilterBar** and **useMediaFiltering** hook are well-designed but need integration
+- **EnhancedLightbox** and **BatchProgressModal** are being used properly
+- **WebSocket security** is now properly implemented with protocol headers
+
+---
+
+## Detailed Findings
+
+### Backend Router Analysis
+
+#### Decorator Coverage by Router
+
+| Router | Endpoints | Decorated | Missing | Status |
+|--------|-----------|-----------|---------|--------|
+| media.py | 13 | 13 | 0 | 100% |
+| downloads.py | 10 | 10 | 0 | 100% |
+| review.py | 10 | 10 | 0 | 100% |
+| discovery.py | 34 | 34 | 0 | 100% |
+| celebrity.py | 34 | 34 | 0 | 100% |
+| video_queue.py | 21 | 20 | 1 | 95% |
+| health.py | 4 | 3 | 1 | 75% |
+| appearances.py | 25 | 0 | 25 | 0% CRITICAL |
+| dashboard.py | 3 | 0 | 3 | 0% CRITICAL |
+| files.py | 1 | 0 | 1 | 0% CRITICAL |
+
+#### Rate Limits Distribution
+
+| Limit | Count | Endpoints | Notes |
+|-------|-------|-----------|-------|
+| 5/min | 2 | Cache rebuild, clear functions | Very restrictive - admin |
+| 10/min | 5 | Batch operations | Write operations |
+| 20/min | 2 | Add operations | Upload/creation |
+| 30/min | 4 | Updates, settings | Moderate writes |
+| 60/min | 6 | Get operations, status | Read heavy |
+| 100/min | 5 | Get filters, stats, deletes | General reads |
+| 500/min | 1 | Get downloads | Base read |
+| 1000/min | 1 | Metadata check | High frequency |
+| 5000/min | 13 | Preview, thumbnail, search | Very high volume |
+
+### Frontend Component Analysis
+
+#### TypeScript `as any` by File
+
+| File | Count | Notes |
+|------|-------|-------|
+| Configuration.tsx | 32 | 2FA status and appearance config |
+| VideoDownloader.tsx | 7 | Video API calls |
+| RecycleBin.tsx | 3 | Response casting |
+| Health.tsx | 3 | Health status |
+| Notifications.tsx | 2 | API responses |
+| Discovery.tsx | 2 | Tab/filter state |
+| TwoFactorAuth.tsx | 1 | Status object |
+| Review.tsx | 1 | API response |
+| Media.tsx | 1 | API response |
+| Appearances.tsx | 1 | API response |
+
+#### Large Page Components
+
+| File | Lines | Recommendation |
+|------|-------|----------------|
+| Configuration.tsx | 8,576 | Split into TwoFactorAuthConfig, AppearanceConfig, PlatformConfigs |
+| InternetDiscovery.tsx | 2,389 | Extract search results, filters |
+| Dashboard.tsx | 2,182 | Extract cards, charts |
+| VideoDownloader.tsx | 1,699 | Extract queue management |
+| Downloads.tsx | 1,623 | Use FilterBar component |
+| Discovery.tsx | 1,464 | Use shared hooks |
+| Review.tsx | 1,463 | Use FilterBar, extract modals |
+| DownloadQueue.tsx | 1,431 | Extract queue items |
+| Media.tsx | 1,378 | Use FilterBar, extract modals |
+
+### Python Module Analysis
+
+#### Database Pattern Violations
+
+| Module | Pattern Used | Should Use |
+|--------|-------------|------------|
+| thumbnail_cache_builder.py | Direct `sqlite3.connect()` | `with db.get_connection(for_write=True)` |
+| forum_downloader.py | Direct `sqlite3.connect()` | `with db.get_connection(for_write=True)` |
+| download_manager.py | Direct `sqlite3.connect()` | `with db.get_connection(for_write=True)` |
+| easynews_monitor.py | Direct `sqlite3.connect()` | `with db.get_connection(for_write=True)` |
+| scheduler.py | `closing(sqlite3.connect())` | `with db.get_connection(for_write=True)` |
+
+---
+
+## Files Referenced
+
+### Backend
+- `/opt/media-downloader/web/backend/routers/appearances.py` - Missing decorators
+- `/opt/media-downloader/web/backend/routers/dashboard.py` - Missing decorators
+- `/opt/media-downloader/web/backend/routers/files.py` - Missing decorator
+- `/opt/media-downloader/web/backend/routers/video_queue.py` - Line 820 missing decorator
+- `/opt/media-downloader/web/backend/routers/media.py` - Line 1483 response key
+- `/opt/media-downloader/web/backend/routers/semantic.py` - Line 96 count vs total
+- `/opt/media-downloader/web/backend/core/utils.py` - Centralized utilities
+- `/opt/media-downloader/web/backend/core/exceptions.py` - @handle_exceptions decorator
+
+### Frontend
+- `/opt/media-downloader/web/frontend/src/pages/Configuration.tsx` - 8,576 lines
+- `/opt/media-downloader/web/frontend/src/components/FilterBar.tsx` - Unused
+- `/opt/media-downloader/web/frontend/src/hooks/useMediaFiltering.ts` - Unused
+- `/opt/media-downloader/web/frontend/src/lib/api.ts` - Type definitions
+
+### Modules
+- `/opt/media-downloader/modules/thumbnail_cache_builder.py` - 11 direct connects
+- `/opt/media-downloader/modules/forum_downloader.py` - 26 bare exceptions
+- `/opt/media-downloader/modules/download_manager.py` - 4 direct connects
+- `/opt/media-downloader/modules/easynews_monitor.py` - 3 direct connects
+- `/opt/media-downloader/modules/scheduler.py` - 6 closing() patterns
+- `/opt/media-downloader/modules/unified_database.py` - Reference implementation