# Comprehensive Code Review **Version:** 13.13.1 **Review Date:** 2025-12-04 **Last Updated:** 2026-03-29 **Status:** Active Review Document --- ## Table of Contents 1. [Executive Summary](#executive-summary) 2. [Codebase Statistics](#codebase-statistics) 3. [Architecture Overview](#architecture-overview) 4. [Critical Issues](#critical-issues) 5. [Code Quality Analysis](#code-quality-analysis) 6. [Performance Analysis](#performance-analysis) 7. [Security Analysis](#security-analysis) 8. [Technical Debt Tracker](#technical-debt-tracker) 9. [Inconsistencies Found](#inconsistencies-found) 10. [Recommended Next Big Features](#recommended-next-big-features) 11. [Version Update Checklist](#version-update-checklist) 12. [Changelog](#changelog) --- ## Executive Summary Media Downloader is a **production-grade, enterprise-level media archival system** with approximately 79,000 lines of code across Python backend and React/TypeScript frontend. The codebase demonstrates mature software engineering practices with recent significant improvements: 1. **~~Monolithic API file~~ RESOLVED** - API modularized into 17 routers (828 lines down from 10,465) 2. **Broad exception handling** (622 instances of `except Exception`) - gradual improvement ongoing 3. **Synchronous HTTP in async context** - blocking calls in FastAPI (partial fix with run_in_executor) 4. **Large module files** exceeding 2,000+ lines each - still needs attention ### Health Score: 8.3/10 (up from 8.2) | Category | Score | Notes | |----------|-------|-------| | Functionality | 9/10 | Feature-rich, comprehensive platform support | | Code Quality | 8/10 | API modularization complete, well-structured | | Performance | 8.5/10 | Correlated subqueries replaced with JOINs, PostgreSQL fully operational | | Security | 8.5/10 | 12-point security audit + SQL injection defense-in-depth + proxy domain hardening | | Maintainability | 7/10 | Significantly improved with modular routers | | Test Coverage | 4/10 | Limited test suite | | Documentation | 9/10 | Excellent inline and external docs | --- ## Codebase Statistics ### Current Version: 13.13.1 | Metric | Count | Change Since 12.0.0 | |--------|-------|-------------------| | **Total Lines (Python)** | ~70,500 | +21,800 (+ press.py, cloud_backup.py, cloud_backup_sync.py, backfill_press.py, besteyecandy_client.py, coppermine_client.py, reddit_client.py + all previous) | | **Total Lines (TypeScript/TSX)** | ~39,500 | +14,000 (+ Gallery page, GalleryLightbox, Press page, PressArticleModal, cloud backup dashboard widget, shuffle support + all previous) | | **Python Modules** | 56 | +26 (+ besteyecandy_client, coppermine_client, reddit_client + all previous) | | **API Routers** | 28 | +4 (+ press, cloud_backup) | | **API Endpoints** | 280+ | +110 (+ press CRUD/fetch + cloud backup config/sync/status + all previous) | | **Frontend Pages** | 39 | +16 (+ Gallery page, Press page) | | **Frontend Components** | 16+ | +5 (+ GalleryLightbox, PressArticleModal) | | **Frontend Contexts** | 2 | BreadcrumbContext, PrivateGalleryContext | | **Frontend Hooks** | 3 | useBreadcrumb, usePrivateGalleryAuth, useEnabledFeatures | | **Database Tables** | 89 | +63 (PostgreSQL migration + private_media_reddit_communities + private_media_reddit_history) | | **Database Backend** | PostgreSQL 16 | Migrated from SQLite in v13.0.0 via runtime adapter | | **Database Indexes** | 51+ | +26 (paid content + private gallery + reddit community + message indexes) | ### File Size Analysis (Updated) | File | Lines | Severity | Status | |------|-------|----------|--------| | `web/backend/api.py` | 828 | ✅ RESOLVED | Modularized into 17 routers | | `web/frontend/src/pages/Configuration.tsx` | 5,945 | HIGH | Component extraction needed | | `modules/forum_downloader.py` | 4,882 | HIGH | Modular redesign needed | | `modules/unified_database.py` | 4,479 | MEDIUM | -200 lines (unused adapters removed) | | `modules/imginn_module.py` | 3,485 | MEDIUM | Uses unified_db directly now | | `media-downloader.py` | 2,866 | MEDIUM | Command pattern | | `modules/fastdl_module.py` | 2,236 | MEDIUM | Uses centralized instagram_utils | | `modules/instagram_client_module.py` | 1,651 | ✅ NEW | Direct Instagram API client (v13.4.0) | | `modules/paid_content/instagram_client.py` | 1,081 | ✅ NEW | Instagram API for Paid Content (v13.4.0) | | `modules/instagram_utils.py` | 429 | ✅ NEW | Centralized Instagram functions | ### API Router Breakdown (NEW) | Router | Lines | Purpose | |--------|-------|---------| | `private_gallery.py` | 6,707 | Private gallery auth, media, persons, encryption, features, async job tracking, URL import, Reddit monitor endpoints | | `paid_content.py` | 6,255 | Paid content CRUD, services, creators, feed, notifications, messages, OnlyFans/Fansly setup, auto health checks, bulk service sync | | `media.py` | 1,334 | Media serving, thumbnails, gallery, date editing | | `face.py` | 985 | Face recognition endpoints | | `discovery.py` | 952 | Source discovery, platforms | | `video.py` | 939 | Video downloads, info | | `review.py` | 904 | Review queue management | | `downloads.py` | 836 | Download history, analytics | | `scheduler.py` | 630 | Scheduler control | | `stats.py` | 521 | Dashboard stats, monitoring | | `config.py` | 521 | Configuration management | | `scrapers.py` | 485 | Scraper controls, errors | | Others | ~2,000 | Auth, health, recycle, etc. | ### Async/Sync Function Distribution - **Async functions in API:** 190 - **Sync functions in API:** 27 - **Concern:** Many sync functions call blocking `requests` library --- ## Architecture Overview ``` ┌─────────────────────────────────────────────────────────────────────┐ │ MEDIA DOWNLOADER │ ├─────────────────────────────────────────────────────────────────────┤ │ FRONTEND (React 18 + TypeScript + Tailwind) │ │ ├── 35 Pages (Dashboard, Media, Downloads, Review, Config, etc.) │ │ ├── 12 Reusable Components │ │ ├── TanStack Query for data fetching │ │ └── WebSocket for real-time updates │ ├─────────────────────────────────────────────────────────────────────┤ │ API LAYER (FastAPI + Uvicorn) │ │ ├── 231+ REST Endpoints │ │ ├── JWT + Session Authentication │ │ ├── 2FA (TOTP, Duo, Passkeys) │ │ ├── Rate Limiting (slowapi) │ │ └── Redis Caching │ ├─────────────────────────────────────────────────────────────────────┤ │ CORE MODULES (36 Python modules) │ │ ├── Platform Downloaders (Instagram x4, TikTok, Snapchat, Forums) │ │ ├── Face Recognition (InsightFace + DeepFace + dlib fallback) │ │ ├── Semantic Search (CLIP embeddings) │ │ ├── Scheduler (randomized intervals) │ │ └── File Management (move, hash, dedupe) │ ├─────────────────────────────────────────────────────────────────────┤ │ DATA LAYER │ │ ├── PostgreSQL 16 (via pg_adapter runtime translation, v13.0.0) │ │ ├── Redis (caching, sessions) │ │ └── File System (organized by platform/source) │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## Critical Issues ### 1. ✅ RESOLVED: Monolithic API File **File:** `/opt/media-downloader/web/backend/api.py` **Lines:** 828 (down from 10,465 - 92% reduction!) **Status:** RESOLVED as of v6.52.44 **Solution Implemented:** ``` web/backend/ ├── api.py (app initialization, WebSocket only - 828 lines) ├── core/ │ ├── dependencies.py (shared state, auth) │ ├── exceptions.py (custom exception classes) │ └── config.py (configuration) ├── routers/ │ ├── auth.py (authentication, 2FA) │ ├── downloads.py (download history, analytics) │ ├── media.py (media serving, gallery) │ ├── scheduler.py (scheduler control) │ ├── discovery.py (source discovery) │ ├── face.py (face recognition) │ ├── recycle.py (recycle bin) │ ├── review.py (review queue) │ ├── video.py (video downloads) │ ├── stats.py (dashboard, monitoring) │ ├── config.py (configuration) │ ├── scrapers.py (scraper controls, errors) │ ├── health.py (health checks) │ ├── notifications.py (push notifications) │ ├── manual_import.py (manual imports) │ ├── semantic_search.py (CLIP search) │ └── cache.py (cache management) └── (17 routers total) ``` **Resolution Date:** 2025-12-05 **Effort Spent:** ~8 hours --- ### 2. HIGH: Broad Exception Handling **Count:** 622 instances of `except Exception` **Impact:** Debugging difficulty, silent failures, error masking **Examples Found:** ```python # Anti-pattern (throughout codebase) try: result = do_something() except Exception as e: logger.error(f"Error: {e}") return None ``` **Recommended Fix:** ```python # Catch specific exceptions try: result = do_something() except requests.RequestException as e: logger.error(f"Network error: {e}") raise DownloadError(f"Failed to download: {e}") from e except ValueError as e: logger.error(f"Invalid data: {e}") raise ValidationError(f"Invalid input: {e}") from e ``` **Priority:** MEDIUM (gradual refactoring) --- ### 3. HIGH: Synchronous HTTP in Async Context **Issue:** FastAPI is async but many endpoints use synchronous `requests` library **Impact:** - Blocks event loop - Reduces concurrent request handling - Performance bottleneck **Current Pattern:** ```python @app.get("/api/something") async def get_something(): # BLOCKING CALL in async function! response = requests.get("https://external-api.com/data") return response.json() ``` **Solution Options:** 1. **Use httpx (recommended):** ```python import httpx @app.get("/api/something") async def get_something(): async with httpx.AsyncClient() as client: response = await client.get("https://external-api.com/data") return response.json() ``` 2. **Use run_in_executor for sync code:** ```python import asyncio from concurrent.futures import ThreadPoolExecutor executor = ThreadPoolExecutor(max_workers=10) @app.get("/api/something") async def get_something(): loop = asyncio.get_event_loop() response = await loop.run_in_executor( executor, lambda: requests.get("https://external-api.com/data") ) return response.json() ``` **Effort:** 24-40 hours **Priority:** HIGH (for performance-critical endpoints) --- ### 4. MEDIUM: Instagram Module Duplication **Files Affected:** - `modules/fastdl_module.py` (2,236 lines) - `modules/imginn_module.py` (2,870 lines) - `modules/toolzu_module.py` (1,025 lines) **Duplication:** 60-70% shared code including: - Cookie management - FlareSolverr integration - HTML parsing - Download logic - Error handling **Solution:** Create base class: ```python # modules/instagram/base_instagram.py class BaseInstagramDownloader(ABC): def __init__(self, config, unified_db): self.config = config self.unified_db = unified_db self.cookie_manager = CookieManager() self.flaresolverr = FlareSolverrClient() def _get_or_create_session(self): ... def _parse_stories(self, html): ... def _download_media(self, url, save_path): ... @abstractmethod def _get_content_urls(self, username) -> List[str]: pass ``` **Effort:** 12-16 hours **Priority:** MEDIUM --- ## Code Quality Analysis ### Positive Patterns Found 1. **Unified Database Layer** - Single source of truth with adapters 2. **Universal Logger** - Centralized logging with web visibility 3. **Database-First File Tracking** - 50-100x faster than filesystem scanning 4. **Face Recognition Fallback Chain** - InsightFace → DeepFace → dlib 5. **Soft Delete with Recycle Bin** - File recovery capability 6. **Comprehensive Documentation** - 560KB+ in docs folder 7. **Real-Time Updates** - WebSocket for logs and errors ### Areas Needing Improvement | Area | Current State | Target State | |------|---------------|--------------| | Exception handling | 622 broad catches | Specific exception types | | Type hints | Partial coverage | Full type annotations | | Test coverage | ~10% estimated | 70%+ target | | API documentation | No OpenAPI docs | Full Swagger/ReDoc | | Database migrations | Ad-hoc ALTER TABLE | Alembic migrations | | Code formatting | Inconsistent | Black + isort | ### Code Smells Detected 1. **God Objects:** `ForumDownloader` class does too much 2. **Long Methods:** Several methods exceed 100 lines 3. **Deep Nesting:** 5+ levels in some functions 4. **Magic Numbers:** Hardcoded timeouts, limits, paths 5. **Missing Type Hints:** Many functions lack annotations --- ## Performance Analysis ### Current Optimizations (Good) 1. **Database Indexes:** 22+ optimized indexes 2. **Connection Pooling:** PostgreSQL ThreadedConnectionPool (2-30 connections, v13.0.0) 3. **Thumbnail Caching:** SHA256-keyed cache with on-demand generation 4. **Query Performance:** Sub-10ms for indexed queries 5. **Batch Face Lookup:** N+1 query pattern fixed in 6.52.32 6. **Lazy Loading:** On-demand metadata fetching in lightbox ### Performance Bottlenecks Identified | Issue | Location | Impact | Fix | |-------|----------|--------|-----| | Sync HTTP in async | api.py | High | Use httpx/aiohttp | | No query result caching | Multiple endpoints | Medium | Add Redis caching | | Full table scans | Some search queries | Medium | Add missing indexes | | Large response payloads | `/api/downloads` | Low | Pagination + field selection | ### Recommended Performance Improvements 1. **Add Redis caching for expensive queries:** ```python @app.get("/api/stats") async def get_stats(): cached = await redis.get("stats:global") if cached: return json.loads(cached) stats = compute_expensive_stats() await redis.setex("stats:global", 300, json.dumps(stats)) return stats ``` 2. **Implement field selection for API responses:** ```python @app.get("/api/downloads") async def get_downloads(fields: str = None): if fields: # Return only requested fields selected = fields.split(",") return [{k: v for k, v in d.items() if k in selected} for d in downloads] ``` --- ## Security Analysis ### Current Security Measures (Strong) | Feature | Implementation | Status | |---------|----------------|--------| | Authentication | JWT + Session cookies | ✅ | | Password Hashing | bcrypt | ✅ | | 2FA | TOTP + Duo + Passkeys | ✅ | | CSRF Protection | starlette-csrf | ✅ | | Rate Limiting | slowapi | ✅ | | SQL Injection | safe_query_builder module | ✅ | | Path Traversal | Validation on file paths | ✅ | ### Security Concerns 1. **Hardcoded Paths:** ```python # Found in multiple files base_path = Path("/opt/immich/md") # Should be configurable ``` 2. **Potential Secret Exposure:** ```python # .env file handling - ensure proper permissions # Check: ls -la .env should show 600 permissions ``` 3. **Missing Security Headers:** ```python # Recommend adding in middleware response.headers["X-Content-Type-Options"] = "nosniff" response.headers["X-Frame-Options"] = "DENY" response.headers["X-XSS-Protection"] = "1; mode=block" ``` --- ## Technical Debt Tracker ### Outstanding Technical Debt | ID | Description | Priority | Effort | Status | |----|-------------|----------|--------|--------| | TD-001 | Monolithic api.py (10,465 lines) | HIGH | 24h | **In Progress** - Router framework created | | TD-002 | Broad exception handling (622 cases) | MEDIUM | 16h | **Resolved** - Custom exceptions created | | TD-003 | Instagram module duplication | MEDIUM | 16h | **Resolved** - Base class created | | TD-004 | Missing database migrations | MEDIUM | 8h | Open | | TD-005 | Sync HTTP in async context | HIGH | 32h | Open | | TD-006 | Missing API documentation | LOW | 6h | Open | | TD-007 | Low test coverage (~10%) | MEDIUM | 32h | Open | | TD-008 | Hardcoded configuration values | LOW | 4h | **Resolved** - Unified config manager | | TD-009 | Missing type hints | LOW | 16h | Open | | TD-010 | Large frontend components | MEDIUM | 12h | Open | ### Recently Resolved | ID | Description | Resolved In | Notes | |----|-------------|-------------|-------| | TD-002 | Broad exception handling | 6.52.38 | Custom exception classes in `core/exceptions.py` | | TD-003 | Instagram module duplication | 6.52.38 | Base class in `modules/instagram/base.py` | | TD-008 | Hardcoded configuration values | 6.52.38 | Unified config in `core/config.py` | | TD-015 | Inconsistent date formats | 6.52.38 | ISO 8601 utilities in `core/responses.py` | | TD-016 | Inconsistent error responses | 6.52.38 | Standardized in `core/responses.py` | | TD-011 | N+1 query in downloads | 6.52.32 | Batch face lookup added | | TD-012 | Thread safety in forum downloads | 6.52.32 | Context priority fix | | TD-013 | Asyncio.run() in threads | 6.52.32 | run_coroutine_threadsafe | | TD-014 | Missing dimensions on download | 6.52.29 | PIL/ffprobe extraction | --- ## Inconsistencies Found ### 1. ✅ RESOLVED: Logging Inconsistency **Issue:** Mix of logging approaches (duplicate log() methods across modules) **Solution Implemented (v11.19.2):** - Created `modules/base_module.py` with `LoggingMixin` class - All download modules now inherit from `LoggingMixin` - Provides consistent `log()` method with backwards-compatible `log_callback` support - Removed 12+ duplicate `log()` method implementations - Updated 13 modules: instaloader, tiktok, imginn, fastdl, toolzu, coppermine, forum_downloader, download_manager, move_module, scheduler, snapchat, instagram_repost_detector **Resolution Date:** 2025-12-26 --- ### 2. Error Response Format Inconsistency **Issue:** Different error response formats across endpoints ```python # Format 1 raise HTTPException(status_code=400, detail="Invalid request") # Format 2 return {"error": True, "message": "Invalid request"} # Format 3 return JSONResponse(status_code=400, content={"error": "Invalid request"}) ``` **Solution:** Standardize on HTTPException with consistent detail format --- ### 3. Date Format Inconsistency **Issue:** Multiple date formats in database and API ```python # Format 1: ISO "2025-12-04T10:30:00Z" # Format 2: Unix timestamp 1733308200 # Format 3: Custom "2025-12-04 10:30:00" ``` **Solution:** Standardize on ISO 8601 with timezone --- ### 4. Configuration Loading Inconsistency **Issue:** Config loaded from multiple sources ```python # Source 1: Environment variables os.environ.get("MEDIA_PATH") # Source 2: .env file load_dotenv() # Source 3: Database settings settings_manager.get("media_path") # Source 4: Hardcoded defaults DEFAULT_PATH = "/opt/immich/md" ``` **Solution:** Single configuration manager with hierarchy: 1. Environment variables (highest priority) 2. .env file 3. Database settings 4. Hardcoded defaults (lowest priority) --- ## Recommended Next Big Features Based on codebase analysis and user impact potential: ### Tier 1: High Impact, High Value #### 1. **Webhook Integration System** ⭐ RECOMMENDED **Effort:** 6-8 hours | **Value:** HIGH Why this feature: - Zero external dependencies - Enables Discord, Slack, Home Assistant integration - Automation workflows without code changes - Natural extension of existing notification system ```python # Implementation sketch class WebhookManager: events = ['download_complete', 'download_error', 'face_matched', 'new_content'] async def fire_webhook(self, event: str, data: dict): webhooks = await self.get_webhooks_for_event(event) for webhook in webhooks: await self.send_with_retry(webhook, event, data) ``` --- #### 2. **Duplicate Management Dashboard** ⭐ RECOMMENDED **Effort:** 10-12 hours | **Value:** HIGH Why this feature: - File hash infrastructure already exists - Users manually hunting duplicates today - Could reclaim significant storage space - Visual side-by-side comparison - "Keep best quality" automation ``` ┌─────────────────────────────────────────────────────────────┐ │ Duplicates Dashboard 230 GB saved │ ├─────────────────────────────────────────────────────────────┤ │ ┌─────────────────────┬─────────────────────┐ │ │ │ Original │ Duplicate │ │ │ │ [Image Preview] │ [Image Preview] │ │ │ │ 1920x1080 / 2.5 MB │ 1280x720 / 1.8 MB │ │ │ │ Instagram/user1 │ FastDL/user1 │ │ │ │ [✓ Keep] │ [Delete] │ │ │ └─────────────────────┴─────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ ``` --- #### 3. **Mobile-First PWA** **Effort:** 4-6 hours | **Value:** HIGH Why this feature: - Installable app experience - Offline mode capability - Push notifications - Already using React, easy to add - No app store approval needed --- ### Tier 2: Medium Impact, Medium Value | Feature | Effort | Value | Notes | |---------|--------|-------|-------| | Background Job Queue (Celery) | 12-16h | HIGH | Heavy ops in background | | API Modularization | 16-24h | MEDIUM | Enables better testing | | Prometheus Metrics | 6-8h | MEDIUM | Grafana dashboards | | S3/Object Storage | 6-8h | MEDIUM | Cloud storage option | | Plugin System | 16-24h | HIGH | User extensibility | ### Tier 3: Future Considerations | Feature | Effort | Notes | |---------|--------|-------| | RBAC (Role-Based Access) | 12-16h | Multi-user environments | | AI Auto-Tagging | 16-24h | ML-based categorization | | Video Transcoding | 12-16h | Format conversion | | Full-Text Search | 8-12h | Elasticsearch/Meilisearch | --- ## Version Update Checklist Use this checklist when updating to a new version: ### Pre-Update - [ ] Review CHANGELOG.md for breaking changes - [ ] Backup database: `cp database/media_downloader.db database/backup_$(date +%Y%m%d).db` - [ ] Note current version: `cat VERSION` - [ ] Check disk space: `df -h /opt/media-downloader` ### During Update - [ ] Stop services: `systemctl stop media-downloader` - [ ] Pull new code or extract release - [ ] Run database migrations if any - [ ] Update dependencies: `pip install -r requirements.txt` - [ ] Rebuild frontend: `cd web/frontend && npm run build` ### Post-Update - [ ] Start services: `systemctl start media-downloader` - [ ] Verify API health: `curl http://localhost:8000/api/health` - [ ] Check logs for errors: `tail -f logs/$(date +%Y-%m-%d).log` - [ ] Update this document's version number - [ ] Test critical functionality ### Update This Review Document - [ ] Update version number at top - [ ] Update codebase statistics - [ ] Add any new issues found - [ ] Mark resolved issues - [ ] Update recommendations if needed --- ## Changelog ### Review Document Updates | Date | Version Reviewed | Changes | |------|-----------------|---------| | 2026-03-29 | 13.13.1 | **Pinned Post Highlighting, Mobile Header Layout, Feed Navigation Fixes** — Patch release. PostCard component in `Feed.tsx` adds conditional amber background for pinned posts: `bg-amber-50/60 dark:bg-amber-900/15` (unselected) and `bg-amber-100/80 dark:bg-amber-900/30` (selected), replacing the default blue `bg-primary/10` selection for pinned items. New `highlighted` boolean prop added to `PostDetailView.tsx` interface — swaps root div `bg-card` for `bg-amber-50/60 dark:bg-amber-900/15` and sticky header `bg-card` for `bg-amber-50/80 dark:bg-amber-900/25`. Passed as `highlighted={true}` for mobile social view pinned posts, `highlighted={selectedPost.is_pinned === 1}` for desktop social detail panel and mobile post detail modal. Mobile post header layout refactored: subtitle row split into three divs — (1) `@username` + platform badge + pinned indicator (visible always), (2) date-only row (`sm:hidden`), (3) desktop continuation with `· date · Pinned` (`hidden sm:flex`). Feed navigation fixes: new `useEffect` watches `pinnedCollapsed` — when true and `selectedPost.is_pinned === 1`, auto-selects `regularPosts[0]`. Keyboard navigation `handleKeyDown` now uses `pinnedCollapsed ? regularPosts : posts` for the navigation array, preventing arrow keys from selecting hidden pinned posts. Added `regularPosts` and `pinnedCollapsed` to useEffect dependency array. Files modified: Feed.tsx, PostDetailView.tsx. | | 2026-03-29 | 13.13.0 | **Instagram Authenticated API Toggle, Cookie Management UI, Creator Card Badges** — Minor release. New per-creator `use_authenticated_api` toggle (INTEGER DEFAULT 0 column on `paid_content_creators`): when enabled, Instagram sync in `imginn_adapter.py` `_fetch_all_posts()` uses browser cookies as primary fetch method via `_fetch_via_browser_cookies()`, with automatic fallback to unauthenticated GraphQL if cookies fail. Return type changed from `List[Dict]` to `Optional[List[Dict]]` — `None` = API failed (fall back), `[]` = API worked but nothing new (no fallback). `_notify_cookie_expired()` method on 401: updates `scrapers.last_test_status='failed'` and broadcasts `cookie_health_alert` via `_broadcast_cookie_alert()` from health.py. `scraper.py` passes `use_authenticated_api=bool(creator.get('use_authenticated_api'))` to `get_posts()`. Backend: `use_authenticated_api` added to `UpdateCreatorRequest` model and boolean conversion loop in `paid_content.py`, added to `ALLOWED_CREATOR_COLUMNS` in `db_adapter.py`, added to migration list in `unified_database.py`. Frontend: `use_authenticated_api: number` added to `PaidContentCreator` interface and `updatePaidContentCreator()` params in `api.ts`. `CreatorSettingsModal` in `Creators.tsx` adds toggle (Instagram-only, `creator.platform === 'instagram'`). Creator card badges: green Shield + "Authenticated" badge when `use_authenticated_api` is truthy, purple Filter + "Filtered" badge (with hover tooltip showing `@usernames`) when `filter_tagged_users` JSON array is non-empty. Scrapers page: `instagram_browser` scraper renamed to "Instagram (Authenticated)" in DB. `Scrapers.tsx` adds `api` to `typeOrder` array, `typeLabels` ("Authenticated APIs"), and `TypeBadge` styles/labels (emerald). `mds` service manager script: added `backup` → `cloud-backup-sync` and `backupui` → `backup-central` to `SERVICE_MAP` and `ALL_ALIASES`. Files modified: imginn_adapter.py, scraper.py, paid_content.py, db_adapter.py, unified_database.py, api.ts, Creators.tsx, Scrapers.tsx, scripts/mds. | | 2026-03-28 | 13.12.0 | **Sync Service Button on Creators Page** — Minor release. New `POST /api/paid-content/creators/sync-service/{service_id}` endpoint in `paid_content.py` fetches all enabled creators for a given service_id from database, queues sequential background sync via new `_sync_service_background()` helper (loops through creator IDs calling `scraper.sync_creator()` with per-creator error handling), returns `{status, count, creators}`. Rate limited to 5/minute. Frontend: new `syncPaidContentService(serviceId)` API method in `api.ts` with `syncService` namespace entry. `ServiceGroupedCreators` component in `Creators.tsx` gains `onSyncService` and `syncingServices` props. Multi-service collapsible headers now include a "Sync All" button (RefreshCw icon, text hidden on mobile) with `e.stopPropagation()` to prevent toggling the collapsible. Single-service view adds a header bar with service name, count, and sync button. Button disabled and shows spinning icon when `syncingServices` set contains the service ID or any creator in the service is in `syncingIds`. Parent component adds `syncServiceMutation` (useMutation) with success notification showing queued count and service name. Files modified: paid_content.py, api.ts, Creators.tsx. | | 2026-03-28 | 13.11.2 | **Instagram User ID Fix, Unviewed Count Fix, Bundle Sidebar Default** — Patch release. Instagram `_fetch_user_id()` rewritten in `instagram_client_module.py`: new `_fetch_user_id_from_html()` method is now primary — scrapes `"user_id":"NNNNN"` from profile page HTML using unauthenticated GET request, bypassing 401 errors caused by 77 stale browser cookies with expired `sessionid` that made `_has_valid_session()` return True. `web_profile_info` API kept as fallback. User ID cache persistence fix: `self.unified_db` was being set to `None` when `use_database=False`, preventing `_save_user_id_cache()` from writing to the PostgreSQL `settings` table. Fixed by keeping `unified_db` reference regardless of `use_database` flag. `get_unviewed_posts_count()` in `db_adapter.py` now joins `paid_content_creators` and applies the same `filter_tagged_users` clause used by `get_posts()` and `get_posts_count()` — previously counted all unviewed posts with attachments regardless of per-creator tag filters, causing banner to show 12 when only 5 were visible (7 hidden posts from dimitrishair filter `["nicolescherzinger"]` and makkaroo filter `["kyliejenner"]` had no matching tagged users). Bundle sidebar in `BundleLightbox.tsx` and `PrivateGalleryBundleLightbox.tsx` now defaults to collapsed (`useState(false)`) on all devices instead of auto-opening on desktop. `PrivateGalleryBundleLightbox.tsx` auto-hide effect changed from `setSidebarOpen(!isCurrentVideo && !isMobile)` to only closing for videos, preventing sidebar from re-opening when navigating to images. Files modified: instagram_client_module.py, db_adapter.py, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx. | | 2026-03-27 | 13.11.1 | **Instagram GraphQL Post Fetching, Scan Progress, Feed Page Size** — Patch release. Instagram post sync switched from ImgInn API to direct Instagram GraphQL API (`OwnerToTimelineMediaLoggedOutQuery`) in `imginn_adapter.py` `_fetch_all_posts()`. ImgInn had a server-side 480-post limit (40 pages × 12 items) regardless of client (curl_cffi or FlareSolverr). New flow: `_fetch_all_posts()` creates an `InstagramClientDownloader` instance, calls `fetch_posts()` which paginates GraphQL at 50 posts/page with 3-5s delay. Raw v1 nodes converted to ImgInn-style items via `_ig_node_to_item()` with `_from_ig_api=True` flag. Phase 2 fast-path: items with `_from_ig_api=True` skip ImgInn detail page + tag fetching entirely (already have full-res URLs + usertags from GraphQL). `_best_media_url()` static method extracts highest quality URL from v1 media nodes. Real-time scan progress: `progress_callback(count)` called after each GraphQL page in `fetch_posts()` (line 617), passed from `_fetch_all_posts()` via `_scan_progress` wrapper. UI shows "Fetched N posts..." during scanning phase instead of static "Scanning profile". GraphQL delay increased from `random.uniform(2.0, 4.0)` to `random.uniform(3.0, 5.0)` to avoid HTTP 572 rate limiting on large accounts (tested: 5000+ posts with 3s delay succeeds, <3s hits 572 at ~1100 posts). `max_posts` increased from 5,000 to 15,000 in `instagram_adapter.py`. Update syncs remain efficient: `since_date` → small `days_back` → tight date cutoff, `consecutive_old >= 5` stops pagination after 1-2 pages for recent syncs. Feed.tsx `limit` increased from 30 to 50 for faster mobile initial load. Note: accounts blocking logged-out GraphQL ("Invalid owner id") fall back to authenticated API which requires valid session cookies. Files modified: imginn_adapter.py, instagram_adapter.py, instagram_client_module.py, Feed.tsx. | | 2026-03-26 | 13.11.0 | **Media Gallery Page, GalleryLightbox, Server-Side Shuffle, Timeline Jump Fix** — Minor release. New `/gallery` page replacing `/media` with Immich-style justified thumbnail layout (`buildJustifiedRows()` targeting ~180px row height), daily date grouping with smart labels (Today, Yesterday, weekday names, abbreviated dates), timeline scrubber navigation, slideshow mode, and infinite scroll via `useInfiniteQuery`. New `GalleryLightbox.tsx` (~1000 lines) — simplified BundleLightbox without paid-content features, with slideshow auto-advance, embedded metadata display (ffprobe/exiftool via `/api/media/embedded-metadata` — title, description, artist, source URL), zoom/pan, swipe navigation, keyboard controls. Server-side shuffle for media gallery: new `shuffle` and `shuffle_seed` query params on `/api/media/gallery`, PostgreSQL `ORDER BY md5(fi.id::text \|\| seed::text)` for deterministic randomization, separate `useInfiniteQuery` for shuffled results, shuffle button (cyan when active) in lightbox slideshow toolbar. Timeline scrubber date jump fix for both galleries: now sets both `date_from` AND `date_to` (last day of target month) to bound the query, preventing DESC sort from returning newest items when jumping to an old month. `TimelineScrubber.tsx` fixed all `=== activeKey` comparisons to `activeKey.startsWith(key)` for daily section key compatibility (keys are `YYYY-MM-DD`, scrubber uses `YYYY-MM`). Bug fixes: added `autoPlayVideo={true}` to paid content gallery BundleLightbox props; feed unviewed count query adds `EXISTS (SELECT 1 FROM paid_content_attachments)` to exclude empty posts; media gallery date-range endpoint changed `fi.location = 'media'` to `'final'`; removed non-existent `fi.duration` from gallery queries. Route changes: `/media` redirects to `/gallery`, nav/breadcrumbs/feature toggles updated. Files created: Gallery.tsx, GalleryLightbox.tsx. Files modified: media.py, GalleryTimeline.tsx, TimelineScrubber.tsx, App.tsx, api.ts, breadcrumbConfig.ts, Configuration.tsx, Features.tsx, private_gallery.py, db_adapter.py. | | 2026-03-26 | 13.10.0 | **Justified Gallery Layout, Timeline Navigation, Service Health Check Fixes** — Minor release. Paid content gallery now uses justified thumbnail rows (Immich-style) where images display at their natural aspect ratios instead of uniform squares. `buildJustifiedRows()` function packs items into rows targeting ~180px height, scaling widths to fill container exactly. New `JustifiedSection` React.memo component isolates per-section rendering. Timeline scrubber `onJumpToDate` callback: clicking a month not yet loaded via infinite scroll triggers `date_from` API query parameter, resetting infinite scroll to load from that month. Dismissible "Showing from {Month Year}" chip with X button clears filter. Timeline layout changed from centered `top-1/2 -translate-y-1/2 max-h-[70vh]` to full-height `top-16 bottom-4`. Three performance optimizations: (1) callback ref (`useCallback` ref) replaces `useRef` + `useLayoutEffect` for container width measurement — fires synchronously when conditionally-rendered div mounts, eliminating blank first frame; (2) `sectionRowsMap` useMemo pre-computes all justified rows keyed on `[sections, containerWidth]` instead of computing per-section inside render loop; (3) ResizeObserver deadband ignores sub-pixel width changes (threshold >1px). Backend: added dedicated health checks for coppermine (HTTP client availability), besteyecandy (site reachability with Cloudflare 403 acceptance), and reddit (gallery-dl binary check) in `_check_single_service_health()` — these were falling through to generic `PaidContentAPIClient.check_health()` which tried `{base_url}/api/v1/creators` (Coomer/Kemono pattern), producing false down/degraded status. Files modified: TimelineScrubber.tsx, GalleryTimeline.tsx, paid_content.py. | | 2026-03-24 | 13.9.0 | **Press Monitoring, Cloud Backup, BestEyeCandy/Coppermine/Reddit Scrapers, Feed Shuffle** — Minor release. New press monitoring system: celebrity news tracking via Google News RSS + GDELT with full article extraction (readability + BeautifulSoup), local image caching with proxy serving, Pushover notifications, read/unread tracking, filtering by celebrity/domain/read status, full-text search. Dedicated Press page (Press.tsx, 422 lines) with stats dashboard, article cards, and PressArticleModal (292 lines) with HTML sanitization. New cloud backup system: rclone-based backup to B2/S3 with AES256 encryption, cloud_backup.py router (1526 lines), cloud_backup_sync.py daemon (1194 lines) with inotify file watching, per-directory cooldown sync, daily full backups at 3AM with Immich + Media Downloader PostgreSQL dumps, live progress tracking. Dashboard widget shows real-time sync progress with animated gradient border, transfer speed/ETA, phase detection. Three new paid content scrapers: BestEyeCandy client (469 lines) with cookie auth and URL pattern discovery for full-res images; Coppermine client for nested PHP gallery structure (categories > subcategories > albums > photos) with thumbnail-to-fullres URL derivation; Reddit client via gallery-dl integration with subreddit metadata API. Backend-driven shuffle for paid content slideshow: deterministic seed-based randomization in db_adapter.py (random.Random(seed) shuffles all matching IDs, then paginates), frontend mulberry32 seeded PRNG for cross-post attachment mixing. BundleLightbox.tsx gains onShuffleChange/isShuffled props for parent-managed shuffle. Feed page size increased 10→30 fixing mobile feed empty state (20 pinned posts filled first 2 pages at old limit with collapsed pinned section). Feed error handling improved with isPending tracking, retry count 3, error UI in all 4 view modes. Feed setSearchParams skipped on initial mount to prevent startTransition interference. Paid content feed endpoint limit max increased 100→500. Log viewer adds cloudbackup, cloudflare.fastdl/imginn/imginn-stories, taskcheckpoint components. Creators page adds hqcelebcorner/picturepub to SERVICE_URLS. New scripts: backfill_press.py (592 lines) for historical article bulk loading with sliding windows, cloud_backup_sync.py daemon. Configuration.tsx adds cloud_backup and press tabs. App.tsx adds Press route. Breadcrumb config adds /press. Cleaned __pycache__ directories. Files created: press.py, cloud_backup.py, besteyecandy_client.py, coppermine_client.py, reddit_client.py, backfill_press.py, cloud_backup_sync.py, Press.tsx, PressArticleModal.tsx. Files modified: db_adapter.py, scraper.py, utils.py, paid_content.py, __init__.py, Feed.tsx, BundleLightbox.tsx, api.ts, Dashboard.tsx, Configuration.tsx, App.tsx, Logs.tsx, breadcrumbConfig.ts, Creators.tsx. | | 2026-03-19 | 13.8.1 | **Audio Player, Push Notification Accuracy, Reddit Reliability, Health Check Fixes** — Patch release. Inline audio player in PostDetailView (native HTML5 audio with filename, file size, playback controls) and mobile Feed PostCard (compact player with stopPropagation preventing post navigation on tap). Audio attachments excluded from thumbnail grid and BundleLightbox. Push notification audio fix: content_type from scraper metadata checked before file extension, fixing .mp4 audio files misidentified as video; added 🎵 audio category to titles and pushover PLURALS/CONTENT_ICONS; fixed 3 hardcoded `content_type: 'video'` in downloaded_file_info. Health check fixes: HQCelebCorner/PicturePub accept HTTP 200/403, Snapchat checks story.snapchat.com reachability. Reddit community monitor: switched from OAuth to REST API mode (`extractor.reddit.api=rest`) with `--range 1-200` limit; fixed cookie file directory creation. Code quality: fixed function name shadowing in celebrity.py and video_queue.py; fixed backup profile stale paths. Files modified: scraper.py, pushover_notifier.py, reddit_community_monitor.py, paid_content.py, celebrity.py, video_queue.py, PostDetailView.tsx, Feed.tsx, install.sh, add-backup-profile.sh. | | 2026-03-18 | 13.8.0 | **XenForo Forum Support, Snapchat/TikTok HTTP Clients, Cookie Health Monitoring, Story Dedup** — Minor release. New XenForo forum client (xenforo_forum_client.py, hqcelebcorner_client.py) for celebrity image forums with external host resolution, FlareSolverr bypass, CSRF handling. Snapchat HTTP client (snapchat_client.py) with curl_cffi replacing Playwright, __NEXT_DATA__ JSON extraction, EXIF metadata. TikTok hybrid client (tiktok_client.py) with yt-dlp listing + gallery-dl downloading, lazy cookie loading. Instagram direct API module (fastdl_instagram_client.py) with GraphQL + REST via curl_cffi. Cookie health monitoring with per-platform toggles, 30-min cooldown, WebSocket + Pushover alerts. ImgInn adapter: two-phase discovery, FlareSolverr fallback, 1440p+ CDN URLs, NordVPN proxy rotation. Scheduler checkpoint system for crash recovery. Bellazon forum client, Soundgasm audio client, paid content URL parser expansion. Bug fixes: FastDL pk mapping, hash-based story dedup, recycle bin metadata, square video display, failed downloads creator_db_id. Files modified: 20+ files across modules/paid_content/, web/backend/routers/, web/frontend/src/. | | 2026-03-12 | 13.7.0 | **Instagram Sync Settings, Appearance Role Combining, Tagged User Filter** — Minor release. Per-creator Instagram sync settings: new `sync_posts`, `sync_stories`, `sync_highlights` columns on `paid_content_creators` table with toggle UI in creator card menu (Instagram-only). Conditional sync in `scraper.py` wraps stories/highlights/posts fetching in `creator.get('sync_X', 1)` checks. Tagged user filter: new `filter_tagged_users` column (JSON array of usernames), new `GET /creators/{id}/tagged-users` endpoint returning distinct tagged usernames with counts, LIKE-based SQL filter in `get_posts()`, `get_posts_count()`, and `get_media_count()`. CreatorSettingsModal component in Creators.tsx with toggles and multi-select tagged user list. All paid content Instagram operations now route exclusively through `ImgInnAPIAdapter` — removed all `UnifiedInstagramAdapter` usage from scraper.py (was routing through real Instagram API via `instagram_client`). curl_cffi impersonation pinned from auto-resolving `chrome` (→chrome142, unsupported) to explicit `chrome136`. Appearances: upcoming endpoint now uses ShowStats CTE with `GROUP_CONCAT(DISTINCT credit_type)` and `ROW_NUMBER()` grouping matching aired endpoint. New `_enrich_with_roles()` helper fetches structured role data (credit_type, character_name, job_title) for multi-role appearances. Show episodes endpoint groups by season+episode+date, combining credit types. `COUNT(*)` → `COUNT(DISTINCT episode_key)` across all 4 endpoints. AppearanceDetailModal updated for multi-role display with consistent `creditTypeConfig` badge colors. Dashboard UpcomingAppearancesCard shows credit type badges. Notification INSERTs now set `is_sent=1, sent_at=CURRENT_TIMESTAMP`. FilterPopover max-height increased 200→320px. Removed unused `instagram_unified_adapter.py` (155 lines). Cleaned __pycache__ directories. Files modified: scraper.py, imginn_api_module.py, db_adapter.py, appearances.py, paid_content.py, unified_database.py, api.ts, Creators.tsx, AppearanceDetailModal.tsx, UpcomingAppearancesCard.tsx, FilterPopover.tsx. | | 2026-03-07 | 13.6.0 | **ImgInn Paid Content Adapter, Reddit Timezone Fix, Lightbox Improvements, Cleanup** — Minor release. New `imginn_adapter.py` replaces Instagram API-based adapter with ImgInn-based scraping for Paid Content — no Instagram credentials needed, maps ImgInn API data to existing Post/Attachment models, handles posts/stories/reels with CDN URLs, built-in rate limiting and Cloudflare bypass. ImgInn API optimizations: FlareSolverr integration for Cloudflare bypass, OCR disabling for faster scraping, date pre-filtering, profile resolution fixes. Reddit private gallery timezone fix: gallery-dl UTC dates now properly converted to local time using `datetime.timezone`; created_at uses import time instead of post date. Bulk-fixed 556 existing reddit posts and 1366 media entries with incorrect UTC dates via decrypt→convert→re-encrypt cycle. Lightbox fixes: slideshow back arrow not showing (parent-managed shuffle mode), paid content bundle sidebar z-index fix (z-20 over z-10 overlay), bundle sidebar auto-collapse on slideshow, hidden for single-image posts, expand/collapse floating button, stationary slideshow controls. Dashboard platform labels improved. Cleanup: removed 36 stale scripts/utilities (one-off backfills, Fansly maintenance, test scripts, completed migration scripts), legacy SQLite database file, archive directory, __pycache__ directories. | | 2026-03-05 | 13.5.1 | **Landscape Lightbox, Min Resolution Filter, TypeScript Fixes, Cleanup** — Patch release. Ported landscape phone detection and layout from BundleLightbox.tsx to PrivateGalleryBundleLightbox.tsx: touch-based device detection with `isLandscape` state (touch + height<500px), fixed positioning for landscape containers (100dvh), adjusted control positions/visibility, landscape-specific navigation arrows (compact rounded with bg-black/40), sidebar-aware image container (`left: 6rem`), landscape video sizing. Added per-person-group `min_resolution` setting: SQL filter `(file_type != 'image' OR (width >= ? AND height >= ?))` shows all videos but only images meeting threshold; gallery filter dropdown (300/500/720/1080px+ options) with active filter chip; import skip for low-resolution images with skipped count tracking. Fixed TypeScript errors: added `min_resolution?: number` to `getPersonGroups()` and `getPersonGroup()` return types in api.ts, removed unused `editMinRes` state from Config.tsx, fixed `task.next_run` null safety in Scheduler.tsx. Cleaned up stale files: removed migration_backup.json (682KB), media_downloader.db.old (308KB), all __pycache__ directories (~5MB). Files modified: PrivateGalleryBundleLightbox.tsx, private_gallery.py, Gallery.tsx, Config.tsx, api.ts, Scheduler.tsx. | | 2026-03-04 | 13.5.0 | **ImgInn API Module, Instagram Auth Circuit Breaker, Private Gallery Bridge, FastDL Consolidation** — Minor release. New `imginn_api_module.py` (~600 lines) scraper using ImgInn's internal API for posts, stories, reels, and tagged content with subprocess wrapper. New `instagram_rate_limiter.py` (~160 lines) providing shared 180 calls/hour rolling window rate limiter with progressive slowdown, pause file persistence, and cross-module mutex. New `scraper_gallery_bridge.py` (~650 lines) for importing scraper downloads into private gallery with id-based incremental filtering via `last_imported_file_id` column on `private_media_scraper_accounts`. New `paid_content/fastdl_instagram_client.py` replacing unstable curl_cffi direct API with FastDL browser-based approach for Paid Content Instagram. New `fastdl_subprocess_wrapper.py` and `imginn_api_subprocess_wrapper.py` for subprocess isolation. Instagram auth failure circuit breaker: `instagram_client_module.py` sets `auth_failed` flag on 401/403, propagated through subprocess JSON results via `base_subprocess_wrapper.py`; `media-downloader.py` `download_instagram_client()` detects flag, skips stories/tagged for remaining users, continues posts/reels, updates scraper test status to 'failed' in database, sends Pushover notification. Cookie health banner updated: Instagram scrapers now shown in `/health/cookies` when test status is 'failed' (previously fully excluded from display); auto-clears when cookies re-saved. Dashboard "New In" cards fixed: all queries changed from `ORDER BY COALESCE(d.download_date, fi.created_date) DESC` to `ORDER BY fi.id DESC`; dismiss tracking changed from timestamp-based to ID-based (`number | null`). Private gallery bridge added to `download_instagram_client()` (was only wired for fastdl/imginn_api). `get_available_accounts()` now collects `phrase_search.usernames` from all Instagram scrapers. Instagram client TLS: changed from `chrome131` to `edge101` impersonation with matching Edge User-Agent. Subprocess timeout increased 120→300s. FastDL consolidated from 3 browser instances to 1 per user. FastDL scrolling tuned (200px steps, 3s API wait, 5 stale rounds). Cleanup: removed 5 debug scripts, debug screenshot directory, dead `_load_instaloader_session()`, unused imports (`re`, `Set`, `datetime`). Files modified: imginn_api_module.py (NEW), instagram_rate_limiter.py (NEW), scraper_gallery_bridge.py (NEW), paid_content/fastdl_instagram_client.py (NEW), fastdl_subprocess_wrapper.py (NEW), imginn_api_subprocess_wrapper.py (NEW), media-downloader.py, instagram_client_module.py, fastdl_module.py, move_module.py, unified_database.py, scheduler.py, base_subprocess_wrapper.py, dashboard.py, health.py, paid_content.py, platforms.py, private_gallery.py, scrapers.py, config.py (router), api.ts, Dashboard.tsx, Configuration.tsx, Platforms.tsx, Scheduler.tsx, Feed.tsx, Creators.tsx, Config.tsx (private-gallery). | | 2026-02-26 | 13.4.1 | **Private Gallery Unread Fixes, Paid Content Unviewed/Messages Banners** — Patch release. Fixed three Private Gallery unread bugs: (1) `/media` endpoint count/shuffle/fast-path queries missing `LEFT JOIN private_media_posts` causing SQL errors when `unread_only=true`, (2) `markSeenMutation.onSuccess` used `invalidateQueries` instead of `resetQueries` leaving stale `useInfiniteQuery` pagination, (3) upload modal didn't pass `post_id` through `onSuccess` callback so new post wasn't auto-selected. Added Paid Content unviewed posts system: `get_unviewed_posts_count()` and `mark_all_posts_viewed()` db_adapter methods, `/unviewed-count` and `/mark-all-viewed` endpoints, Feed.tsx banner with "View unviewed"/"Mark all viewed" buttons using blue gradient+border style. Added Paid Content unread messages system: `get_total_unread_messages_count()` and `mark_all_messages_read()` db_adapter methods, `/messages/unread-count` and `/messages/mark-all-read` endpoints (placed before `/{creator_id}` path param route), violet gradient banners on Dashboard.tsx and Feed.tsx with "View messages" RouterLink and "Mark all read" button. Private Gallery unread banners restyled from solid to gradient+border matching dashboard pattern. CLAUDE.md updated with PostgreSQL syntax note for direct psql commands. Files modified: private_gallery.py, Gallery.tsx, PrivateGalleryUploadModal.tsx, paid_content.py, db_adapter.py, api.ts, Feed.tsx, Dashboard.tsx, CLAUDE.md. | | 2026-02-23 | 13.4.0 | **Instagram Client API, Module Enable/Disable, Image Info Bar, Snapchat Client** - Minor release. New `instagram_client_module.py` (1,651 lines) provides direct Instagram GraphQL/REST API downloads using `curl_cffi` with browser TLS fingerprinting — 10-20x faster than ImgInn scraping. Posts via public GraphQL (`OwnerToTimelineMediaLoggedOutQuery_connection`), stories/reels/tagged via authenticated REST API with session cookies. File naming matches ImgInn format (`{profile}_{YYYYMMDD_HHMMSS}_{media_id}{ext}`) for dedup compatibility. Separate `paid_content/instagram_client.py` (1,081 lines) for Paid Content system. New `snapchat_client_module.py` with same direct API architecture. Both integrated into main downloader (`media-downloader.py`), scheduler, and platform router with subprocess wrappers. New module enable/disable system: platform-level `enabled` flags in config, scheduler filters by `config.get('', {}).get('enabled')`, health router `_is_scraper_module_enabled()` checks module status, frontend Scheduler.tsx filters `enabledTasks`. Image info bar added to `EnhancedLightbox.tsx` — images now show source, file size, date, resolution (emerald highlight), and platform below the image (matching video info bar, minus Duration). New `CookieHealthBanner.tsx` component for cookie expiration warnings via WebSocket. Major cleanup: removed 7.5 GB `backup_sqlite_migration/` directory, empty .db placeholders, 365 debug screenshots (19 MB), stale test logs, __pycache__ directories. Files modified: instagram_client_module.py (NEW), snapchat_client_module.py (NEW), paid_content/instagram_client.py (NEW), instagram_client_subprocess_wrapper.py (NEW), snapchat_client_subprocess_wrapper.py (NEW), CookieHealthBanner.tsx (NEW), media-downloader.py, scheduler.py, platforms.py, scrapers.py, health.py, EnhancedLightbox.tsx, Dashboard.tsx, Scheduler.tsx, Logs.tsx, Configuration.tsx, api.ts, Settings.tsx, Feed.tsx, Creators.tsx, Gallery.tsx, move_module.py. | | 2026-02-21 | 13.3.0 | **Reddit Community Monitor, Private Gallery Fixes, Log & Scheduler Standardization** - Minor release. New `reddit_community_monitor.py` module (~1,050 lines) enables automated Reddit community monitoring for Private Gallery. Users map subreddits to persons via Config tab; system uses gallery-dl to download new posts (including imgur/redgifs attachments), encrypts files with AES-256-GCM, and imports into gallery with 'reddit' tag. Two-layer duplicate detection: post-level tracking via `private_media_reddit_history` table (survives post deletion) and file-level SHA-256 hash checking. Crypto key file approach (`/opt/immich/private/.reddit_monitor_key`) enables scheduler process to encrypt independently. Real-time progress tracking with phase updates (downloading/processing/importing/encrypting). Batch subreddit input (comma-separated). Auto-cleanup of empty reddit-tagged posts after dedup deletion. Fixed Content-Disposition headers for filenames with special characters (emojis, smart quotes, double quotes) using RFC 5987 `filename*=UTF-8''` encoding with `quote(filename, safe='')`. Fixed `total_media_found` showing accumulated count instead of live COUNT(*). Standardized scheduler task ID to `reddit_monitor`. Added `filesrouter`/`mediaidentifier`/`redditmonitor` to log viewer with display names and default components. Dashboard and Scheduler pages updated with Reddit Monitor platform mapping. DB: 2 new tables (`private_media_reddit_communities`, `private_media_reddit_history`), 3 new indexes. Files modified: reddit_community_monitor.py (NEW), private_gallery_crypto.py, private_gallery.py, unified_database.py, scheduler.py, api.ts, Config.tsx, Logs.tsx, Scheduler.tsx, Dashboard.tsx. | | 2026-02-16 | 13.2.0 | **Profile Image Caching, FastDL Stories, TikTok Bio Fix, Log Viewer Fix** - Minor release. New `_cache_profile_image()` method in scraper.py downloads creator avatars/banners locally during sync across all 11 platforms (Instagram, Coomer/Kemono, YouTube, PornHub, XHamster, TikTok, Twitch, Fansly add/sync, OnlyFans add/sync), eliminating dependency on expiring CDN URLs. Uses `curl_cffi` with browser TLS impersonation for Instagram CDN (plain requests blocked). New `/api/paid-content/cache/profile-image/{filename}` endpoint serves cached images with 7-day browser cache. Instagram stories switched from ImgInn to FastDL for real post timestamps (`published_at` from `p.media-content__meta-time`), ImgInn kept as fallback. FastDL's duplicate tracking disabled (`use_database=False`) since paid content has own deduplication. `_files_to_posts()` regex updated for both FastDL/ImgInn filename formats. TikTok bio emoji corruption fixed: replaced `.encode().decode('unicode_escape')` with `json.loads()` and added `__UNIVERSAL_DATA_FOR_REHYDRATION__` JSON parsing as primary extraction. Log viewer `get_logs` endpoint fixed to handle `YYYYMMDD_component.log` format (was only matching `YYYYMMDD_HHMMSS_component.log`). Messages.tsx proxy wrapper fixed to skip local `/api/` URLs. Instagram profile always refreshes during sync (removed stale guard). Added `curl_cffi>=0.7.0` to requirements.txt and dependency updater. Added `data/cache/profile_images/` to installer. Files modified: scraper.py, instagram_client.py, tiktok_client.py, paid_content.py, config.py (router), Messages.tsx, requirements.txt, dependency_updater.py, install.sh, update-all-versions.sh, core/config.py. | | 2026-02-16 | 13.1.1 | **PostgreSQL Boolean Fixes, Fansly Quality Recheck Scheduler, Dashboard Pinned Posts** - Patch release. Fixed celebrity appearances ON CONFLICT error by creating partial unique indexes for TV and Movie appearance types. Fixed forum subprocess boolean handling (`1` → `TRUE` for `searches.active`, `downloaded`, `active`). Added missing `download_queue` columns to PostgreSQL (`thread_id`, `post_id`, `forum_name`, `file_hash`, `downloaded_date`). Converted `paid_content_posts.downloaded` and `forum_posts.has_images` from integer to boolean; added both to pg_adapter `_BOOLEAN_COLUMNS`. Fixed FastDL URL redirect (`/en` → `/en2`). Fixed Fansly quality recheck scheduler: `_auto_quality_recheck_background()` was only called from API single-creator sync, not from `sync_paid_content_all()`. Added `skip_pinned` parameter to feed endpoints so Dashboard recent posts excludes pinned posts. Synced 11 missing fastdl/imginn accounts from SQLite to PostgreSQL. Files modified: appearances.py, forum_downloader.py, fastdl_module.py, pg_adapter.py, db_adapter.py, paid_content.py, Dashboard.tsx, api.ts, install.sh. | | 2026-02-15 | 13.1.0 | **PostgreSQL Compatibility Fixes, Discovery Queue, Security Hardening** - Minor release. Comprehensive fixes for 15+ PostgreSQL edge cases: `db_bootstrap.py` loads `.env` before checking `DATABASE_BACKEND` (fixes all 5 entry points silently falling back to SQLite). pg_adapter datetime parameter interval translation fixed (`%s` double-escaped to `%%s` breaking forum thread monitoring). `download_monitor.success` converted from integer to boolean, `SUM(success)` rewritten as `SUM(CASE WHEN...)`. Analytics `DATE()` returns Python date objects (added `str()` conversion). Paid content quality recheck `text < timestamp` fixed with explicit CAST. Passkey/2FA `username` column→`user_id`, schema type integer→text. `error_tracking` migration replaced sqlite_master check with `information_schema` constraint check. Review page and media gallery correlated subqueries replaced with pre-aggregated LEFT JOINs. Recycle bin COALESCE type mismatch fixed. Forum threads GROUP BY rewritten with proper subquery JOIN. `strftime()` → `TO_CHAR()` and `datetime(col)` → `col::timestamp` translations added to pg_adapter. `fromisoformat()` now handles datetime objects from PostgreSQL. Celebrity appearances boolean migration fixed. Duplicate `/api/logs/context` route removed. 5 missing DiscoverySystem queue methods added (`get_queue_stats`, `get_pending_queue`, `add_to_queue`, `bulk_add_to_queue`, `clear_queue`). Image proxy switched to suffix-based domain matching adding Fansly/TikTok/xHamster CDN support. SQL injection defense: server-side `sort_order` and `table_alias` validation. Boolean column translation (`has_match`/`success`/`notified` = 1 → = TRUE). Systemd services include DATABASE_BACKEND env vars. Files modified: pg_adapter.py, db_bootstrap.py, downloader_monitor.py, discovery_system.py, unified_database.py, passkey_manager.py, scheduler.py, paid_content.py, Creators.tsx, downloads.py, private_gallery.py, core/utils.py, config.py, review.py, media.py, db_adapter.py, CLAUDE.md. | | 2026-02-13 | 13.0.1 | **Dashboard Text-Only Post Placeholder Fix** - Patch. Paid Content Dashboard grey media placeholder no longer renders on text-only posts (has_attachments=0). Added `post.has_attachments === 1` condition gate. Files modified: Dashboard.tsx. | | 2026-02-13 | 13.0.0 | **PostgreSQL Migration, Private Gallery Fix, Code Cleanup** - Major infrastructure migration. New `pg_adapter.py` (~840 lines) provides drop-in `sqlite3` replacement via `sys.modules` monkey-patching when `DATABASE_BACKEND=postgresql`. SQL translation engine with LRU cache (4096 entries) handles 27+ SQLite→PostgreSQL dialect conversions (PRAGMA→no-op, datetime()→NOW(), GROUP_CONCAT→STRING_AGG, INSERT OR REPLACE→ON CONFLICT, json_extract→jsonb, etc.). `_replace_question_marks()` converts `?` params to `%s` with proper string-literal awareness. `PgConnection`/`PgCursor` classes wrap psycopg2 with `Row` class providing full sqlite3.Row compatibility (index, name, keys, iteration). `ThreadedConnectionPool` (2-30 connections) with lazy init and double-check locking. `db_bootstrap.py` patches `sys.modules['sqlite3']` at import time in all 5 entry points. All 87 tables from 6 SQLite databases (media_downloader.db, auth.db, scheduler_state.db, thumbnails.db, media_metadata.db, easynews_monitor.db) migrated to single PostgreSQL instance. Bug fixes: `%` escaping in pg_adapter (was only escaping outside quotes, psycopg2 needs all `%` escaped), cursor leak in `commit()`/`rollback()` (cursors created but never closed), removed unused imports (datetime, psycopg2.extras), added explicit `psycopg2.errors` import. Fixed Private Gallery feature toggle 400 error (stale `/scraping-monitor` in saved config rejected by validation; now silently filters removed features). Added `_is_lock_error()` helper in unified_database.py consolidating 9 inline lock-error checks to support both SQLite and PostgreSQL errors (deadlock detected, could not obtain lock). Added psycopg2-binary to requirements.txt and dependency updater. Updated installer with PostgreSQL setup. Cleaned up 6 empty .db files, stale CSV, moved sync_pg_schema.py to archive. Files modified: pg_adapter.py (NEW), db_bootstrap.py (NEW), unified_database.py, private_gallery.py, activity_status.py, requirements.txt, dependency_updater.py, install.sh, media-downloader.py, api.py, thumbnail_cache_builder.py, generate-embeddings.py, enrich_celebrity_metadata.py, CRITICAL_RULES.md, media-cache-builder.service. | | 2026-02-12 | 12.14.0 | **Streaming Decryption, Download Reliability, Auto-Migration** - Minor release. Generator-based streaming decryption for all encrypted videos: `decrypt_file_generator()` yields 8MB decrypted chunks for chunked files or entire content for single-shot files via `StreamingResponse`, eliminating memory spikes for multi-GB videos. `decrypt_file_range_generator()` calculates which encrypted chunks overlap a byte range, seeks to positions, decrypts only needed chunks. Updated `stream_video`, `get_file`, and `export_single` endpoints to use streaming. Auto-migration on gallery unlock: background thread converts single-shot encrypted files >50MB to chunked format via `re_encrypt_to_chunked()` with atomic rename and unique temp filenames (`secrets.token_hex(4)`). Once-per-process guard prevents duplicate migrations. `POST /migrate-chunked` endpoint for manual triggering. Download reliability: stall detection with 30s read timeout, 3 retries with HTTP Range resume from last byte. All direct downloads now multi-threaded with 5 parallel segments (removed 5MB threshold). Fixed subprocess stdin double-close: replaced `stdin.write()`/`stdin.close()` + `communicate()` with `communicate(input=...)`. Fixed duplicate filename check to verify `.enc` file exists on disk. Fixed `sync_tmdb_appearances` NoneType error in scheduler context: added `db` parameter, scheduler passes `self.unified_db`. Subprocess error→warning for FastDL/ImgInn/Toolzu/Snapchat timeouts/failures. Smart escalation for page load failures: warning normally, error after 5+ per session via `_page_load_failures` counter. Files modified: private_gallery_crypto.py, private_gallery.py, media-downloader.py, imginn_module.py, appearances.py, scheduler.py. | | 2026-02-11 | 12.13.0 | **Security Hardening, Message Read Tracking, Easynews Fix** - Minor release. Comprehensive 12-point security audit across backend routers: fixed missing `require_admin` dependency in appearances.py, added CSRF enforcement to config/manual_import/files/video routers, corrected `get_connection(for_write=True)` on 8 write operations across celebrity.py/stats.py/dashboard.py/easynews.py/private_gallery.py/review.py, added path traversal validation to files.py serve endpoint, added rate limiting to paid_content.py sync/setup endpoints. New mark-as-read feature: IntersectionObserver on unread message bubbles (0.5 threshold) with 500ms debounced batch API calls via `POST /messages/{creator_id}/mark-read`, `mark_messages_read()` DB method, `markMessagesRead()` API client. Auto-scroll to first unread message on conversation open. Message sync now calls `_download_message_attachments()` for pending attachments after sync. Fixed Easynews search: removed strict content-type check in easynews_client.py that rejected valid JSON when Easynews API changed Content-Type from `application/json` to `text/html`. Fixed Dashboard dismiss button: added `parseAsUTC()` helper normalizing timezone-naive DB timestamps (e.g. `2026-02-11 12:25:21`) to UTC before comparing with ISO dismiss timestamps (`...Z`), fixing timezone mismatch in shouldShowCard/getVisibleItems. Health check fixes: auto-check persists timeout/error to DB, single service handles timeout, stores 'down' not 'error', first load runs checks synchronously. Files modified: appearances.py, celebrity.py, config.py, dashboard.py, easynews.py, files.py, manual_import.py, paid_content.py, private_gallery.py, review.py, stats.py, video.py, api.py, easynews_client.py, db_adapter.py, Dashboard.tsx, Messages.tsx, Creators.tsx, api.ts, App.tsx, Configuration.tsx, Login.tsx. | | 2026-02-10 | 12.12.1 | **Health Check Reliability, Message Notifications, Sync Pagination Fix** - Patch release. Health checks no longer block async event loop: rate limiter delays disabled for health check clients via `_init_rate_limiter(min_delay=0)`, 30s per-service timeout with `asyncio.wait_for()`. Refactored 3 duplicate health check implementations into shared `_check_single_service_health()` function (~150 lines removed). Fixed OnlyFans scheduled sync paginating through ALL posts: timezone-aware vs timezone-naive datetime comparison in `get_posts()` silently failed in except block, now normalizes both to naive with `.replace(tzinfo=None)`. Added push notifications for new messages during paid content sync: `_send_creator_notification()` accepts `new_messages` param, shows "💬 N New Messages" title for message-only syncs at low priority (-1), includes "💬 Messages: N" line in combined download notifications. PiP menu item hidden on desktop lightboxes (both Paid Content and Private Gallery), only shown on mobile. Files modified: scraper.py, onlyfans_client.py, paid_content.py, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx. | | 2026-02-10 | 12.12.0 | **Messages & Chat, OnlyFans Direct Setup, Health Check Auto-Trigger** - Major feature release. New Messages/Chat page for viewing OnlyFans and Fansly creator direct messages with two-panel responsive chat UI (conversation list + chat thread), infinite scroll pagination, PPV/tip badges, and inline media lightbox. New `paid_content_messages` database table with `message_id` column on attachments. Message sync via OnlyFans `GET /chats/{user_id}/messages` and Fansly `GET /messaging/groups` + `GET /message?groupId={id}` APIs. New `Message` dataclass in models.py. DB adapter methods: `upsert_message()`, `get_conversations()`, `get_messages()`, `get_pending_message_attachments()`. OnlyFans Direct credential setup UI supporting manual cookie paste, HAR file upload, and Cookie-Editor JSON import with `POST /onlyfans-direct/verify-auth` and `POST /onlyfans-direct/sync` endpoints. Fansly Direct verify-auth and sync endpoints. Picture-in-Picture support in both BundleLightbox.tsx and PrivateGalleryBundleLightbox.tsx with Safari `webkitSetPresentationMode` fallback. Auto health checks: `GET /services` now triggers background checks when stale (>5min) via `asyncio.create_task()` with throttle. Mobile video unmute fix (start muted for autoplay policy, unmute after play()). Service worker caches Vite hashed JS bundles (cache-first). Multi-word search with COLLATE NOCASE in paid content and private gallery. Push notification platform names show "Fansly"/"OnlyFans" not "Fansly Direct"/"Onlyfans Direct" via `_get_platform_display_name()`. Feature enable/disable fix with `known_features` tracking. Search count functions (get_posts_count, get_media_count) fixed to include search/date filters. Settings page polls services every 60s. Cleaned up scheduler.py.migrated to archive. Files modified: unified_database.py, models.py, db_adapter.py, onlyfans_client.py, fansly_direct_client.py, scraper.py, paid_content.py, private_gallery.py, api.ts, App.tsx, Messages.tsx (NEW), Settings.tsx, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx, sw.js, install.sh. | | 2026-02-08 | 12.11.1 | **Lightbox Navigation Fixes & Gallery Filter Fix** - Lightbox arrow buttons, keyboard arrows, and swipe gestures now navigate in shuffled order when shuffle mode is enabled (was always navigating in date order). Added shuffle-aware `navigatePrev()`/`navigateNext()` functions replacing `handleManualNavigate()`. Navigation arrows now render above video player (z-index raised from z-10 to z-20). Desktop arrows auto-hide with `text-transparent hover:text-white` pattern matching info/close buttons. Widened arrow hover detection from small circular `p-2` targets to tall rectangular edge zones `px-6 py-16` anchored to screen edges. Videos autoplay during slideshow in Paid Content lightbox. Private Gallery file_type filter now filters individual media items within posts (was only filtering which posts to show, returning all media types within matched posts). Code review fixes: added EXIF date extraction to URL import background task, fixed asyncio.run() → asyncio.new_event_loop() in background tasks, progress modal title shows 'Processing Imports' for URL imports, session keepalive runs during URL imports. Files modified: PrivateGalleryBundleLightbox.tsx, BundleLightbox.tsx, PrivateGalleryUploadModal.tsx, private_gallery.py. | | 2026-02-08 | 12.11.0 | **Config Page Redesign & Multi-URL Import** - Private Gallery Config page redesigned from sidebar + pill tabs to horizontal underline tabs matching Configuration.tsx and Paid Content Settings.tsx pattern. Removed sticky header with back arrow, replaced with standard h1 + subtitle. Content area spans full tab width. Page root changed from own container to `space-y-6` div matching parent layout. Upload modal now supports multi-URL import: textarea for pasting URLs (one per line), supports Bunkr/Pixeldrain/Gofile/Cyberdrop via FileHostDownloader + direct HTTP fallback. New `importUrlsMutation`, URL count display, Import/Upload/Create Post button switching. Backend: fixed ImportUrlRequest model (tag_ids now optional), added POST /import-urls endpoint, added `_import_urls_to_gallery_background()` background task with per-file download→hash→dedup→encrypt→thumbnail→DB insert pipeline. API client: added `importUrls` method, added 'import' to job operation types. Files modified: Config.tsx, PrivateGalleryUploadModal.tsx, api.ts, private_gallery.py. | | 2026-02-08 | 12.10.0 | **Slideshow Mode & Gallery Improvements** - Added slideshow mode to both Private Gallery (`PrivateGalleryBundleLightbox`) and Paid Content (`BundleLightbox`) lightboxes with auto-advance, configurable interval (3s/5s/8s/10s), Fisher-Yates shuffle mode, and video-aware advancement. Slideshow toolbar button in both gallery pages is context-aware: gallery view slideshows all items across all posts, feed/social view slideshows the selected post only. Added `total_media` count to both Private Gallery and Paid Content backend APIs for accurate position indicators. Filter bar now shows "X items" in gallery view and "X posts" in feed/social view. Fixed gallery item ordering mismatch (galleryLightboxItems now sorts by media_date matching grid). Fixed Paid Content gallery not auto-loading pages on filter change. Added scroll-to-top on filter change. Slideshow controls hidden when only 1 item. Files modified: PrivateGalleryBundleLightbox.tsx, BundleLightbox.tsx, Gallery.tsx, Feed.tsx, api.ts, private_gallery.py, paid_content.py, db_adapter.py. | | 2026-02-07 | 12.9.0 | **Thumbnail Progress Modal & Async Gallery Operations** - New shared `ThumbnailProgressModal` component replaces text-based progress displays with visual thumbnail-grid progress tracking across Copy to Gallery, Upload, and Manual Import. Private Gallery copy and upload operations now run asynchronously via BackgroundTasks with in-memory job tracking and 500ms polling. New `GET /private-gallery/job-status/{job_id}` endpoint with dual auth. Added 3 Instagram filename patterns for browser extension formats (video/photo/shortcode). Fixed video thumbnail generation for short videos (fallback seek time). Added temp/manual_import to ALLOWED_PATHS. Fixed ThrottledImage IntersectionObserver in nested scroll containers. private_gallery.py grew from ~1,800 to 3,696 lines. | | 2026-02-06 | 12.7.5 | **Code Review: Bug Fixes & Code Quality** - Fixed critical bug in shared `get_or_create_thumbnail()` in core/utils.py where it queried non-existent `content_hash` column instead of correct `file_hash` primary key (caused cache miss on every request, regenerating thumbnails unnecessarily). Fixed `get_current_user_media()` token parameter missing `Query()` annotation (query param tokens from ``/`