LIComputerGuy/media-downloader
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
media-downloader/docs/COMPREHENSIVE_CODE_REVIEW.md
Todd 0d7b2b1aab Initial commit 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 22:42:55 -04:00

154 KiB
Raw Permalink Blame History

Comprehensive Code Review

Version: 13.13.1 Review Date: 2025-12-04 Last Updated: 2026-03-29 Status: Active Review Document

Table of Contents

  1. Executive Summary
  2. Codebase Statistics
  3. Architecture Overview
  4. Critical Issues
  5. Code Quality Analysis
  6. Performance Analysis
  7. Security Analysis
  8. Technical Debt Tracker
  9. Inconsistencies Found
  10. Recommended Next Big Features
  11. Version Update Checklist
  12. Changelog

Executive Summary

Media Downloader is a production-grade, enterprise-level media archival system with approximately 79,000 lines of code across Python backend and React/TypeScript frontend. The codebase demonstrates mature software engineering practices with recent significant improvements:

  1. Monolithic API file RESOLVED - API modularized into 17 routers (828 lines down from 10,465)
  2. Broad exception handling (622 instances of except Exception) - gradual improvement ongoing
  3. Synchronous HTTP in async context - blocking calls in FastAPI (partial fix with run_in_executor)
  4. Large module files exceeding 2,000+ lines each - still needs attention

Health Score: 8.3/10 (up from 8.2)

Category Score Notes
Functionality 9/10 Feature-rich, comprehensive platform support
Code Quality 8/10 API modularization complete, well-structured
Performance 8.5/10 Correlated subqueries replaced with JOINs, PostgreSQL fully operational
Security 8.5/10 12-point security audit + SQL injection defense-in-depth + proxy domain hardening
Maintainability 7/10 Significantly improved with modular routers
Test Coverage 4/10 Limited test suite
Documentation 9/10 Excellent inline and external docs

Codebase Statistics

Current Version: 13.13.1

Metric Count Change Since 12.0.0
Total Lines (Python) ~70,500 +21,800 (+ press.py, cloud_backup.py, cloud_backup_sync.py, backfill_press.py, besteyecandy_client.py, coppermine_client.py, reddit_client.py + all previous)
Total Lines (TypeScript/TSX) ~39,500 +14,000 (+ Gallery page, GalleryLightbox, Press page, PressArticleModal, cloud backup dashboard widget, shuffle support + all previous)
Python Modules 56 +26 (+ besteyecandy_client, coppermine_client, reddit_client + all previous)
API Routers 28 +4 (+ press, cloud_backup)
API Endpoints 280+ +110 (+ press CRUD/fetch + cloud backup config/sync/status + all previous)
Frontend Pages 39 +16 (+ Gallery page, Press page)
Frontend Components 16+ +5 (+ GalleryLightbox, PressArticleModal)
Frontend Contexts 2 BreadcrumbContext, PrivateGalleryContext
Frontend Hooks 3 useBreadcrumb, usePrivateGalleryAuth, useEnabledFeatures
Database Tables 89 +63 (PostgreSQL migration + private_media_reddit_communities + private_media_reddit_history)
Database Backend PostgreSQL 16 Migrated from SQLite in v13.0.0 via runtime adapter
Database Indexes 51+ +26 (paid content + private gallery + reddit community + message indexes)

File Size Analysis (Updated)

File Lines Severity Status
web/backend/api.py 828 ✅ RESOLVED Modularized into 17 routers
web/frontend/src/pages/Configuration.tsx 5,945 HIGH Component extraction needed
modules/forum_downloader.py 4,882 HIGH Modular redesign needed
modules/unified_database.py 4,479 MEDIUM -200 lines (unused adapters removed)
modules/imginn_module.py 3,485 MEDIUM Uses unified_db directly now
media-downloader.py 2,866 MEDIUM Command pattern
modules/fastdl_module.py 2,236 MEDIUM Uses centralized instagram_utils
modules/instagram_client_module.py 1,651 ✅ NEW Direct Instagram API client (v13.4.0)
modules/paid_content/instagram_client.py 1,081 ✅ NEW Instagram API for Paid Content (v13.4.0)
modules/instagram_utils.py 429 ✅ NEW Centralized Instagram functions

API Router Breakdown (NEW)

Router Lines Purpose
private_gallery.py 6,707 Private gallery auth, media, persons, encryption, features, async job tracking, URL import, Reddit monitor endpoints
paid_content.py 6,255 Paid content CRUD, services, creators, feed, notifications, messages, OnlyFans/Fansly setup, auto health checks, bulk service sync
media.py 1,334 Media serving, thumbnails, gallery, date editing
face.py 985 Face recognition endpoints
discovery.py 952 Source discovery, platforms
video.py 939 Video downloads, info
review.py 904 Review queue management
downloads.py 836 Download history, analytics
scheduler.py 630 Scheduler control
stats.py 521 Dashboard stats, monitoring
config.py 521 Configuration management
scrapers.py 485 Scraper controls, errors
Others ~2,000 Auth, health, recycle, etc.

Async/Sync Function Distribution

  • Async functions in API: 190
  • Sync functions in API: 27
  • Concern: Many sync functions call blocking requests library

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                         MEDIA DOWNLOADER                             │
├─────────────────────────────────────────────────────────────────────┤
│  FRONTEND (React 18 + TypeScript + Tailwind)                        │
│  ├── 35 Pages (Dashboard, Media, Downloads, Review, Config, etc.)   │
│  ├── 12 Reusable Components                                         │
│  ├── TanStack Query for data fetching                               │
│  └── WebSocket for real-time updates                                │
├─────────────────────────────────────────────────────────────────────┤
│  API LAYER (FastAPI + Uvicorn)                                      │
│  ├── 231+ REST Endpoints                                            │
│  ├── JWT + Session Authentication                                   │
│  ├── 2FA (TOTP, Duo, Passkeys)                                      │
│  ├── Rate Limiting (slowapi)                                        │
│  └── Redis Caching                                                  │
├─────────────────────────────────────────────────────────────────────┤
│  CORE MODULES (36 Python modules)                                   │
│  ├── Platform Downloaders (Instagram x4, TikTok, Snapchat, Forums)  │
│  ├── Face Recognition (InsightFace + DeepFace + dlib fallback)      │
│  ├── Semantic Search (CLIP embeddings)                              │
│  ├── Scheduler (randomized intervals)                               │
│  └── File Management (move, hash, dedupe)                           │
├─────────────────────────────────────────────────────────────────────┤
│  DATA LAYER                                                         │
│  ├── PostgreSQL 16 (via pg_adapter runtime translation, v13.0.0)   │
│  ├── Redis (caching, sessions)                                      │
│  └── File System (organized by platform/source)                     │
└─────────────────────────────────────────────────────────────────────┘

Critical Issues

1. ✅ RESOLVED: Monolithic API File

File: /opt/media-downloader/web/backend/api.py Lines: 828 (down from 10,465 - 92% reduction!) Status: RESOLVED as of v6.52.44

Solution Implemented:

web/backend/
├── api.py (app initialization, WebSocket only - 828 lines)
├── core/
│   ├── dependencies.py (shared state, auth)
│   ├── exceptions.py (custom exception classes)
│   └── config.py (configuration)
├── routers/
│   ├── auth.py (authentication, 2FA)
│   ├── downloads.py (download history, analytics)
│   ├── media.py (media serving, gallery)
│   ├── scheduler.py (scheduler control)
│   ├── discovery.py (source discovery)
│   ├── face.py (face recognition)
│   ├── recycle.py (recycle bin)
│   ├── review.py (review queue)
│   ├── video.py (video downloads)
│   ├── stats.py (dashboard, monitoring)
│   ├── config.py (configuration)
│   ├── scrapers.py (scraper controls, errors)
│   ├── health.py (health checks)
│   ├── notifications.py (push notifications)
│   ├── manual_import.py (manual imports)
│   ├── semantic_search.py (CLIP search)
│   └── cache.py (cache management)
└── (17 routers total)

Resolution Date: 2025-12-05 Effort Spent: ~8 hours

2. HIGH: Broad Exception Handling

Count: 622 instances of except Exception Impact: Debugging difficulty, silent failures, error masking

Examples Found:

# Anti-pattern (throughout codebase)
try:
    result = do_something()
except Exception as e:
    logger.error(f"Error: {e}")
    return None

Recommended Fix:

# Catch specific exceptions
try:
    result = do_something()
except requests.RequestException as e:
    logger.error(f"Network error: {e}")
    raise DownloadError(f"Failed to download: {e}") from e
except ValueError as e:
    logger.error(f"Invalid data: {e}")
    raise ValidationError(f"Invalid input: {e}") from e

Priority: MEDIUM (gradual refactoring)

3. HIGH: Synchronous HTTP in Async Context

Issue: FastAPI is async but many endpoints use synchronous requests library

Impact:

  • Blocks event loop
  • Reduces concurrent request handling
  • Performance bottleneck

Current Pattern:

@app.get("/api/something")
async def get_something():
    # BLOCKING CALL in async function!
    response = requests.get("https://external-api.com/data")
    return response.json()

Solution Options:

  1. Use httpx (recommended):
import httpx

@app.get("/api/something")
async def get_something():
    async with httpx.AsyncClient() as client:
        response = await client.get("https://external-api.com/data")
        return response.json()
  1. Use run_in_executor for sync code:
import asyncio
from concurrent.futures import ThreadPoolExecutor

executor = ThreadPoolExecutor(max_workers=10)

@app.get("/api/something")
async def get_something():
    loop = asyncio.get_event_loop()
    response = await loop.run_in_executor(
        executor,
        lambda: requests.get("https://external-api.com/data")
    )
    return response.json()

Effort: 24-40 hours Priority: HIGH (for performance-critical endpoints)

4. MEDIUM: Instagram Module Duplication

Files Affected:

  • modules/fastdl_module.py (2,236 lines)
  • modules/imginn_module.py (2,870 lines)
  • modules/toolzu_module.py (1,025 lines)

Duplication: 60-70% shared code including:

  • Cookie management
  • FlareSolverr integration
  • HTML parsing
  • Download logic
  • Error handling

Solution: Create base class:

# modules/instagram/base_instagram.py
class BaseInstagramDownloader(ABC):
    def __init__(self, config, unified_db):
        self.config = config
        self.unified_db = unified_db
        self.cookie_manager = CookieManager()
        self.flaresolverr = FlareSolverrClient()

    def _get_or_create_session(self): ...
    def _parse_stories(self, html): ...
    def _download_media(self, url, save_path): ...

    @abstractmethod
    def _get_content_urls(self, username) -> List[str]:
        pass

Effort: 12-16 hours Priority: MEDIUM

Code Quality Analysis

Positive Patterns Found

  1. Unified Database Layer - Single source of truth with adapters
  2. Universal Logger - Centralized logging with web visibility
  3. Database-First File Tracking - 50-100x faster than filesystem scanning
  4. Face Recognition Fallback Chain - InsightFace → DeepFace → dlib
  5. Soft Delete with Recycle Bin - File recovery capability
  6. Comprehensive Documentation - 560KB+ in docs folder
  7. Real-Time Updates - WebSocket for logs and errors

Areas Needing Improvement

Area Current State Target State
Exception handling 622 broad catches Specific exception types
Type hints Partial coverage Full type annotations
Test coverage ~10% estimated 70%+ target
API documentation No OpenAPI docs Full Swagger/ReDoc
Database migrations Ad-hoc ALTER TABLE Alembic migrations
Code formatting Inconsistent Black + isort

Code Smells Detected

  1. God Objects: ForumDownloader class does too much
  2. Long Methods: Several methods exceed 100 lines
  3. Deep Nesting: 5+ levels in some functions
  4. Magic Numbers: Hardcoded timeouts, limits, paths
  5. Missing Type Hints: Many functions lack annotations

Performance Analysis

Current Optimizations (Good)

  1. Database Indexes: 22+ optimized indexes
  2. Connection Pooling: PostgreSQL ThreadedConnectionPool (2-30 connections, v13.0.0)
  3. Thumbnail Caching: SHA256-keyed cache with on-demand generation
  4. Query Performance: Sub-10ms for indexed queries
  5. Batch Face Lookup: N+1 query pattern fixed in 6.52.32
  6. Lazy Loading: On-demand metadata fetching in lightbox

Performance Bottlenecks Identified

Issue Location Impact Fix
Sync HTTP in async api.py High Use httpx/aiohttp
No query result caching Multiple endpoints Medium Add Redis caching
Full table scans Some search queries Medium Add missing indexes
Large response payloads /api/downloads Low Pagination + field selection

Recommended Performance Improvements

  1. Add Redis caching for expensive queries:
@app.get("/api/stats")
async def get_stats():
    cached = await redis.get("stats:global")
    if cached:
        return json.loads(cached)

    stats = compute_expensive_stats()
    await redis.setex("stats:global", 300, json.dumps(stats))
    return stats
  1. Implement field selection for API responses:
@app.get("/api/downloads")
async def get_downloads(fields: str = None):
    if fields:
        # Return only requested fields
        selected = fields.split(",")
        return [{k: v for k, v in d.items() if k in selected}
                for d in downloads]

Security Analysis

Current Security Measures (Strong)

Feature Implementation Status
Authentication JWT + Session cookies ✅
Password Hashing bcrypt ✅
2FA TOTP + Duo + Passkeys ✅
CSRF Protection starlette-csrf ✅
Rate Limiting slowapi ✅
SQL Injection safe_query_builder module ✅
Path Traversal Validation on file paths ✅

Security Concerns

  1. Hardcoded Paths:
# Found in multiple files
base_path = Path("/opt/immich/md")  # Should be configurable
  1. Potential Secret Exposure:
# .env file handling - ensure proper permissions
# Check: ls -la .env should show 600 permissions
  1. Missing Security Headers:
# Recommend adding in middleware
response.headers["X-Content-Type-Options"] = "nosniff"
response.headers["X-Frame-Options"] = "DENY"
response.headers["X-XSS-Protection"] = "1; mode=block"

Technical Debt Tracker

Outstanding Technical Debt

ID Description Priority Effort Status
TD-001 Monolithic api.py (10,465 lines) HIGH 24h In Progress - Router framework created
TD-002 Broad exception handling (622 cases) MEDIUM 16h Resolved - Custom exceptions created
TD-003 Instagram module duplication MEDIUM 16h Resolved - Base class created
TD-004 Missing database migrations MEDIUM 8h Open
TD-005 Sync HTTP in async context HIGH 32h Open
TD-006 Missing API documentation LOW 6h Open
TD-007 Low test coverage (~10%) MEDIUM 32h Open
TD-008 Hardcoded configuration values LOW 4h Resolved - Unified config manager
TD-009 Missing type hints LOW 16h Open
TD-010 Large frontend components MEDIUM 12h Open

Recently Resolved

ID Description Resolved In Notes
TD-002 Broad exception handling 6.52.38 Custom exception classes in core/exceptions.py
TD-003 Instagram module duplication 6.52.38 Base class in modules/instagram/base.py
TD-008 Hardcoded configuration values 6.52.38 Unified config in core/config.py
TD-015 Inconsistent date formats 6.52.38 ISO 8601 utilities in core/responses.py
TD-016 Inconsistent error responses 6.52.38 Standardized in core/responses.py
TD-011 N+1 query in downloads 6.52.32 Batch face lookup added
TD-012 Thread safety in forum downloads 6.52.32 Context priority fix
TD-013 Asyncio.run() in threads 6.52.32 run_coroutine_threadsafe
TD-014 Missing dimensions on download 6.52.29 PIL/ffprobe extraction

Inconsistencies Found

1. ✅ RESOLVED: Logging Inconsistency

Issue: Mix of logging approaches (duplicate log() methods across modules)

Solution Implemented (v11.19.2):

  • Created modules/base_module.py with LoggingMixin class
  • All download modules now inherit from LoggingMixin
  • Provides consistent log() method with backwards-compatible log_callback support
  • Removed 12+ duplicate log() method implementations
  • Updated 13 modules: instaloader, tiktok, imginn, fastdl, toolzu, coppermine, forum_downloader, download_manager, move_module, scheduler, snapchat, instagram_repost_detector

Resolution Date: 2025-12-26

2. Error Response Format Inconsistency

Issue: Different error response formats across endpoints

# Format 1
raise HTTPException(status_code=400, detail="Invalid request")

# Format 2
return {"error": True, "message": "Invalid request"}

# Format 3
return JSONResponse(status_code=400, content={"error": "Invalid request"})

Solution: Standardize on HTTPException with consistent detail format

3. Date Format Inconsistency

Issue: Multiple date formats in database and API

# Format 1: ISO
"2025-12-04T10:30:00Z"

# Format 2: Unix timestamp
1733308200

# Format 3: Custom
"2025-12-04 10:30:00"

Solution: Standardize on ISO 8601 with timezone

4. Configuration Loading Inconsistency

Issue: Config loaded from multiple sources

# Source 1: Environment variables
os.environ.get("MEDIA_PATH")

# Source 2: .env file
load_dotenv()

# Source 3: Database settings
settings_manager.get("media_path")

# Source 4: Hardcoded defaults
DEFAULT_PATH = "/opt/immich/md"

Solution: Single configuration manager with hierarchy:

  1. Environment variables (highest priority)
  2. .env file
  3. Database settings
  4. Hardcoded defaults (lowest priority)

Recommended Next Big Features

Based on codebase analysis and user impact potential:

Tier 1: High Impact, High Value

1. Webhook Integration System ⭐ RECOMMENDED

Effort: 6-8 hours | Value: HIGH

Why this feature:

  • Zero external dependencies
  • Enables Discord, Slack, Home Assistant integration
  • Automation workflows without code changes
  • Natural extension of existing notification system
# Implementation sketch
class WebhookManager:
    events = ['download_complete', 'download_error', 'face_matched', 'new_content']

    async def fire_webhook(self, event: str, data: dict):
        webhooks = await self.get_webhooks_for_event(event)
        for webhook in webhooks:
            await self.send_with_retry(webhook, event, data)

2. Duplicate Management Dashboard ⭐ RECOMMENDED

Effort: 10-12 hours | Value: HIGH

Why this feature:

  • File hash infrastructure already exists
  • Users manually hunting duplicates today
  • Could reclaim significant storage space
  • Visual side-by-side comparison
  • "Keep best quality" automation
┌─────────────────────────────────────────────────────────────┐
│ Duplicates Dashboard                            230 GB saved │
├─────────────────────────────────────────────────────────────┤
│ ┌─────────────────────┬─────────────────────┐               │
│ │ Original            │ Duplicate           │               │
│ │ [Image Preview]     │ [Image Preview]     │               │
│ │ 1920x1080 / 2.5 MB  │ 1280x720 / 1.8 MB   │               │
│ │ Instagram/user1     │ FastDL/user1        │               │
│ │ [✓ Keep]            │ [Delete]            │               │
│ └─────────────────────┴─────────────────────┘               │
└─────────────────────────────────────────────────────────────┘

3. Mobile-First PWA

Effort: 4-6 hours | Value: HIGH

Why this feature:

  • Installable app experience
  • Offline mode capability
  • Push notifications
  • Already using React, easy to add
  • No app store approval needed

Tier 2: Medium Impact, Medium Value

Feature Effort Value Notes
Background Job Queue (Celery) 12-16h HIGH Heavy ops in background
API Modularization 16-24h MEDIUM Enables better testing
Prometheus Metrics 6-8h MEDIUM Grafana dashboards
S3/Object Storage 6-8h MEDIUM Cloud storage option
Plugin System 16-24h HIGH User extensibility

Tier 3: Future Considerations

Feature Effort Notes
RBAC (Role-Based Access) 12-16h Multi-user environments
AI Auto-Tagging 16-24h ML-based categorization
Video Transcoding 12-16h Format conversion
Full-Text Search 8-12h Elasticsearch/Meilisearch

Version Update Checklist

Use this checklist when updating to a new version:

Pre-Update

  • Review CHANGELOG.md for breaking changes
  • Backup database: cp database/media_downloader.db database/backup_$(date +%Y%m%d).db
  • Note current version: cat VERSION
  • Check disk space: df -h /opt/media-downloader

During Update

  • Stop services: systemctl stop media-downloader
  • Pull new code or extract release
  • Run database migrations if any
  • Update dependencies: pip install -r requirements.txt
  • Rebuild frontend: cd web/frontend && npm run build

Post-Update

  • Start services: systemctl start media-downloader
  • Verify API health: curl http://localhost:8000/api/health
  • Check logs for errors: tail -f logs/$(date +%Y-%m-%d).log
  • Update this document's version number
  • Test critical functionality

Update This Review Document

  • Update version number at top
  • Update codebase statistics
  • Add any new issues found
  • Mark resolved issues
  • Update recommendations if needed

Changelog

Review Document Updates

Date Version Reviewed Changes
2026-03-29 13.13.1 Pinned Post Highlighting, Mobile Header Layout, Feed Navigation Fixes — Patch release. PostCard component in Feed.tsx adds conditional amber background for pinned posts: bg-amber-50/60 dark:bg-amber-900/15 (unselected) and bg-amber-100/80 dark:bg-amber-900/30 (selected), replacing the default blue bg-primary/10 selection for pinned items. New highlighted boolean prop added to PostDetailView.tsx interface — swaps root div bg-card for bg-amber-50/60 dark:bg-amber-900/15 and sticky header bg-card for bg-amber-50/80 dark:bg-amber-900/25. Passed as highlighted={true} for mobile social view pinned posts, highlighted={selectedPost.is_pinned === 1} for desktop social detail panel and mobile post detail modal. Mobile post header layout refactored: subtitle row split into three divs — (1) @username + platform badge + pinned indicator (visible always), (2) date-only row (sm:hidden), (3) desktop continuation with · date · Pinned (hidden sm:flex). Feed navigation fixes: new useEffect watches pinnedCollapsed — when true and selectedPost.is_pinned === 1, auto-selects regularPosts[0]. Keyboard navigation handleKeyDown now uses pinnedCollapsed ? regularPosts : posts for the navigation array, preventing arrow keys from selecting hidden pinned posts. Added regularPosts and pinnedCollapsed to useEffect dependency array. Files modified: Feed.tsx, PostDetailView.tsx.
2026-03-29 13.13.0 Instagram Authenticated API Toggle, Cookie Management UI, Creator Card Badges — Minor release. New per-creator use_authenticated_api toggle (INTEGER DEFAULT 0 column on paid_content_creators): when enabled, Instagram sync in imginn_adapter.py _fetch_all_posts() uses browser cookies as primary fetch method via _fetch_via_browser_cookies(), with automatic fallback to unauthenticated GraphQL if cookies fail. Return type changed from List[Dict] to Optional[List[Dict]] — None = API failed (fall back), [] = API worked but nothing new (no fallback). _notify_cookie_expired() method on 401: updates scrapers.last_test_status='failed' and broadcasts cookie_health_alert via _broadcast_cookie_alert() from health.py. scraper.py passes use_authenticated_api=bool(creator.get('use_authenticated_api')) to get_posts(). Backend: use_authenticated_api added to UpdateCreatorRequest model and boolean conversion loop in paid_content.py, added to ALLOWED_CREATOR_COLUMNS in db_adapter.py, added to migration list in unified_database.py. Frontend: use_authenticated_api: number added to PaidContentCreator interface and updatePaidContentCreator() params in api.ts. CreatorSettingsModal in Creators.tsx adds toggle (Instagram-only, creator.platform === 'instagram'). Creator card badges: green Shield + "Authenticated" badge when use_authenticated_api is truthy, purple Filter + "Filtered" badge (with hover tooltip showing @usernames) when filter_tagged_users JSON array is non-empty. Scrapers page: instagram_browser scraper renamed to "Instagram (Authenticated)" in DB. Scrapers.tsx adds api to typeOrder array, typeLabels ("Authenticated APIs"), and TypeBadge styles/labels (emerald). mds service manager script: added backup → cloud-backup-sync and backupui → backup-central to SERVICE_MAP and ALL_ALIASES. Files modified: imginn_adapter.py, scraper.py, paid_content.py, db_adapter.py, unified_database.py, api.ts, Creators.tsx, Scrapers.tsx, scripts/mds.
2026-03-28 13.12.0 Sync Service Button on Creators Page — Minor release. New POST /api/paid-content/creators/sync-service/{service_id} endpoint in paid_content.py fetches all enabled creators for a given service_id from database, queues sequential background sync via new _sync_service_background() helper (loops through creator IDs calling scraper.sync_creator() with per-creator error handling), returns {status, count, creators}. Rate limited to 5/minute. Frontend: new syncPaidContentService(serviceId) API method in api.ts with syncService namespace entry. ServiceGroupedCreators component in Creators.tsx gains onSyncService and syncingServices props. Multi-service collapsible headers now include a "Sync All" button (RefreshCw icon, text hidden on mobile) with e.stopPropagation() to prevent toggling the collapsible. Single-service view adds a header bar with service name, count, and sync button. Button disabled and shows spinning icon when syncingServices set contains the service ID or any creator in the service is in syncingIds. Parent component adds syncServiceMutation (useMutation) with success notification showing queued count and service name. Files modified: paid_content.py, api.ts, Creators.tsx.
2026-03-28 13.11.2 Instagram User ID Fix, Unviewed Count Fix, Bundle Sidebar Default — Patch release. Instagram _fetch_user_id() rewritten in instagram_client_module.py: new _fetch_user_id_from_html() method is now primary — scrapes "user_id":"NNNNN" from profile page HTML using unauthenticated GET request, bypassing 401 errors caused by 77 stale browser cookies with expired sessionid that made _has_valid_session() return True. web_profile_info API kept as fallback. User ID cache persistence fix: self.unified_db was being set to None when use_database=False, preventing _save_user_id_cache() from writing to the PostgreSQL settings table. Fixed by keeping unified_db reference regardless of use_database flag. get_unviewed_posts_count() in db_adapter.py now joins paid_content_creators and applies the same filter_tagged_users clause used by get_posts() and get_posts_count() — previously counted all unviewed posts with attachments regardless of per-creator tag filters, causing banner to show 12 when only 5 were visible (7 hidden posts from dimitrishair filter ["nicolescherzinger"] and makkaroo filter ["kyliejenner"] had no matching tagged users). Bundle sidebar in BundleLightbox.tsx and PrivateGalleryBundleLightbox.tsx now defaults to collapsed (useState(false)) on all devices instead of auto-opening on desktop. PrivateGalleryBundleLightbox.tsx auto-hide effect changed from setSidebarOpen(!isCurrentVideo && !isMobile) to only closing for videos, preventing sidebar from re-opening when navigating to images. Files modified: instagram_client_module.py, db_adapter.py, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx.
2026-03-27 13.11.1 Instagram GraphQL Post Fetching, Scan Progress, Feed Page Size — Patch release. Instagram post sync switched from ImgInn API to direct Instagram GraphQL API (OwnerToTimelineMediaLoggedOutQuery) in imginn_adapter.py _fetch_all_posts(). ImgInn had a server-side 480-post limit (40 pages × 12 items) regardless of client (curl_cffi or FlareSolverr). New flow: _fetch_all_posts() creates an InstagramClientDownloader instance, calls fetch_posts() which paginates GraphQL at 50 posts/page with 3-5s delay. Raw v1 nodes converted to ImgInn-style items via _ig_node_to_item() with _from_ig_api=True flag. Phase 2 fast-path: items with _from_ig_api=True skip ImgInn detail page + tag fetching entirely (already have full-res URLs + usertags from GraphQL). _best_media_url() static method extracts highest quality URL from v1 media nodes. Real-time scan progress: progress_callback(count) called after each GraphQL page in fetch_posts() (line 617), passed from _fetch_all_posts() via _scan_progress wrapper. UI shows "Fetched N posts..." during scanning phase instead of static "Scanning profile". GraphQL delay increased from random.uniform(2.0, 4.0) to random.uniform(3.0, 5.0) to avoid HTTP 572 rate limiting on large accounts (tested: 5000+ posts with 3s delay succeeds, <3s hits 572 at ~1100 posts). max_posts increased from 5,000 to 15,000 in instagram_adapter.py. Update syncs remain efficient: since_date → small days_back → tight date cutoff, consecutive_old >= 5 stops pagination after 1-2 pages for recent syncs. Feed.tsx limit increased from 30 to 50 for faster mobile initial load. Note: accounts blocking logged-out GraphQL ("Invalid owner id") fall back to authenticated API which requires valid session cookies. Files modified: imginn_adapter.py, instagram_adapter.py, instagram_client_module.py, Feed.tsx.
2026-03-26 13.11.0 Media Gallery Page, GalleryLightbox, Server-Side Shuffle, Timeline Jump Fix — Minor release. New /gallery page replacing /media with Immich-style justified thumbnail layout (buildJustifiedRows() targeting ~180px row height), daily date grouping with smart labels (Today, Yesterday, weekday names, abbreviated dates), timeline scrubber navigation, slideshow mode, and infinite scroll via useInfiniteQuery. New GalleryLightbox.tsx (~1000 lines) — simplified BundleLightbox without paid-content features, with slideshow auto-advance, embedded metadata display (ffprobe/exiftool via /api/media/embedded-metadata — title, description, artist, source URL), zoom/pan, swipe navigation, keyboard controls. Server-side shuffle for media gallery: new shuffle and shuffle_seed query params on /api/media/gallery, PostgreSQL ORDER BY md5(fi.id::text || seed::text) for deterministic randomization, separate useInfiniteQuery for shuffled results, shuffle button (cyan when active) in lightbox slideshow toolbar. Timeline scrubber date jump fix for both galleries: now sets both date_from AND date_to (last day of target month) to bound the query, preventing DESC sort from returning newest items when jumping to an old month. TimelineScrubber.tsx fixed all === activeKey comparisons to activeKey.startsWith(key) for daily section key compatibility (keys are YYYY-MM-DD, scrubber uses YYYY-MM). Bug fixes: added autoPlayVideo={true} to paid content gallery BundleLightbox props; feed unviewed count query adds EXISTS (SELECT 1 FROM paid_content_attachments) to exclude empty posts; media gallery date-range endpoint changed fi.location = 'media' to 'final'; removed non-existent fi.duration from gallery queries. Route changes: /media redirects to /gallery, nav/breadcrumbs/feature toggles updated. Files created: Gallery.tsx, GalleryLightbox.tsx. Files modified: media.py, GalleryTimeline.tsx, TimelineScrubber.tsx, App.tsx, api.ts, breadcrumbConfig.ts, Configuration.tsx, Features.tsx, private_gallery.py, db_adapter.py.
2026-03-26 13.10.0 Justified Gallery Layout, Timeline Navigation, Service Health Check Fixes — Minor release. Paid content gallery now uses justified thumbnail rows (Immich-style) where images display at their natural aspect ratios instead of uniform squares. buildJustifiedRows() function packs items into rows targeting ~180px height, scaling widths to fill container exactly. New JustifiedSection React.memo component isolates per-section rendering. Timeline scrubber onJumpToDate callback: clicking a month not yet loaded via infinite scroll triggers date_from API query parameter, resetting infinite scroll to load from that month. Dismissible "Showing from {Month Year}" chip with X button clears filter. Timeline layout changed from centered top-1/2 -translate-y-1/2 max-h-[70vh] to full-height top-16 bottom-4. Three performance optimizations: (1) callback ref (useCallback ref) replaces useRef + useLayoutEffect for container width measurement — fires synchronously when conditionally-rendered div mounts, eliminating blank first frame; (2) sectionRowsMap useMemo pre-computes all justified rows keyed on [sections, containerWidth] instead of computing per-section inside render loop; (3) ResizeObserver deadband ignores sub-pixel width changes (threshold >1px). Backend: added dedicated health checks for coppermine (HTTP client availability), besteyecandy (site reachability with Cloudflare 403 acceptance), and reddit (gallery-dl binary check) in _check_single_service_health() — these were falling through to generic PaidContentAPIClient.check_health() which tried {base_url}/api/v1/creators (Coomer/Kemono pattern), producing false down/degraded status. Files modified: TimelineScrubber.tsx, GalleryTimeline.tsx, paid_content.py.
2026-03-24 13.9.0 Press Monitoring, Cloud Backup, BestEyeCandy/Coppermine/Reddit Scrapers, Feed Shuffle — Minor release. New press monitoring system: celebrity news tracking via Google News RSS + GDELT with full article extraction (readability + BeautifulSoup), local image caching with proxy serving, Pushover notifications, read/unread tracking, filtering by celebrity/domain/read status, full-text search. Dedicated Press page (Press.tsx, 422 lines) with stats dashboard, article cards, and PressArticleModal (292 lines) with HTML sanitization. New cloud backup system: rclone-based backup to B2/S3 with AES256 encryption, cloud_backup.py router (1526 lines), cloud_backup_sync.py daemon (1194 lines) with inotify file watching, per-directory cooldown sync, daily full backups at 3AM with Immich + Media Downloader PostgreSQL dumps, live progress tracking. Dashboard widget shows real-time sync progress with animated gradient border, transfer speed/ETA, phase detection. Three new paid content scrapers: BestEyeCandy client (469 lines) with cookie auth and URL pattern discovery for full-res images; Coppermine client for nested PHP gallery structure (categories > subcategories > albums > photos) with thumbnail-to-fullres URL derivation; Reddit client via gallery-dl integration with subreddit metadata API. Backend-driven shuffle for paid content slideshow: deterministic seed-based randomization in db_adapter.py (random.Random(seed) shuffles all matching IDs, then paginates), frontend mulberry32 seeded PRNG for cross-post attachment mixing. BundleLightbox.tsx gains onShuffleChange/isShuffled props for parent-managed shuffle. Feed page size increased 10→30 fixing mobile feed empty state (20 pinned posts filled first 2 pages at old limit with collapsed pinned section). Feed error handling improved with isPending tracking, retry count 3, error UI in all 4 view modes. Feed setSearchParams skipped on initial mount to prevent startTransition interference. Paid content feed endpoint limit max increased 100→500. Log viewer adds cloudbackup, cloudflare.fastdl/imginn/imginn-stories, taskcheckpoint components. Creators page adds hqcelebcorner/picturepub to SERVICE_URLS. New scripts: backfill_press.py (592 lines) for historical article bulk loading with sliding windows, cloud_backup_sync.py daemon. Configuration.tsx adds cloud_backup and press tabs. App.tsx adds Press route. Breadcrumb config adds /press. Cleaned pycache directories. Files created: press.py, cloud_backup.py, besteyecandy_client.py, coppermine_client.py, reddit_client.py, backfill_press.py, cloud_backup_sync.py, Press.tsx, PressArticleModal.tsx. Files modified: db_adapter.py, scraper.py, utils.py, paid_content.py, init.py, Feed.tsx, BundleLightbox.tsx, api.ts, Dashboard.tsx, Configuration.tsx, App.tsx, Logs.tsx, breadcrumbConfig.ts, Creators.tsx.
2026-03-19 13.8.1 Audio Player, Push Notification Accuracy, Reddit Reliability, Health Check Fixes — Patch release. Inline audio player in PostDetailView (native HTML5 audio with filename, file size, playback controls) and mobile Feed PostCard (compact player with stopPropagation preventing post navigation on tap). Audio attachments excluded from thumbnail grid and BundleLightbox. Push notification audio fix: content_type from scraper metadata checked before file extension, fixing .mp4 audio files misidentified as video; added 🎵 audio category to titles and pushover PLURALS/CONTENT_ICONS; fixed 3 hardcoded content_type: 'video' in downloaded_file_info. Health check fixes: HQCelebCorner/PicturePub accept HTTP 200/403, Snapchat checks story.snapchat.com reachability. Reddit community monitor: switched from OAuth to REST API mode (extractor.reddit.api=rest) with --range 1-200 limit; fixed cookie file directory creation. Code quality: fixed function name shadowing in celebrity.py and video_queue.py; fixed backup profile stale paths. Files modified: scraper.py, pushover_notifier.py, reddit_community_monitor.py, paid_content.py, celebrity.py, video_queue.py, PostDetailView.tsx, Feed.tsx, install.sh, add-backup-profile.sh.
2026-03-18 13.8.0 XenForo Forum Support, Snapchat/TikTok HTTP Clients, Cookie Health Monitoring, Story Dedup — Minor release. New XenForo forum client (xenforo_forum_client.py, hqcelebcorner_client.py) for celebrity image forums with external host resolution, FlareSolverr bypass, CSRF handling. Snapchat HTTP client (snapchat_client.py) with curl_cffi replacing Playwright, NEXT_DATA JSON extraction, EXIF metadata. TikTok hybrid client (tiktok_client.py) with yt-dlp listing + gallery-dl downloading, lazy cookie loading. Instagram direct API module (fastdl_instagram_client.py) with GraphQL + REST via curl_cffi. Cookie health monitoring with per-platform toggles, 30-min cooldown, WebSocket + Pushover alerts. ImgInn adapter: two-phase discovery, FlareSolverr fallback, 1440p+ CDN URLs, NordVPN proxy rotation. Scheduler checkpoint system for crash recovery. Bellazon forum client, Soundgasm audio client, paid content URL parser expansion. Bug fixes: FastDL pk mapping, hash-based story dedup, recycle bin metadata, square video display, failed downloads creator_db_id. Files modified: 20+ files across modules/paid_content/, web/backend/routers/, web/frontend/src/.
2026-03-12 13.7.0 Instagram Sync Settings, Appearance Role Combining, Tagged User Filter — Minor release. Per-creator Instagram sync settings: new sync_posts, sync_stories, sync_highlights columns on paid_content_creators table with toggle UI in creator card menu (Instagram-only). Conditional sync in scraper.py wraps stories/highlights/posts fetching in creator.get('sync_X', 1) checks. Tagged user filter: new filter_tagged_users column (JSON array of usernames), new GET /creators/{id}/tagged-users endpoint returning distinct tagged usernames with counts, LIKE-based SQL filter in get_posts(), get_posts_count(), and get_media_count(). CreatorSettingsModal component in Creators.tsx with toggles and multi-select tagged user list. All paid content Instagram operations now route exclusively through ImgInnAPIAdapter — removed all UnifiedInstagramAdapter usage from scraper.py (was routing through real Instagram API via instagram_client). curl_cffi impersonation pinned from auto-resolving chrome (→chrome142, unsupported) to explicit chrome136. Appearances: upcoming endpoint now uses ShowStats CTE with GROUP_CONCAT(DISTINCT credit_type) and ROW_NUMBER() grouping matching aired endpoint. New _enrich_with_roles() helper fetches structured role data (credit_type, character_name, job_title) for multi-role appearances. Show episodes endpoint groups by season+episode+date, combining credit types. COUNT(*) → COUNT(DISTINCT episode_key) across all 4 endpoints. AppearanceDetailModal updated for multi-role display with consistent creditTypeConfig badge colors. Dashboard UpcomingAppearancesCard shows credit type badges. Notification INSERTs now set is_sent=1, sent_at=CURRENT_TIMESTAMP. FilterPopover max-height increased 200→320px. Removed unused instagram_unified_adapter.py (155 lines). Cleaned pycache directories. Files modified: scraper.py, imginn_api_module.py, db_adapter.py, appearances.py, paid_content.py, unified_database.py, api.ts, Creators.tsx, AppearanceDetailModal.tsx, UpcomingAppearancesCard.tsx, FilterPopover.tsx.
2026-03-07 13.6.0 ImgInn Paid Content Adapter, Reddit Timezone Fix, Lightbox Improvements, Cleanup — Minor release. New imginn_adapter.py replaces Instagram API-based adapter with ImgInn-based scraping for Paid Content — no Instagram credentials needed, maps ImgInn API data to existing Post/Attachment models, handles posts/stories/reels with CDN URLs, built-in rate limiting and Cloudflare bypass. ImgInn API optimizations: FlareSolverr integration for Cloudflare bypass, OCR disabling for faster scraping, date pre-filtering, profile resolution fixes. Reddit private gallery timezone fix: gallery-dl UTC dates now properly converted to local time using datetime.timezone; created_at uses import time instead of post date. Bulk-fixed 556 existing reddit posts and 1366 media entries with incorrect UTC dates via decrypt→convert→re-encrypt cycle. Lightbox fixes: slideshow back arrow not showing (parent-managed shuffle mode), paid content bundle sidebar z-index fix (z-20 over z-10 overlay), bundle sidebar auto-collapse on slideshow, hidden for single-image posts, expand/collapse floating button, stationary slideshow controls. Dashboard platform labels improved. Cleanup: removed 36 stale scripts/utilities (one-off backfills, Fansly maintenance, test scripts, completed migration scripts), legacy SQLite database file, archive directory, pycache directories.
2026-03-05 13.5.1 Landscape Lightbox, Min Resolution Filter, TypeScript Fixes, Cleanup — Patch release. Ported landscape phone detection and layout from BundleLightbox.tsx to PrivateGalleryBundleLightbox.tsx: touch-based device detection with isLandscape state (touch + height<500px), fixed positioning for landscape containers (100dvh), adjusted control positions/visibility, landscape-specific navigation arrows (compact rounded with bg-black/40), sidebar-aware image container (left: 6rem), landscape video sizing. Added per-person-group min_resolution setting: SQL filter (file_type != 'image' OR (width >= ? AND height >= ?)) shows all videos but only images meeting threshold; gallery filter dropdown (300/500/720/1080px+ options) with active filter chip; import skip for low-resolution images with skipped count tracking. Fixed TypeScript errors: added min_resolution?: number to getPersonGroups() and getPersonGroup() return types in api.ts, removed unused editMinRes state from Config.tsx, fixed task.next_run null safety in Scheduler.tsx. Cleaned up stale files: removed migration_backup.json (682KB), media_downloader.db.old (308KB), all pycache directories (~5MB). Files modified: PrivateGalleryBundleLightbox.tsx, private_gallery.py, Gallery.tsx, Config.tsx, api.ts, Scheduler.tsx.
2026-03-04 13.5.0 ImgInn API Module, Instagram Auth Circuit Breaker, Private Gallery Bridge, FastDL Consolidation — Minor release. New imginn_api_module.py (~600 lines) scraper using ImgInn's internal API for posts, stories, reels, and tagged content with subprocess wrapper. New instagram_rate_limiter.py (~160 lines) providing shared 180 calls/hour rolling window rate limiter with progressive slowdown, pause file persistence, and cross-module mutex. New scraper_gallery_bridge.py (~650 lines) for importing scraper downloads into private gallery with id-based incremental filtering via last_imported_file_id column on private_media_scraper_accounts. New paid_content/fastdl_instagram_client.py replacing unstable curl_cffi direct API with FastDL browser-based approach for Paid Content Instagram. New fastdl_subprocess_wrapper.py and imginn_api_subprocess_wrapper.py for subprocess isolation. Instagram auth failure circuit breaker: instagram_client_module.py sets auth_failed flag on 401/403, propagated through subprocess JSON results via base_subprocess_wrapper.py; media-downloader.py download_instagram_client() detects flag, skips stories/tagged for remaining users, continues posts/reels, updates scraper test status to 'failed' in database, sends Pushover notification. Cookie health banner updated: Instagram scrapers now shown in /health/cookies when test status is 'failed' (previously fully excluded from display); auto-clears when cookies re-saved. Dashboard "New In" cards fixed: all queries changed from ORDER BY COALESCE(d.download_date, fi.created_date) DESC to ORDER BY fi.id DESC; dismiss tracking changed from timestamp-based to ID-based (`number
2026-02-26 13.4.1 Private Gallery Unread Fixes, Paid Content Unviewed/Messages Banners — Patch release. Fixed three Private Gallery unread bugs: (1) /media endpoint count/shuffle/fast-path queries missing LEFT JOIN private_media_posts causing SQL errors when unread_only=true, (2) markSeenMutation.onSuccess used invalidateQueries instead of resetQueries leaving stale useInfiniteQuery pagination, (3) upload modal didn't pass post_id through onSuccess callback so new post wasn't auto-selected. Added Paid Content unviewed posts system: get_unviewed_posts_count() and mark_all_posts_viewed() db_adapter methods, /unviewed-count and /mark-all-viewed endpoints, Feed.tsx banner with "View unviewed"/"Mark all viewed" buttons using blue gradient+border style. Added Paid Content unread messages system: get_total_unread_messages_count() and mark_all_messages_read() db_adapter methods, /messages/unread-count and /messages/mark-all-read endpoints (placed before /{creator_id} path param route), violet gradient banners on Dashboard.tsx and Feed.tsx with "View messages" RouterLink and "Mark all read" button. Private Gallery unread banners restyled from solid to gradient+border matching dashboard pattern. CLAUDE.md updated with PostgreSQL syntax note for direct psql commands. Files modified: private_gallery.py, Gallery.tsx, PrivateGalleryUploadModal.tsx, paid_content.py, db_adapter.py, api.ts, Feed.tsx, Dashboard.tsx, CLAUDE.md.
2026-02-23 13.4.0 Instagram Client API, Module Enable/Disable, Image Info Bar, Snapchat Client - Minor release. New instagram_client_module.py (1,651 lines) provides direct Instagram GraphQL/REST API downloads using curl_cffi with browser TLS fingerprinting — 10-20x faster than ImgInn scraping. Posts via public GraphQL (OwnerToTimelineMediaLoggedOutQuery_connection), stories/reels/tagged via authenticated REST API with session cookies. File naming matches ImgInn format ({profile}_{YYYYMMDD_HHMMSS}_{media_id}{ext}) for dedup compatibility. Separate paid_content/instagram_client.py (1,081 lines) for Paid Content system. New snapchat_client_module.py with same direct API architecture. Both integrated into main downloader (media-downloader.py), scheduler, and platform router with subprocess wrappers. New module enable/disable system: platform-level enabled flags in config, scheduler filters by config.get('<platform>', {}).get('enabled'), health router _is_scraper_module_enabled() checks module status, frontend Scheduler.tsx filters enabledTasks. Image info bar added to EnhancedLightbox.tsx — images now show source, file size, date, resolution (emerald highlight), and platform below the image (matching video info bar, minus Duration). New CookieHealthBanner.tsx component for cookie expiration warnings via WebSocket. Major cleanup: removed 7.5 GB backup_sqlite_migration/ directory, empty .db placeholders, 365 debug screenshots (19 MB), stale test logs, pycache directories. Files modified: instagram_client_module.py (NEW), snapchat_client_module.py (NEW), paid_content/instagram_client.py (NEW), instagram_client_subprocess_wrapper.py (NEW), snapchat_client_subprocess_wrapper.py (NEW), CookieHealthBanner.tsx (NEW), media-downloader.py, scheduler.py, platforms.py, scrapers.py, health.py, EnhancedLightbox.tsx, Dashboard.tsx, Scheduler.tsx, Logs.tsx, Configuration.tsx, api.ts, Settings.tsx, Feed.tsx, Creators.tsx, Gallery.tsx, move_module.py.
2026-02-21 13.3.0 Reddit Community Monitor, Private Gallery Fixes, Log & Scheduler Standardization - Minor release. New reddit_community_monitor.py module (~1,050 lines) enables automated Reddit community monitoring for Private Gallery. Users map subreddits to persons via Config tab; system uses gallery-dl to download new posts (including imgur/redgifs attachments), encrypts files with AES-256-GCM, and imports into gallery with 'reddit' tag. Two-layer duplicate detection: post-level tracking via private_media_reddit_history table (survives post deletion) and file-level SHA-256 hash checking. Crypto key file approach (/opt/immich/private/.reddit_monitor_key) enables scheduler process to encrypt independently. Real-time progress tracking with phase updates (downloading/processing/importing/encrypting). Batch subreddit input (comma-separated). Auto-cleanup of empty reddit-tagged posts after dedup deletion. Fixed Content-Disposition headers for filenames with special characters (emojis, smart quotes, double quotes) using RFC 5987 filename*=UTF-8'' encoding with quote(filename, safe=''). Fixed total_media_found showing accumulated count instead of live COUNT(*). Standardized scheduler task ID to reddit_monitor. Added filesrouter/mediaidentifier/redditmonitor to log viewer with display names and default components. Dashboard and Scheduler pages updated with Reddit Monitor platform mapping. DB: 2 new tables (private_media_reddit_communities, private_media_reddit_history), 3 new indexes. Files modified: reddit_community_monitor.py (NEW), private_gallery_crypto.py, private_gallery.py, unified_database.py, scheduler.py, api.ts, Config.tsx, Logs.tsx, Scheduler.tsx, Dashboard.tsx.
2026-02-16 13.2.0 Profile Image Caching, FastDL Stories, TikTok Bio Fix, Log Viewer Fix - Minor release. New _cache_profile_image() method in scraper.py downloads creator avatars/banners locally during sync across all 11 platforms (Instagram, Coomer/Kemono, YouTube, PornHub, XHamster, TikTok, Twitch, Fansly add/sync, OnlyFans add/sync), eliminating dependency on expiring CDN URLs. Uses curl_cffi with browser TLS impersonation for Instagram CDN (plain requests blocked). New /api/paid-content/cache/profile-image/{filename} endpoint serves cached images with 7-day browser cache. Instagram stories switched from ImgInn to FastDL for real post timestamps (published_at from p.media-content__meta-time), ImgInn kept as fallback. FastDL's duplicate tracking disabled (use_database=False) since paid content has own deduplication. _files_to_posts() regex updated for both FastDL/ImgInn filename formats. TikTok bio emoji corruption fixed: replaced .encode().decode('unicode_escape') with json.loads() and added __UNIVERSAL_DATA_FOR_REHYDRATION__ JSON parsing as primary extraction. Log viewer get_logs endpoint fixed to handle YYYYMMDD_component.log format (was only matching YYYYMMDD_HHMMSS_component.log). Messages.tsx proxy wrapper fixed to skip local /api/ URLs. Instagram profile always refreshes during sync (removed stale guard). Added curl_cffi>=0.7.0 to requirements.txt and dependency updater. Added data/cache/profile_images/ to installer. Files modified: scraper.py, instagram_client.py, tiktok_client.py, paid_content.py, config.py (router), Messages.tsx, requirements.txt, dependency_updater.py, install.sh, update-all-versions.sh, core/config.py.
2026-02-16 13.1.1 PostgreSQL Boolean Fixes, Fansly Quality Recheck Scheduler, Dashboard Pinned Posts - Patch release. Fixed celebrity appearances ON CONFLICT error by creating partial unique indexes for TV and Movie appearance types. Fixed forum subprocess boolean handling (1 → TRUE for searches.active, downloaded, active). Added missing download_queue columns to PostgreSQL (thread_id, post_id, forum_name, file_hash, downloaded_date). Converted paid_content_posts.downloaded and forum_posts.has_images from integer to boolean; added both to pg_adapter _BOOLEAN_COLUMNS. Fixed FastDL URL redirect (/en → /en2). Fixed Fansly quality recheck scheduler: _auto_quality_recheck_background() was only called from API single-creator sync, not from sync_paid_content_all(). Added skip_pinned parameter to feed endpoints so Dashboard recent posts excludes pinned posts. Synced 11 missing fastdl/imginn accounts from SQLite to PostgreSQL. Files modified: appearances.py, forum_downloader.py, fastdl_module.py, pg_adapter.py, db_adapter.py, paid_content.py, Dashboard.tsx, api.ts, install.sh.
2026-02-15 13.1.0 PostgreSQL Compatibility Fixes, Discovery Queue, Security Hardening - Minor release. Comprehensive fixes for 15+ PostgreSQL edge cases: db_bootstrap.py loads .env before checking DATABASE_BACKEND (fixes all 5 entry points silently falling back to SQLite). pg_adapter datetime parameter interval translation fixed (%s double-escaped to %%s breaking forum thread monitoring). download_monitor.success converted from integer to boolean, SUM(success) rewritten as SUM(CASE WHEN...). Analytics DATE() returns Python date objects (added str() conversion). Paid content quality recheck text < timestamp fixed with explicit CAST. Passkey/2FA username column→user_id, schema type integer→text. error_tracking migration replaced sqlite_master check with information_schema constraint check. Review page and media gallery correlated subqueries replaced with pre-aggregated LEFT JOINs. Recycle bin COALESCE type mismatch fixed. Forum threads GROUP BY rewritten with proper subquery JOIN. strftime() → TO_CHAR() and datetime(col) → col::timestamp translations added to pg_adapter. fromisoformat() now handles datetime objects from PostgreSQL. Celebrity appearances boolean migration fixed. Duplicate /api/logs/context route removed. 5 missing DiscoverySystem queue methods added (get_queue_stats, get_pending_queue, add_to_queue, bulk_add_to_queue, clear_queue). Image proxy switched to suffix-based domain matching adding Fansly/TikTok/xHamster CDN support. SQL injection defense: server-side sort_order and table_alias validation. Boolean column translation (has_match/success/notified = 1 → = TRUE). Systemd services include DATABASE_BACKEND env vars. Files modified: pg_adapter.py, db_bootstrap.py, downloader_monitor.py, discovery_system.py, unified_database.py, passkey_manager.py, scheduler.py, paid_content.py, Creators.tsx, downloads.py, private_gallery.py, core/utils.py, config.py, review.py, media.py, db_adapter.py, CLAUDE.md.
2026-02-13 13.0.1 Dashboard Text-Only Post Placeholder Fix - Patch. Paid Content Dashboard grey media placeholder no longer renders on text-only posts (has_attachments=0). Added post.has_attachments === 1 condition gate. Files modified: Dashboard.tsx.
2026-02-13 13.0.0 PostgreSQL Migration, Private Gallery Fix, Code Cleanup - Major infrastructure migration. New pg_adapter.py (~840 lines) provides drop-in sqlite3 replacement via sys.modules monkey-patching when DATABASE_BACKEND=postgresql. SQL translation engine with LRU cache (4096 entries) handles 27+ SQLite→PostgreSQL dialect conversions (PRAGMA→no-op, datetime()→NOW(), GROUP_CONCAT→STRING_AGG, INSERT OR REPLACE→ON CONFLICT, json_extract→jsonb, etc.). _replace_question_marks() converts ? params to %s with proper string-literal awareness. PgConnection/PgCursor classes wrap psycopg2 with Row class providing full sqlite3.Row compatibility (index, name, keys, iteration). ThreadedConnectionPool (2-30 connections) with lazy init and double-check locking. db_bootstrap.py patches sys.modules['sqlite3'] at import time in all 5 entry points. All 87 tables from 6 SQLite databases (media_downloader.db, auth.db, scheduler_state.db, thumbnails.db, media_metadata.db, easynews_monitor.db) migrated to single PostgreSQL instance. Bug fixes: % escaping in pg_adapter (was only escaping outside quotes, psycopg2 needs all % escaped), cursor leak in commit()/rollback() (cursors created but never closed), removed unused imports (datetime, psycopg2.extras), added explicit psycopg2.errors import. Fixed Private Gallery feature toggle 400 error (stale /scraping-monitor in saved config rejected by validation; now silently filters removed features). Added _is_lock_error() helper in unified_database.py consolidating 9 inline lock-error checks to support both SQLite and PostgreSQL errors (deadlock detected, could not obtain lock). Added psycopg2-binary to requirements.txt and dependency updater. Updated installer with PostgreSQL setup. Cleaned up 6 empty .db files, stale CSV, moved sync_pg_schema.py to archive. Files modified: pg_adapter.py (NEW), db_bootstrap.py (NEW), unified_database.py, private_gallery.py, activity_status.py, requirements.txt, dependency_updater.py, install.sh, media-downloader.py, api.py, thumbnail_cache_builder.py, generate-embeddings.py, enrich_celebrity_metadata.py, CRITICAL_RULES.md, media-cache-builder.service.
2026-02-12 12.14.0 Streaming Decryption, Download Reliability, Auto-Migration - Minor release. Generator-based streaming decryption for all encrypted videos: decrypt_file_generator() yields 8MB decrypted chunks for chunked files or entire content for single-shot files via StreamingResponse, eliminating memory spikes for multi-GB videos. decrypt_file_range_generator() calculates which encrypted chunks overlap a byte range, seeks to positions, decrypts only needed chunks. Updated stream_video, get_file, and export_single endpoints to use streaming. Auto-migration on gallery unlock: background thread converts single-shot encrypted files >50MB to chunked format via re_encrypt_to_chunked() with atomic rename and unique temp filenames (secrets.token_hex(4)). Once-per-process guard prevents duplicate migrations. POST /migrate-chunked endpoint for manual triggering. Download reliability: stall detection with 30s read timeout, 3 retries with HTTP Range resume from last byte. All direct downloads now multi-threaded with 5 parallel segments (removed 5MB threshold). Fixed subprocess stdin double-close: replaced stdin.write()/stdin.close() + communicate() with communicate(input=...). Fixed duplicate filename check to verify .enc file exists on disk. Fixed sync_tmdb_appearances NoneType error in scheduler context: added db parameter, scheduler passes self.unified_db. Subprocess error→warning for FastDL/ImgInn/Toolzu/Snapchat timeouts/failures. Smart escalation for page load failures: warning normally, error after 5+ per session via _page_load_failures counter. Files modified: private_gallery_crypto.py, private_gallery.py, media-downloader.py, imginn_module.py, appearances.py, scheduler.py.
2026-02-11 12.13.0 Security Hardening, Message Read Tracking, Easynews Fix - Minor release. Comprehensive 12-point security audit across backend routers: fixed missing require_admin dependency in appearances.py, added CSRF enforcement to config/manual_import/files/video routers, corrected get_connection(for_write=True) on 8 write operations across celebrity.py/stats.py/dashboard.py/easynews.py/private_gallery.py/review.py, added path traversal validation to files.py serve endpoint, added rate limiting to paid_content.py sync/setup endpoints. New mark-as-read feature: IntersectionObserver on unread message bubbles (0.5 threshold) with 500ms debounced batch API calls via POST /messages/{creator_id}/mark-read, mark_messages_read() DB method, markMessagesRead() API client. Auto-scroll to first unread message on conversation open. Message sync now calls _download_message_attachments() for pending attachments after sync. Fixed Easynews search: removed strict content-type check in easynews_client.py that rejected valid JSON when Easynews API changed Content-Type from application/json to text/html. Fixed Dashboard dismiss button: added parseAsUTC() helper normalizing timezone-naive DB timestamps (e.g. 2026-02-11 12:25:21) to UTC before comparing with ISO dismiss timestamps (...Z), fixing timezone mismatch in shouldShowCard/getVisibleItems. Health check fixes: auto-check persists timeout/error to DB, single service handles timeout, stores 'down' not 'error', first load runs checks synchronously. Files modified: appearances.py, celebrity.py, config.py, dashboard.py, easynews.py, files.py, manual_import.py, paid_content.py, private_gallery.py, review.py, stats.py, video.py, api.py, easynews_client.py, db_adapter.py, Dashboard.tsx, Messages.tsx, Creators.tsx, api.ts, App.tsx, Configuration.tsx, Login.tsx.
2026-02-10 12.12.1 Health Check Reliability, Message Notifications, Sync Pagination Fix - Patch release. Health checks no longer block async event loop: rate limiter delays disabled for health check clients via _init_rate_limiter(min_delay=0), 30s per-service timeout with asyncio.wait_for(). Refactored 3 duplicate health check implementations into shared _check_single_service_health() function (~150 lines removed). Fixed OnlyFans scheduled sync paginating through ALL posts: timezone-aware vs timezone-naive datetime comparison in get_posts() silently failed in except block, now normalizes both to naive with .replace(tzinfo=None). Added push notifications for new messages during paid content sync: _send_creator_notification() accepts new_messages param, shows "💬 N New Messages" title for message-only syncs at low priority (-1), includes "💬 Messages: N" line in combined download notifications. PiP menu item hidden on desktop lightboxes (both Paid Content and Private Gallery), only shown on mobile. Files modified: scraper.py, onlyfans_client.py, paid_content.py, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx.
2026-02-10 12.12.0 Messages & Chat, OnlyFans Direct Setup, Health Check Auto-Trigger - Major feature release. New Messages/Chat page for viewing OnlyFans and Fansly creator direct messages with two-panel responsive chat UI (conversation list + chat thread), infinite scroll pagination, PPV/tip badges, and inline media lightbox. New paid_content_messages database table with message_id column on attachments. Message sync via OnlyFans GET /chats/{user_id}/messages and Fansly GET /messaging/groups + GET /message?groupId={id} APIs. New Message dataclass in models.py. DB adapter methods: upsert_message(), get_conversations(), get_messages(), get_pending_message_attachments(). OnlyFans Direct credential setup UI supporting manual cookie paste, HAR file upload, and Cookie-Editor JSON import with POST /onlyfans-direct/verify-auth and POST /onlyfans-direct/sync endpoints. Fansly Direct verify-auth and sync endpoints. Picture-in-Picture support in both BundleLightbox.tsx and PrivateGalleryBundleLightbox.tsx with Safari webkitSetPresentationMode fallback. Auto health checks: GET /services now triggers background checks when stale (>5min) via asyncio.create_task() with throttle. Mobile video unmute fix (start muted for autoplay policy, unmute after play()). Service worker caches Vite hashed JS bundles (cache-first). Multi-word search with COLLATE NOCASE in paid content and private gallery. Push notification platform names show "Fansly"/"OnlyFans" not "Fansly Direct"/"Onlyfans Direct" via _get_platform_display_name(). Feature enable/disable fix with known_features tracking. Search count functions (get_posts_count, get_media_count) fixed to include search/date filters. Settings page polls services every 60s. Cleaned up scheduler.py.migrated to archive. Files modified: unified_database.py, models.py, db_adapter.py, onlyfans_client.py, fansly_direct_client.py, scraper.py, paid_content.py, private_gallery.py, api.ts, App.tsx, Messages.tsx (NEW), Settings.tsx, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx, sw.js, install.sh.
2026-02-08 12.11.1 Lightbox Navigation Fixes & Gallery Filter Fix - Lightbox arrow buttons, keyboard arrows, and swipe gestures now navigate in shuffled order when shuffle mode is enabled (was always navigating in date order). Added shuffle-aware navigatePrev()/navigateNext() functions replacing handleManualNavigate(). Navigation arrows now render above video player (z-index raised from z-10 to z-20). Desktop arrows auto-hide with text-transparent hover:text-white pattern matching info/close buttons. Widened arrow hover detection from small circular p-2 targets to tall rectangular edge zones px-6 py-16 anchored to screen edges. Videos autoplay during slideshow in Paid Content lightbox. Private Gallery file_type filter now filters individual media items within posts (was only filtering which posts to show, returning all media types within matched posts). Code review fixes: added EXIF date extraction to URL import background task, fixed asyncio.run() → asyncio.new_event_loop() in background tasks, progress modal title shows 'Processing Imports' for URL imports, session keepalive runs during URL imports. Files modified: PrivateGalleryBundleLightbox.tsx, BundleLightbox.tsx, PrivateGalleryUploadModal.tsx, private_gallery.py.
2026-02-08 12.11.0 Config Page Redesign & Multi-URL Import - Private Gallery Config page redesigned from sidebar + pill tabs to horizontal underline tabs matching Configuration.tsx and Paid Content Settings.tsx pattern. Removed sticky header with back arrow, replaced with standard h1 + subtitle. Content area spans full tab width. Page root changed from own container to space-y-6 div matching parent layout. Upload modal now supports multi-URL import: textarea for pasting URLs (one per line), supports Bunkr/Pixeldrain/Gofile/Cyberdrop via FileHostDownloader + direct HTTP fallback. New importUrlsMutation, URL count display, Import/Upload/Create Post button switching. Backend: fixed ImportUrlRequest model (tag_ids now optional), added POST /import-urls endpoint, added _import_urls_to_gallery_background() background task with per-file download→hash→dedup→encrypt→thumbnail→DB insert pipeline. API client: added importUrls method, added 'import' to job operation types. Files modified: Config.tsx, PrivateGalleryUploadModal.tsx, api.ts, private_gallery.py.
2026-02-08 12.10.0 Slideshow Mode & Gallery Improvements - Added slideshow mode to both Private Gallery (PrivateGalleryBundleLightbox) and Paid Content (BundleLightbox) lightboxes with auto-advance, configurable interval (3s/5s/8s/10s), Fisher-Yates shuffle mode, and video-aware advancement. Slideshow toolbar button in both gallery pages is context-aware: gallery view slideshows all items across all posts, feed/social view slideshows the selected post only. Added total_media count to both Private Gallery and Paid Content backend APIs for accurate position indicators. Filter bar now shows "X items" in gallery view and "X posts" in feed/social view. Fixed gallery item ordering mismatch (galleryLightboxItems now sorts by media_date matching grid). Fixed Paid Content gallery not auto-loading pages on filter change. Added scroll-to-top on filter change. Slideshow controls hidden when only 1 item. Files modified: PrivateGalleryBundleLightbox.tsx, BundleLightbox.tsx, Gallery.tsx, Feed.tsx, api.ts, private_gallery.py, paid_content.py, db_adapter.py.
2026-02-07 12.9.0 Thumbnail Progress Modal & Async Gallery Operations - New shared ThumbnailProgressModal component replaces text-based progress displays with visual thumbnail-grid progress tracking across Copy to Gallery, Upload, and Manual Import. Private Gallery copy and upload operations now run asynchronously via BackgroundTasks with in-memory job tracking and 500ms polling. New GET /private-gallery/job-status/{job_id} endpoint with dual auth. Added 3 Instagram filename patterns for browser extension formats (video/photo/shortcode). Fixed video thumbnail generation for short videos (fallback seek time). Added temp/manual_import to ALLOWED_PATHS. Fixed ThrottledImage IntersectionObserver in nested scroll containers. private_gallery.py grew from ~1,800 to 3,696 lines.
2026-02-06 12.7.5 Code Review: Bug Fixes & Code Quality - Fixed critical bug in shared get_or_create_thumbnail() in core/utils.py where it queried non-existent content_hash column instead of correct file_hash primary key (caused cache miss on every request, regenerating thumbnails unnecessarily). Fixed get_current_user_media() token parameter missing Query() annotation (query param tokens from <img>/<video> tags silently ignored). Fixed 6 direct sqlite3.connect() calls without context managers in media.py (5) and video_queue.py (1) that could leak connections on exceptions. Fixed deprecated datetime.utcnow() (2 locations) and import ordering in core/responses.py. Added missing @handle_exceptions to /api/health endpoint. Added WAL checkpoint support for non-pool database connections. Consolidated 9 duplicate Pydantic models from api.py into models/api_models.py imports. Removed 3 duplicate thumbnail functions from media.py (now uses shared core/utils.py). Added debug logging to 4 bare except: pass blocks in downloads.py and review.py. Fixed 4 incorrect file paths in Worker Process Architecture Proposal document.
2026-02-04 12.7.1 Security & Code Quality Fixes - Fixed critical authentication bypass in dashboard.py where underscore-prefixed _user parameter was ignored by FastAPI (now uses current_user). Fixed missing authentication on public thumbnail endpoint in files.py. Fixed 7 SQLite connection leaks across auth.py, scheduler.py, health.py, and recycle.py (converted to context managers). Fixed SQL injection risk in paid_content/db_adapter.py by adding column whitelist validation. Fixed 5 bare except: clauses in unified_database.py to catch specific sqlite3.OperationalError. Fixed bare except in paid_content JSON parsing to specific exceptions. Consolidated duplicate ThumbnailLRUCache class into shared core/utils.py. Consolidated path validation into shared validate_file_path() function. Fixed forum_db_adapter _execute_with_retry() to properly raise instead of returning None. Deprecated logger usage updated in dashboard.py.
2026-02-02 12.6.6 Unified Filter Bar, Expanded Search & Creator Bios - Created unified FilterBar component with clean 'Filter Button + Popover' design across all pages (Downloads, Media, Review, Download Queue, Channel Monitors, Appearances, Recycle Bin, Notifications, Paid Content Feed/Creators/RecycleBin). Expanded search functionality now queries platform, source, content_type, URL, and description fields (not just filename). Fixed z-index stacking context issue with filter popover using React Portal to render at document body level. Fixed Dashboard 'New in Media/Review/Discovery' card lightboxes to have same action buttons as main pages (added Quick Add Face and Delete buttons). Added creator bio/description support for Fansly, YouTube, and Twitch - bios are fetched during sync and shown in popup modal on Creators page. Added proxy cache clearing documentation to CLAUDE.md and CRITICAL_RULES.md.
2026-02-02 12.6.5 Universal Logger Migration - Migrated 3 modules to universal logger (easynews_client.py, media_identifier.py, web/backend/routers/files.py) that were still using standard logging.getLogger(). Removed unused import logging from 7 modules (activity_status.py, date_utils.py, dependency_updater.py, forum_db_adapter.py, pushover_notifier.py, settings_manager.py, unified_database.py). All 78 core modules now use universal logger for web UI log visibility.
2026-02-01 12.6.2 Comprehensive Code Review Bug Fixes - Fixed RecycleBin.tsx API response mismatch (deleted_count accessed incorrectly). Added missing @handle_exceptions decorators to 3 dashboard.py endpoints (get_recent_items, get_dismissed_cards, set_dismissed_cards). Replaced 89 logger.log() calls with proper logger.debug/info/warning/error methods (video_queue.py: 39, celebrity.py: 50). Added logging to 3 silent exception handlers (config.py broadcast, health.py scheduler state and cache builder status). Removed unused import logging from 5 modules (move_module.py, download_manager.py, face_recognition_module.py, immich_face_integration.py, service_health_monitor.py). Added default cookie path to toolzu_subprocess_wrapper.py for consistency. Added config key validation to forum_subprocess_wrapper.py and base_subprocess_wrapper.py (returns error if required keys missing).
2026-01-31 12.2.4 Fansly 4K Video & Image Quality Fixes - Fixed critical 4K video download issue where CloudFront signed HLS streams failed because segment URLs didn't include signed params. Scraper now fetches variant playlist, normalizes CRLF line endings, and rewrites segment URLs with signed params. Fixed image quality issue where images were downloaded from 'variants' (scaled thumbnails maxing at 1080p) instead of 'main locations' (original full resolution up to 1875x2500). Fixed push notifications to correctly show post count and include image/video attachment preview. Added real-time progress tracking for HLS/m3u8 streaming downloads by parsing EXTINF duration tags and ffmpeg progress output. Created utility scripts: check_4k_upgrades.py, check_4k_batch.py, upgrade_fansly_to_4k.py (4K video upgrades), check_image_upgrades.py (image upgrades), refresh_fansly_post_urls.py (refresh expired signed URLs). Archived PostgreSQL migration scripts.
2026-01-30 12.2.3 Podcast Notification Poster Fix & Taddy Improvements - Podcast push notifications now include artwork images (was broken - URL was incorrectly prefixed with TMDb base URL). Taddy API fallback account now properly triggers on 500 errors (catches ServiceError exception). Taddy API response parsing updated for new 'search' query format (was using 'searchForTerm'). Taddy filtering changed from blacklist to whitelist - only accepts episodes matching interview patterns. Taddy API uses EXACT_PHRASE matching for more accurate celebrity name searches. Taddy API usage reduced (lookback 730→180 days, max results 250→100). Appearance detail modal redesigned for mobile/iOS with bottom sheet pattern and touch-friendly buttons. New Taddy fallback account support for quota management.
2026-01-29 12.2.2 Mobile Touch Sensitivity & Lightbox Fixes - Feed page 'pages.length' error fixed with defensive null checks for infinite query. Mobile lightbox close button now visible (zoom controls hidden on mobile via hidden sm:flex). Mobile Back button improved with larger touch target (44x44px) and onTouchEnd handler. Feed items no longer open accidentally while scrolling - added touch movement detection with 20px threshold and 50ms minimum duration. Thumbnails no longer open lightbox accidentally while scrolling (same threshold). Thumbnail taps properly isolated from post selection (event propagation stopped). Desktop click handlers separated from touch handlers.
2026-01-29 12.2.1 Notifications Page Enhancements & Scheduler Fixes - Scheduler now prioritizes most overdue task instead of alphabetical order. Scheduler preserves original scheduled time for missed tasks (was resetting to now). Push notification import fixed (was importing from non-existent modules.notifier, now modules.pushover_notifier). Notifications page newlines now render correctly (whitespace-pre-line CSS). Duplicate push notifications prevented (only sends when files actually downloaded, not when new_posts > 0). YouTube scheduled sync 10x faster with --dateafter and --playlist-items 1:20 filters. Twitch scheduled sync faster with --dateafter and --playlist-items 1:50 filters. Notification lightbox shows full metadata (resolution, size, post content, date, platform). Notification thumbnails show Feed-style overlay (filename, size, resolution, duration). Lightbox resolution falls back to loaded image dimensions when metadata unavailable. New /thumbnail and /preview endpoints for serving files by path. New get_attachment_by_path() method in db_adapter.
2026-01-27 12.1.0 Fansly Direct API Integration - Major new feature for downloading content directly from Fansly API (bypasses Coomer archives). FanslyDirectClient class with full API support (auth, posts, media, bundles). 4-hour scheduled sync checking last 3 days for new content. Push notifications with image attachment preview when new content downloaded. Fansly auth token configuration in Settings page. Service health check validates token and shows connected username. Added no_description column to exclude posts without descriptions from filter. YouTube and Twitch scheduled syncs now only check last 3 days (performance optimization). Fansly Direct client skips non-media content types (contentType 42001). Creators page shows "Fansly" instead of "fansly_direct". "View on Fansly" links to creator's profile. PPV filenames renamed from random Coomer-style to date-based format (YYYY-MM-DD).
2026-01-26 12.0.1 Cloudflare Cookie Fingerprint Fix - Critical fix for cf_clearance cookies not working. Fixed user_agent not being saved with cookies in imginn_module.py, toolzu_module.py, and snapchat_scraper.py. Added key_value_store database table for persistent settings. Added get_setting()/set_setting() to UnifiedDatabase. FlareSolverr fingerprint now persists to database (instant loading on restart). Added set_fingerprint_database() initialization in scheduler. All Playwright scrapers now use consistent fingerprinting from FlareSolverr.
2026-01-23 12.0.0 Paid Content Feature - Creator Platform Archival - Major new feature for tracking and downloading content from subscription-based creator platforms. Integrates with Coomer.su (OnlyFans, Fansly, CanDFans) and Kemono.su (Patreon, Fanbox, Gumroad, SubscribeStar, Discord). YouTube channel support via yt-dlp. 8 new pages: Dashboard, Feed, Creators, AddContent, Notifications, Settings, Queue, RecycleBin. Split-view feed browser with gallery/thumbnail toggle. Identity system for linking same creator across platforms. 10 new database tables. New paid_content module (api_client, db_adapter, scraper, file_host_downloader, embed_downloader, youtube_client). 25+ new API endpoints. Favorites system. Manual import from Bunkr, Pixeldrain, Gofile, Cyberdrop. iOS safe area fixes for mobile feed view.
2026-01-15 11.29.1 Dashboard Cards Cross-Browser Sync & Fixes - Dashboard cards now exclude items moved between Review/Media (added moved_from_review/moved_from_media flags). Removed catch-up notification feature. Cross-browser dismiss sync via server-side storage in user_preferences table. New GET/POST /api/dashboard/dismissed-cards endpoints. Optimistic updates for instant UI feedback.
2026-01-15 11.29.0 Breadcrumb Navigation System - Context-aware breadcrumb navigation below header on all 23 pages. Dynamic updates based on user actions (media type filter, celebrity selection). New BreadcrumbContext, Breadcrumb component, useBreadcrumb hook, and breadcrumbConfig. All levels clickable for navigation. Long labels truncate with ellipsis.
2026-01-15 11.28.0 Dashboard New Items Cards - Three dismissible cards showing recent additions with thumbnail previews (New in Media, New in Review, New in Internet Discovery). Click thumbnails to open lightbox or preview modal. Hover overlay with action buttons. Session-based tracking removes actioned items. Backend /api/dashboard/recent-items endpoint. New RecentItemsCard component.
2026-01-14 11.27.0 Performance Optimizations & Frontend Modernization - Review page SQL query fixed (158x faster). 3-tier thumbnail caching (memory → SQLite → generate). Thumbnails pre-generated during MoveManager processing. ThrottledImage optimized with shared IntersectionObserver. API retry logic with exponential backoff. New shared FilterBar/MediaGrid components. SuspenseBoundary with skeleton loading. useMediaFiltering hook with useTransition/useDeferredValue.
2026-01-13 11.26.3 Cross-Process Sync Status & Notification Fixes - Fixed dashboard cards not showing for scheduler-triggered TMDB/Easynews syncs (cross-process issue). Now uses ActivityStatusManager for database-based status tracking. TMDB sync now runs in background thread (non-blocking). Easynews notifications only sent for truly new content (not quality upgrades). Fixed Easynews thumbnail URL storage. Fixed YouTube channel monitor notifications not being sent from scheduler.
2026-01-10 11.26.2 TMDb Appearance Prediction Fix - Fixed false positive appearance predictions where guest/host credits predicted future episodes incorrectly. Added get_episode_credits() and is_person_in_episode() methods to TMDbClient. Now verifies celebrity is in specific episode credits before adding prediction. Removed 10 incorrect upcoming appearances.
2026-01-10 11.26.1 Security Hardening & Code Cleanup - Fixed SQL injection risks in maintenance.py (whitelist validation) and media.py (explicit SQL branches). Added require_admin to scheduler restart. Added missing for_write=True on video DELETE. Forum monitoring now uses configured auto_track_days. Subprocess wrappers refactored with factory pattern (67% code reduction). forum_db_adapter now uses _execute_with_retry consistently. Added utility functions to core/utils.py and core/responses.py.
2026-01-09 11.26.0 Snapchat Direct Scraper - Complete rewrite of Snapchat downloading. New Playwright-based scraper replaces StoryClon proxy module. Supports Stories and Spotlight content. EXIF metadata preservation. Removed video stitching feature (individual clips now). Archived old snapchat_module.py. Updated docs (README, SCRAPER_PROXY_SYSTEM, DOWNLOADER_MONITORING).
2026-01-04 11.23.2 API Module Import Fix - Fixed critical bug where all API endpoints returned 500 errors due to inconsistent module import paths. api.py was importing from core.dependencies (relative) while routers used ..core.dependencies, resolving to different Python modules and creating separate AppState singletons. Changed to consistent web.backend.core.dependencies absolute imports. Removed 3 orphaned Python cache files (imdb_client, tvdb_client, picturepub_workaround).
2026-01-04 11.23.1 ImgInn Download All Posts Option - Added "Download All Posts (skip phrase filtering)" toggle to ImgInn phrase search configuration. When enabled, all posts from specified usernames are downloaded without phrase matching. UI dynamically hides phrase-related fields when Download All is enabled.
2026-01-04 11.23.0 Appearances Navigation & Midnight Reminders - Promoted Appearances to main navigation bar. Reminders now sent at midnight for same-day appearances only. Added scheduler schedule_daily_at_time() method. Added Plex 'Watch' button to notification cards. Fixed celebrity filter default from 'Eva Longoria' to 'all'. Fixed credit_type TypeScript type inconsistencies. Fixed bare except clauses in appearances.py. Fixed memory leak with Plex match polling cleanup. Added missing sync status fields to API types. Notification stats cards now match main Notifications page design.
2026-01-02 11.22.4 Recycle Bin & Appearances Enhancements - Fixed Review page batch delete failing when >500 files (now chunks into batches of 500). Fixed recycle bin lightbox showing deletion date instead of download date. Added download_date and post_date sort options to recycle bin (default: download_date). Added "On Plex" filter toggle to Appearances page. Fixed Plex filter to only show items with actual episode/movie match (not just show match).
2026-01-02 11.22.3 Plex Integration Fixes & Automation - Fixed Plex matching for TV show rating keys. Fixed false Plex links for episodes not in library. Removed Plex links from upcoming appearances. Added automatic Plex matching via systemd timer (6am/6pm).
2026-01-01 11.22.2 Codebase Cleanup & Dead Code Removal - Removed 3 unused Python modules (imdb_client.py, listen_notes_client.py, picturepub_workaround.py). Removed unused timeline_cache table. Removed orphaned /presets page. Cleaned up 34 unused API wrapper methods in api.ts (~327 lines, 12% reduction). No functional changes.
2025-12-28 11.22.0 Manual Platform Run Monitoring - New dashboard card showing real-time progress for manual downloads. Platform buttons persist running state across page navigation with stop functionality. Live Scraping Monitor loads active sessions on mount. Backend tracking with GET /api/platforms/running, POST /api/platforms/{platform}/stop, GET /api/scraper-sessions/active endpoints. Session cleanup fixed to prevent duplicates. Black glass button styling for consistency.
2025-12-28 11.21.1 Coppermine Cloudflare retry fix - Added automatic retry mechanism for Cloudflare challenges during downloads. New _request_with_retry() method detects and handles challenges in real-time with fresh cookies.
2025-12-28 11.21.0 Server-side search & channel management - Added "Add Channel" button to YouTube Channel Monitors page with modal. All pages now use server-side search (Internet Discovery, Search Presets). Fixed scheduler duplicate task. Search queries now search entire database instead of current page.
2025-12-26 11.20.0 YouTube Channel Monitor Management - Dedicated channel monitors page with auto-pause, always-active flag, bulk actions, status filters
2025-12-26 11.19.4 Configuration & Monitoring fixes - Fixed loading issues on Configuration tabs. Service Health Monitoring now uses consecutive runs instead of time windows. Removed scan interval from Error Monitoring. Face recognition hidden on Media page.
2025-12-26 11.19.3 Face recognition fixes - Lightbox now shows confidence for non-matched items. Fixed review API SQL JOIN to use filename matching. Fixed move_module UPDATE query for face scan paths.
2025-12-26 11.19.2 Logging refactoring - Created LoggingMixin, standardized logging across 13 modules. Hash standardization - All MD5 changed to SHA256 for consistency (11 files). Fixed instagram_repost_detector inverted logging priority.
2025-12-26 11.19.1 Server-side search for Media/Downloads/Review, fixed Dependencies section on Health page, removed 4 unused files, TypeScript fixes
2025-12-22 11.17.0 Dashboard stats fix - total downloads now includes all sources (media+review+recycle), file ownership/permissions fix
2025-12-21 11.16.0 Search boxes on Downloads/Internet Discovery, server-side Video Downloader search, jpg thumbnail standardization
2025-12-21 11.15.0 Portrait video resolution label fix, Download Queue thumbnail caching via backend
2025-12-21 11.14.0 Config page hover fix, Video Downloader settings now honored, removed unused Download Settings
2025-12-21 11.16.0 Semantic versioning overhaul - 246 versions renumbered for correct PATCH/MINOR/MAJOR classification
2025-12-20 11.13.1 Renamed Celebrity Discovery → Internet Discovery, removed unused config sections
2025-12-20 11.13.0 Face recognition path fix for Review page, parse_with_fallbacks() for Instagram filename patterns
2025-12-20 11.12.0 Downloads page batch operations, Instagram Stories filename fix, per-day summary stats, dead code cleanup
2025-12-20 11.11.0 Modern UI styling (custom scrollbars, animated counters, thumbnail zoom, glass cards), codebase cleanup (11 unused files removed)
2025-12-17 11.10.2 Code review security & stability fixes - cookie security, path traversal, thread safety, connection leaks
2025-12-17 11.10.0 Dynamic aspect ratio for embedded videos, max_width database column, yt-dlp width capture
2025-12-17 11.9.0 Forum filename date prefix, proportional video sizing, GIF fixes, forum notification fix
2025-12-16 11.8.1 Mobile/iOS Configuration UI, YouTube Monitor notification and app_state fixes
2025-12-14 11.8.0 Dailymotion HLS streaming, centralized timestamps, Dashboard video queue card
2025-12-13 11.7.0 YouTube scheduler interval fix, Search Presets Full Scan button, Instagram duplicate recording fix
2025-12-13 11.6.1 Scheduler UI fix - YouTube Monitor display name and description
2025-12-13 11.6.0 Video download timezone fix - UTC instead of local time
2025-12-13 11.5.1 YouTube monitor auto-start queue fix, direct queue processor access, service documentation
2025-12-13 11.5.0 YouTube Channel Monitor feature, thumbnail jpg fix, recycle bin metadata restore, error notification context fix
2025-12-12 11.4.0 Post date editing from lightbox, batch date changes, deferred database recording for reliability
2025-12-11 11.3.1 Notifications 503 fix with lazy loading, silent exception handling improvements, proxy rate limits
2025-12-10 11.3.0 Celebrity Discovery UX improvements, forum date fixes, queue settings persistence
2025-12-10 11.2.0 Anti-bot protection for yt-dlp, queue sort toggle, stuck download reset, cookie expiration handling
2025-12-09 11.1.0 Download Selected feature, configurable base directory, mobile menu safe area fix
2025-12-09 11.0.0 Major release - Celebrity Discovery, Mobile UI, Native Video Player, Unified Download Queue
2025-12-07 9.1.0 Video streaming settings UI, default player selection, queue resolution fixes
2025-12-07 9.0.0 Native video player with yt-dlp streaming, privacy-focused YouTube alternative
2025-12-07 8.23.3 Critical memory leak fixes - OOM kills resolved, error banner dropdown fix
2025-12-07 8.23.2 UI consistency update - standardized page headers, menu icons, layouts
2025-12-07 8.20.0 Recycle Bin platform/source filters, Celebrity Discovery system
2025-12-06 8.18.1 Error banner fixes, Instagram username extraction, memory optimizations
2025-12-05 8.17.0 API modularization complete, error monitoring UI, dashboard fix
2025-12-05 8.14.0 Phase 1 refactoring complete, debt tracker updated
2025-12-04 8.13.0 Initial comprehensive review

Key Findings by Version

13.8.1

  • Inline Audio Player:
    • PostDetailView: native HTML5 audio player with filename, file size, playback controls
    • Feed PostCard: compact mobile-only audio player with touch event isolation (stopPropagation)
    • Audio attachments excluded from thumbnail grid and BundleLightbox (image/video only)
  • Push Notification Audio Support:
    • Notification code now checks content_type from scraper metadata before falling back to file extension
    • Fixes .mp4 files with audio content_type being misidentified as video
    • New 🎵 audio category in notification titles, pushover PLURALS/CONTENT_ICONS updated
    • Fixed hardcoded 'content_type': 'video' in 3 downloaded_file_info append sites to use att.get('file_type', 'video')
  • Service Health Check Fixes:
    • HQCelebCorner/PicturePub: dedicated health checks accepting HTTP 200/403 (FlareSolverr used at runtime)
    • Snapchat: checks story.snapchat.com reachability (200/301/302)
    • Fixes discrepancy between paid dashboard ("degraded") and scrapers page ("healthy")
  • Reddit Community Monitor Reliability:
    • Switched from OAuth API (shared rate-limited client-id 6N9uN0krSDE-ig) to REST API mode (extractor.reddit.api=rest)
    • Added --range 1-200 limit to prevent full history pagination timeouts
    • Fixed cookie file write error with os.makedirs(temp_dir, exist_ok=True)
  • Code Quality:
    • Fixed function name shadowing: get_preset_categories in celebrity.py, get_queue_settings in video_queue.py
    • Fixed backup profile stale paths in add-backup-profile.sh
  • Files modified: scraper.py, pushover_notifier.py, reddit_community_monitor.py, paid_content.py, celebrity.py, video_queue.py, PostDetailView.tsx, Feed.tsx, install.sh, add-backup-profile.sh

13.8.0

  • XenForo Forum Support:
    • Generic XenForo-based forum client (xenforo_forum_client.py) for celebrity image forums
    • HQCelebCorner/PicturePub compatibility layer (hqcelebcorner_client.py)
    • External image host resolution (ImageBam, Pixhost, Imgur, imgbox, postimg, etc.)
    • FlareSolverr Cloudflare bypass, CSRF token handling, junk URL filtering
  • Snapchat HTTP Client:
    • Direct curl_cffi-based client replacing Playwright dependency (snapchat_client.py)
    • NEXT_DATA JSON extraction from story.snapchat.com
    • EXIF metadata preservation including GPS, view counts, timestamps
  • TikTok Hybrid Client:
    • Unified yt-dlp (listing) + gallery-dl (downloading) interface (tiktok_client.py)
    • Automatic cookie refresh and merge-back to database
    • Carousel/slideshow detection, creator profile scraping
  • Instagram Direct API Module:
    • Direct GraphQL + REST API via curl_cffi (instagram_client_module.py)
    • No Playwright or Cloudflare dependency, 10-20x faster than ImgInn
    • Stories/reels/tagged via REST API, public posts via GraphQL
  • Cookie Health Monitoring:
    • Comprehensive health endpoint with per-platform monitoring toggles
    • Background health checks with 30-min cooldown, WebSocket + Pushover alerts
    • Scraper module enable/disable detection to avoid false positives
  • ImgInn Adapter Enhancements:
    • Two-phase post discovery (API + HTML profile scraping)
    • Detail page fetching for full-res 1440p+ CDN URLs
    • Direct Instagram tag fetching with NordVPN proxy rotation
  • Bug Fixes:
    • FastDL browser fallback story duplicates (pk mapping + MD5 hash fallback)
    • Recycle bin restore metadata inference (platform from path, source from filename)
    • Square video display (1:1 aspect ratio treated as portrait)
    • Failed downloads missing creator_db_id
    • Function name shadowing in celebrity.py and video_queue.py
  • Cleanup:
    • Removed 30 empty log files, empty unified.db, cleared pycache directories
    • Fixed stale backup profile paths in add-backup-profile.sh
  • Files Created: xenforo_forum_client.py, hqcelebcorner_client.py, bellazon_client.py, soundgasm_client.py, snapchat_client.py (paid_content), tiktok_client.py, youtube_client.py, backfill_tagged_users.py, task_checkpoint.py
  • Files Modified: fastdl_module.py, scraper.py, db_adapter.py, unified_database.py, imginn_adapter.py, imginn_api_module.py, snapchat_client_module.py, instagram_client_module.py, health.py, scrapers.py, paid_content.py, PostDetailView.tsx, BundleLightbox.tsx, celebrity.py, video_queue.py, add-backup-profile.sh

13.7.0

  • Per-Creator Instagram Sync Settings:
    • Toggle sync for posts, stories, and highlights individually from creator card menu
    • Settings stored as database columns with defaults preserving existing behavior
  • Tagged User Filter:
    • Per-creator feed filter that only shows posts where selected tagged users appear
    • Multi-select UI with username autocomplete from existing tagged data
  • Appearance Role Combining:
    • Shows with multiple roles display as single entry with all roles
    • Episode counts use COUNT(DISTINCT) to avoid inflation
    • Consistent badge colors via shared creditTypeConfig

13.6.0

  • ImgInn Paid Content Adapter:
    • New imginn_adapter.py replaces Instagram API-based adapter with ImgInn-based scraping
    • No Instagram credentials needed — uses ImgInn's CDN URLs for posts, stories, reels
    • Maps ImgInn API data to existing Post/Attachment models
    • Built-in rate limiting and Cloudflare bypass
  • ImgInn API Optimizations:
    • FlareSolverr integration for reliable Cloudflare bypass
    • OCR disabling for faster scraping performance
    • Date pre-filtering to skip old posts during sync
    • Profile resolution improvements
  • Reddit Timezone Fix:
    • gallery-dl metadata dates were stored as naive UTC strings without timezone conversion
    • Added UTC→local conversion using datetime.timezone
    • Fixed created_at to use import time instead of post date
    • Bulk-fixed 556 existing reddit posts and 1366 media entries via decrypt→convert→re-encrypt cycle
  • Lightbox Improvements:
    • Fixed slideshow back arrow not showing — parent-managed shuffle mode set shuffleEnabled=true but never built shuffleOrder
    • Fixed paid content bundle sidebar invisible on desktop — z-index stacking context (added relative z-20 to sidebar)
    • Bundle sidebar auto-collapses when slideshow starts, hidden for single-image posts
    • Expand/collapse with floating button, slideshow controls stay stationary regardless of sidebar state
  • Dashboard: Improved platform labels for better display naming
  • Cleanup:
    • Removed 20 stale scripts (one-off 4K checks, Fansly maintenance, enrichment, test scripts)
    • Removed 15 stale utilities (completed backfills, cleanups, test scripts, standalone db_manager)
    • Removed legacy SQLite database file (database/media_downloader.db)
    • Removed archive/ directory (migrated scheduler + migration scripts)
    • Cleared all __pycache__ directories
  • Files Modified: imginn_adapter.py (NEW), reddit_community_monitor.py, private_gallery.py, BundleLightbox.tsx, PrivateGalleryBundleLightbox.tsx, Gallery.tsx, Feed.tsx, Dashboard.tsx

13.5.1

  • Landscape Lightbox for Private Gallery:
    • Ported all landscape phone detection from BundleLightbox.tsx to PrivateGalleryBundleLightbox.tsx
    • Touch-based device detection: isMobile (portrait phone), isLandscape (touch + height<500px), desktop (neither)
    • Fixed positioning for landscape containers using 100dvh for mobile browser chrome compatibility
    • Controls use text-white/70 on landscape (hover doesn't work on touch devices)
    • Navigation arrows: compact p-3 bg-black/40 rounded-full with w-7 h-7 icons
    • Image container sidebar offset: left: 6rem accounts for w-24 sidebar
    • Video bottom offset: isMobile ? 67 : isLandscape ? 50 : 120
  • Per-Person-Group Min Resolution Filter:
    • SQL filter: (file_type != 'image' OR (width >= ? AND height >= ?)) — shows all videos, filters only images
    • Gallery filter dropdown with preset options (300/500/720/1080px+) and active filter chip
    • Import skip: low-res images skipped during URL/upload import with status tracking
    • Setting stored per person group in config
  • TypeScript Fixes:
    • Added min_resolution?: number to getPersonGroups() and getPersonGroup() return types in api.ts
    • Removed unused editMinRes state variable from Config.tsx
    • Fixed task.next_run null safety in Scheduler.tsx (|| new Date().toISOString() fallback)
  • Cleanup:
    • Removed data/migration_backup.json (682KB stale migration data)
    • Removed archive/media_downloader.db.old (308KB obsolete database)
    • Removed all __pycache__ directories (~5MB across 9 directories)
  • Files Modified: PrivateGalleryBundleLightbox.tsx, private_gallery.py, Gallery.tsx, Config.tsx, api.ts, Scheduler.tsx

12.15.0

  • UsernameListEditor Component:
    • New reusable UsernameListEditor.tsx component replacing 6 comma-separated username fields
    • Add usernames via text input + Enter key or + button, all normalized to lowercase
    • Alphabetically sorted display with scrollable container (max-h-48) and custom-scrollbar
    • Duplicate detection with amber "DUP" badges and one-click "Remove Duplicates" button
    • "Paste List" modal for bulk import of usernames separated by commas or newlines
    • User count badge in header, empty state with dashed border placeholder
    • Remove individual usernames via X button on hover
  • Configuration Page Cleanup:
    • Replaced FastDL regular + phrase search username fields (2 textareas)
    • Replaced ImgInn regular + phrase search username fields (1 input + 1 textarea)
    • Replaced Toolzu username field (1 textarea)
    • Replaced Snapchat username field (1 textarea)
    • FastDL and ImgInn phrase search usernames converted from uncontrolled (defaultValue/onBlur) to controlled components
    • ImgInn phrase search label dynamically switches based on download_all setting
  • Files Created: web/frontend/src/components/UsernameListEditor.tsx (155 lines)
  • Files Modified: web/frontend/src/pages/Configuration.tsx (6 field replacements + import)

12.14.0

  • Streaming Decryption for All Encrypted Videos:
    • decrypt_file_generator() yields 8MB decrypted chunks for chunked format files
    • decrypt_file_range_generator() calculates encrypted chunk indices from byte offsets for Range requests
    • stream_video, get_file, export_single endpoints all use StreamingResponse
    • Eliminates full-file memory loading for multi-GB encrypted videos
  • Auto-Migration to Chunked Encryption:
    • On gallery unlock, background thread converts single-shot files >50MB to chunked format
    • re_encrypt_to_chunked() in crypto module with atomic rename
    • Unique temp filenames (secrets.token_hex(4)) prevent race conditions from double unlock
    • Once-per-process guard via getattr(unlock_gallery, '_migration_done', False)
    • POST /migrate-chunked endpoint for manual triggering
  • Download Reliability Improvements:
    • Stall detection: 30-second read timeout on requests.get() via timeout=(30, 30) tuple
    • Retry logic: up to 3 retries with HTTP Range resume from last downloaded byte
    • All direct downloads multi-threaded with 5 parallel segments (removed 5MB size threshold)
  • Bug Fixes:
    • Subprocess stdin double-close: communicate(input=...) replaces manual stdin.write()/stdin.close()
    • Duplicate filename check verifies .enc file exists on disk
    • sync_tmdb_appearances gets db parameter for scheduler context (was NoneType error)
    • Re-encryption race condition fixed with unique temp filenames
  • Log Level Tuning:
    • FastDL, ImgInn, Toolzu, Snapchat subprocess errors → warnings
    • Page load timeout/failure: warning normally, error after 5+ per session (_page_load_failures counter)
    • "Failed to download single image" 403 → warning
  • Files Modified:
    • modules/private_gallery_crypto.py - Streaming generators, re-encryption method
    • web/backend/routers/private_gallery.py - Streaming endpoints, duplicate check, download retry, migration
    • media-downloader.py - Subprocess stdin fix, error→warning changes
    • modules/imginn_module.py - Error→warning, page load failure counter
    • web/backend/routers/appearances.py - sync_tmdb_appearances db parameter
    • modules/scheduler.py - Pass db to sync_tmdb_appearances

12.11.1

  • Lightbox Shuffle-Aware Navigation:
    • Arrow buttons, keyboard arrows, and swipe gestures now respect shuffle order when shuffle mode is enabled
    • Added navigatePrev()/navigateNext() callbacks with shuffleOrder/shufflePosition tracking
    • Removed unused handleManualNavigate() function (caught by TypeScript)
    • Mobile position indicator shows shuffle position when shuffle active
  • Lightbox Arrow UX Improvements:
    • z-index raised from z-10 to z-20 to render above video player elements
    • Desktop arrows auto-hide (text-transparent hover:text-white) matching info/close button pattern
    • Hover detection area widened from p-2 circular to px-6 py-16 edge-anchored rectangular zones
    • Subtle hover:bg-white/5 feedback on desktop arrows
  • Video Autoplay in Slideshow:
    • Paid Content lightbox: videos autoplay when slideshowPlaying is true
    • Added slideshowPlaying to autoPlay condition and video.play() call
  • Private Gallery File Type Filter Fix:
    • Backend media query now includes file_type filter when fetching media items for posts
    • Previously only filtered which posts to show (via EXISTS subquery) but returned all media within those posts
    • Now selecting "Images" correctly hides videos within mixed-media posts
  • URL Import Code Review Fixes:
    • Added EXIF date extraction (_extract_date_from_exif()) before filename-based extraction
    • Fixed asyncio.run() → asyncio.new_event_loop() to avoid RuntimeError in background tasks
    • Progress modal title shows "Processing Imports" for URL imports
    • Session keepalive timer runs during URL imports and job polling
  • Files Modified:
    • web/frontend/src/components/private-gallery/PrivateGalleryBundleLightbox.tsx - Shuffle nav, arrow UX
    • web/frontend/src/components/paid-content/BundleLightbox.tsx - Shuffle nav, arrow UX, video autoplay
    • web/frontend/src/components/private-gallery/PrivateGalleryUploadModal.tsx - Import title, keepalive
    • web/backend/routers/private_gallery.py - File type filter, EXIF date, asyncio fix

11.27.0

  • Review Page SQL Query Fix:
    • Root cause: LEFT JOIN face_recognition_scans frs ON frs.file_path LIKE '%' || fi.filename caused full table scans
    • Changed to: LEFT JOIN face_recognition_scans frs ON frs.file_path = fi.file_path (exact matching)
    • Result: 3.475 seconds → 0.022 seconds (158x faster)
  • 3-Tier Thumbnail Caching:
    • In-memory LRU cache with 500 item limit and 100MB size cap
    • SQLite thumbnails.db as persistent storage (19,225 thumbnails, 401MB)
    • On-demand generation as fallback
    • Cache hierarchy: Memory → SQLite → Generate
  • Thumbnail Pre-generation:
    • _generate_thumbnail_cache() method added to MoveManager
    • Thumbnails generated and cached when files are moved (after move completes)
    • Eliminates generation delay on page load
  • ThrottledImage Optimization:
    • Shared IntersectionObserver across all image instances (was creating new observer per image)
    • Increased rootMargin from 50px to 400px for earlier preloading
    • Removed random delays that caused staggered loading
    • Memoized with React.memo()
  • Frontend Modernization:
    • API retry logic with exponential backoff, jitter, configurable retry count
    • Retries on 408, 429, 500, 502, 503, 504 status codes
    • useTransition for non-blocking filter updates
    • useDeferredValue for smooth search input rendering
  • New Shared Components:
    • FilterBar.tsx - Unified filter component for all pages
    • MediaGrid.tsx - Memoized thumbnail grid with selection support
    • SuspenseBoundary.tsx - Suspense wrapper with skeleton loading states
  • Browser Cache Headers:
    • Extended from 1 hour to 24 hours with immutable flag
    • Reduces redundant thumbnail requests
  • Files Modified:
    • web/backend/routers/review.py - SQL JOIN fix
    • web/backend/routers/media.py - LRU cache, extended cache headers
    • web/frontend/src/lib/api.ts - Retry logic
    • web/frontend/src/components/ThrottledImage.tsx - Shared observer, prefetch
    • web/frontend/src/components/FilterBar.tsx - New file
    • web/frontend/src/components/MediaGrid.tsx - New file
    • web/frontend/src/components/SuspenseBoundary.tsx - New file
    • web/frontend/src/hooks/useMediaFiltering.ts - useTransition, useDeferredValue
    • modules/move_module.py - Thumbnail pre-generation

11.26.2

  • TMDb Appearance Prediction Fix:
    • Fixed false positive appearance predictions where having ANY past credit for a show caused predictions for all future episodes
    • Added get_episode_credits() method to fetch credits for specific TV episodes
    • Added is_person_in_episode() method to verify a person is in episode cast/guest_stars
    • Updated find_upcoming_tv_appearances() to verify guest/host/cameo credits against actual episode credits
    • Credits with ≤5 episodes or credit types 'guest', 'host', 'cameo', 'self' now require episode-level verification
  • Data Cleanup:
    • Removed 10 incorrect upcoming appearances (false predictions for talk shows)
    • Resynced TMDb data with corrected prediction logic
  • Files Modified:
    • modules/tmdb_client.py - Added episode credits verification functions and logic

11.26.1

  • Security Hardening:
    • Fixed SQL injection risk in maintenance.py - added ALLOWED_CLEANUP_TABLES whitelist and _CLEANUP_QUERIES pre-built at module load time
    • Fixed SQL injection risk in media.py - replaced f-string column interpolation with explicit if/else branches for post_date/download_date
    • Added require_admin check to scheduler service restart endpoint (was only requiring authentication)
    • Added missing for_write=True flag on video.py DELETE operation
  • Bug Fixes:
    • Forum monitoring now uses configured auto_track_days from Configuration page instead of hardcoded 30 days
    • Forum monitoring log messages now display actual configured days value
    • Scheduler _check_monitored_threads now uses config value for cleanup and logging
  • Code Cleanup & Refactoring:
    • Subprocess wrappers refactored using factory pattern in base_subprocess_wrapper.py
    • Reduced wrapper code from ~450 lines to ~150 lines (67% reduction)
    • forum_db_adapter.py now uses _execute_with_retry() consistently instead of inline retry logic
    • Added build_media_filter_query() utility to core/utils.py for standardized filter building
    • Added build_platform_list_filter() utility to core/utils.py
    • Enhanced offset_paginated() response helper with include_timestamp and **extra_fields support
    • Added batch_operation_response() helper to core/responses.py for standardized batch operation responses
  • Files Modified:
    • web/backend/routers/maintenance.py - SQL injection fix with whitelist validation
    • web/backend/routers/media.py - SQL injection fix with explicit branches
    • web/backend/routers/scheduler.py - Admin check + forum monitoring config fix
    • web/backend/routers/video.py - for_write=True fix
    • web/backend/core/utils.py - New filter builder utilities
    • web/backend/core/responses.py - Enhanced response helpers
    • modules/forum_downloader.py - auto_track_days parameter added to download_thread
    • modules/forum_db_adapter.py - Refactored to use _execute_with_retry
    • modules/scheduler.py - Forum config auto_track_days usage
    • wrappers/base_subprocess_wrapper.py - Factory pattern added
    • wrappers/fastdl_subprocess_wrapper.py - Refactored to use factory
    • wrappers/imginn_subprocess_wrapper.py - Refactored to use factory
    • wrappers/toolzu_subprocess_wrapper.py - Refactored to use factory
    • wrappers/snapchat_subprocess_wrapper.py - Refactored to use factory
    • wrappers/forum_subprocess_wrapper.py - auto_track_days parameter passed

11.26.0

  • Snapchat Direct Scraper:
    • Complete replacement of StoryClon proxy-based module with direct Playwright scraper
    • Scrapes directly from snapchat.com using __NEXT_DATA__ JSON extraction
    • Supports both Stories and Spotlight content types
    • EXIF metadata (DateTimeOriginal, GPS if available) preserved on all downloads
    • Video stitching feature removed due to audio/video sync issues
    • Each highlight clip downloaded as individual file with proper metadata
    • Archived old snapchat_module.py to docs/archive/snapchat_module_storyclon.py
    • Updated unified_database.py scraper seed data
    • Updated subprocess wrapper to remove stitch_highlights parameter

11.19.4

  • Configuration Page Loading Fixes:
    • Manual Import, Monitoring, and Dependencies tabs now properly wait for all data before rendering
    • Fixed loading states to check all required queries (isLoading, isLoadingErrorMonitoring, config, editedConfig)
  • Service Health Monitoring Improvements:
    • Changed from time-based "Failure Window (hours)" to consecutive run-based counting
    • Simplified logic to use min_consecutive_failures (default: 2 runs)
    • Alert delay now correctly reads from error_monitoring.push_alert_delay_hours setting
  • Error Monitoring Cleanup:
    • Removed unnecessary scan_interval_hours field from UI and backend
    • Errors are logged immediately and checked hourly (hardcoded behavior)
  • Lightbox Enhancement:
    • Added optional hideFaceRecognition prop to EnhancedLightbox component
    • Face recognition info now hidden on Media page, visible on Review page

11.19.3

  • Face Recognition Display Fixes:
    • EnhancedLightbox now shows confidence percentage for non-matched items (format: "No match (X%)" in red)
    • Fixed SQL JOIN in review.py to use filename-based matching instead of exact path matching
    • Face scans logged with /opt/immich/md/ paths but review files at /opt/immich/review/ - now matches correctly
    • Fixed move_module.py UPDATE query to use original_intended_path when updating face scan paths
    • Was incorrectly using source (temp file path) which never matched the logged md path

11.19.2

  • Logging Refactoring:
    • Created modules/base_module.py with LoggingMixin class for consistent logging
    • Updated 13 modules to use LoggingMixin (instaloader, tiktok, imginn, fastdl, toolzu, coppermine, forum_downloader, download_manager, move_module, scheduler, snapchat, instagram_repost_detector)
    • Removed 12+ duplicate log() method implementations
    • Fixed instagram_repost_detector.py inverted logging priority (was callback first, then creating new logger on each call)
  • Hash Standardization:
    • Changed all MD5 usage to SHA256 for consistency (11 files updated)
    • Fixed potential duplicate detection failures between modules using different hash algorithms
    • Files: download_manager.py, forum_downloader.py, forum_db_adapter.py, universal_video_downloader.py, unified_database.py, universal_logger.py, manual_import.py, video.py, review.py, media.py, migrate_video_folders.py
    • No database migration needed - all existing hashes were already SHA256

11.19.1

  • Server-Side Search: Media, Downloads, Review pages now use server-side search filtering for better performance
  • Dependencies Fix: System Health page Dependencies section now works correctly (fixed API path)
  • Codebase Cleanup: Removed 4 unused files (auto_tagger.py, safe_query_builder.py, youtube_downloader.py, security.py)
  • TypeScript Fixes: Resolved WebSocket callback type errors, unused variable warnings

11.16.0

  • Semantic Versioning Overhaul: 246 versions renumbered for correct PATCH/MINOR/MAJOR classification
  • Search Functionality: Added search boxes to Downloads and Internet Discovery pages
  • Video Downloader Search: Now uses server-side filtering (searches all pages, not just current)
  • Thumbnail Standardization: All thumbnails now stored as JPEG, webp converted to jpg

11.13.1

  • Internet Discovery Rename: Celebrity Discovery renamed to Internet Discovery
  • Config Cleanup: Removed two unused configuration sections:
    • Search Presets blocked channels (setting was never applied to filtering)
    • Video Streaming native player (toggle was never implemented)

11.13.0

  • Face Recognition Path Fix: Review page now shows face confidence for all scanned files
    • Root cause: face_recognition_scans paths weren't updated when files moved to review
    • move_module.py now updates paths alongside perceptual hashes
    • Fixed 140 existing database records
  • Instagram Fallback Patterns: Added parse_with_fallbacks() function for cleaner pattern matching

11.12.0

  • Downloads Page Batch Operations: Full batch select/action functionality matching Media page
  • Instagram Stories Fix: Filename pattern corrected from dashes to underscores
  • Dead Code Cleanup: Removed ~40 lines of disabled face_recognition fallback code
  • Database Cleanup: Removed 5 empty/orphaned database files

11.11.0

  • Modern UI Styling:
    • Applied custom scrollbars to scrollable areas across all pages (Dashboard, Discovery, VideoDownloader, Logs, etc.)
    • Added animated number counters on stat cards (Dashboard, Analytics, Monitoring, Scheduler, RecycleBin, etc.)
    • Added thumbnail zoom effects on media grids (Media, Review, RecycleBin, Downloads, VideoDownloader)
    • Applied glass cards with gradient borders on info boxes (Scheduler, Platforms, Scrapers)
    • Smooth theme transition effects when switching light/dark mode
  • Codebase Cleanup:
    • Removed StyleDemo.tsx demo page and route from App.tsx
    • Removed 6 orphaned frontend components:
      • ForumConfig.tsx (never imported)
      • PlatformConfig.tsx (never imported)
      • TOTPSetupModal.tsx (never imported)
      • VideoLightbox.tsx (never imported)
      • VideoPlayerModal.tsx (never imported)
      • VirtualizedGrid.tsx (never imported)
    • Removed deprecated cleanup-old-logs.sh (replaced by Python version)
    • Removed test scripts: test_face_recognition_with_model.py, clear-phun-test-data.sh
    • Removed archive directory with 4 old maintenance scripts
    • Removed unused modules/instagram/ base class directory
  • Code Quality:
    • All Python files pass syntax checks (py_compile)
    • All TypeScript files pass type checks (tsc --noEmit)
    • Identified but retained future-facing API methods (parseFilename, clearMonitoringHistory, verifyDuoCallback)

7.0.18

  • Security Fixes:
    • Fixed hardcoded secure=False in 2FA cookie settings (twofa_routes.py) - now uses settings.SECURE_COOKIES
    • Added path traversal protection in forum_subprocess_wrapper.py using Path.name and .resolve().relative_to()
  • Connection Leak Fixes:
    • Fixed DatabaseAdapter methods (add_to_queue, get_pending_downloads, update_queue_status) with try/finally
    • Fixed health.py database connection leak with try/finally
  • Thread Safety Improvements:
    • Added double-check locking to discovery_system.py singleton
    • Made SettingsManager thread-safe with RLock for writes
  • Exception Handling:
    • Replaced bare except: clauses with specific exception types in:
      • semantic_search.py (OSError for temp file cleanup)
      • activity_status.py (json.JSONDecodeError, TypeError, ValueError)
      • youtube_channel_monitor.py (ValueError, TypeError for date parsing, OSError for cleanup)
      • video.py (OSError for cache file cleanup)
  • Bug Fixes:
    • Fixed return type mismatch in coppermine_module.py:730 (return 0 → return ({}, 0))
    • Fixed Immich config KeyError risk using .get() with validation
    • Fixed React useEffect dependency arrays in Review.tsx
  • Signal Handling:
    • Added SIGTERM/SIGINT handlers to all wrapper scripts (fastdl, imginn, toolzu, snapchat) for graceful cleanup
  • TypeScript Improvements:
    • Added proper interfaces to api.ts: User, UserPreferences, AppConfig, PlatformConfig, SettingValue
    • Added WebAuthn credential types for passkey authentication
    • Added WebSocketMessage and WebSocketCallback types
    • Replaced any types with specific types throughout
  • Cleanup:
    • Removed stale cookies_backup_20251201 folder
    • Removed redundant NEVER_RESTART_SERVICES.txt (info in CLAUDE.md)
    • Cleaned pycache directories outside venv

7.0.16

  • Video Width Metadata Capture:
    • Added max_width column to celebrity_discovered_videos and video_download_queue tables
    • YouTube channel monitor captures width alongside resolution from yt-dlp metadata
    • API endpoints return max_width for future aspect ratio features
    • Embedded videos keep 16:9 containers (YouTube/DM handle internal aspect ratio)

7.0.15

  • Forum Filename Date Prefix:
    • Forum download filenames now include date/time prefix: YYYYMMDD_HHMMSS_filename.ext
    • Prevents filename collisions when same generic filename (like '3.jpg') appears in multiple downloads
    • Date extracted from thread title using DateHandler.extract_date_from_text()
    • Falls back to forum post date if no date in title
    • Updated _get_image_filename() in forum_downloader.py to accept post_date parameter
  • Post Date Recording:
    • Forum db_adapter now passes post_date to unified database record_download()
    • Forum subprocess wrapper passes post_date when recording deferred downloads
    • Fixes Post Date showing wrong in lightbox (was falling back to file_inventory.created_date)
  • Proportional Video Sizing:
    • Video players now resize based on actual video aspect ratio
    • Portrait videos display tall/narrow instead of forced 16:9
    • Local videos use natural sizing: max-w-[90vw] max-h-[70vh] w-auto h-auto
    • Embedded videos (YouTube/Dailymotion) keep 16:9 container (iframes can't report ratio)
    • Updated: EnhancedLightbox, VideoLightbox, CelebrityDiscovery, DownloadQueue
  • GIF Handling:
    • GIF files now treated as images, not videos
    • Removed .gif from isVideo patterns, added to isImage patterns
    • Fixes thumbnail display and lightbox playback for GIFs
    • Updated: Dashboard, Downloads, Media, Review, RecycleBin, Discovery
  • Forum Notification Fix:
    • Notifications now only sent for files with face matches
    • Files moved to review queue (no face match) don't trigger notifications
    • Uses move_manager.moved_files instead of tracking all downloads
  • Cleanup:
    • Removed stale 0-byte database files from root: media-downloader.db, media_tracking.db

7.0.13

  • Dailymotion Discovery & Search Presets:
    • Added Dailymotion channel search as new source type in Search Presets
    • Searches Dailymotion API by celebrity name within specified channel
    • Fetches full metadata via yt-dlp after API discovery
    • Added Dailymotion platform filter on Celebrity Discovery page
    • discover_and_enrich.py has search_dailymotion() function using Dailymotion API
  • Dailymotion HLS Streaming:
    • Added HLS.js streaming support for Dailymotion videos in lightbox native player mode
    • Fixed Native player button not working (playerMode started as null, useEffect didn't trigger)
    • Platform-specific button labels now show "Dailymotion" or "YouTube" correctly
    • Dailymotion buttons use cyan color, YouTube buttons use red for visual distinction
    • Applied fixes to both DownloadQueue.tsx and CelebrityDiscovery.tsx
  • Centralized Timestamp Handling:
    • Fixed forum downloads having inconsistent EXIF timestamps (Immich showing different dates)
    • Root cause: exiftool only set DateTimeOriginal/CreateDate/ModifyDate, not MetadataDate
    • All 5 modules now use comprehensive flags: -AllDates, -MetadataDate, -HistoryWhen=, -FileModifyDate
    • MoveManager now calls DateHandler.update_file_timestamps() for centralized EXIF + filesystem updates
    • Forum downloads pass timestamp=post_date to MoveManager instead of manual os.utime
  • Dashboard Video Queue Card:
    • Added Video Queue Processing card showing download progress
    • Shows spinner/paused icon, current title, completed/failed/remaining counts
    • Clickable link to Download Queue page
  • Code Cleanup:
    • Removed unused _update_file_timestamps method from forum_downloader.py (~110 lines)
    • Removed unused file_timestamp variable
    • Added date pattern support for underscores in forum titles (e.g., 12_10_2025)
    • Removed empty media_downloader.db file from root directory

7.0.12

  • YouTube Scheduler Fix:
    • Removed two-level scheduling - scheduler now reads check_interval_hours directly from YouTube monitor settings
    • Previously scheduler ran every 1 hour, then monitor internally checked if due (redundant)
    • Now scheduler uses configured interval directly (e.g., 6 hours)
  • Search Presets Full Scan:
    • Added "Full Scan" button to Search Presets page
    • Runs discovery and enrichment for all enabled presets
    • Dashboard shows activity card with real-time progress tracking
    • Added activity manager integration to discover_and_enrich.py script
  • Instagram Duplicate Fix:
    • Fixed critical bug where files sent to recycle (as perceptual duplicates) weren't recorded to database
    • Caused re-downloading on subsequent runs (media_id not in downloads table)
    • Removed moved > 0 condition - now records pending downloads regardless of moved count
    • Applied fix to imginn, fastdl, and toolzu modules

7.0.11

  • Scheduler UI Improvements:
    • YouTube Monitor task now displays proper name instead of raw task ID
    • Added platform mappings for youtube_channel_monitor and youtube_monitor
    • Shows "Channel phrase matching" description instead of "Unknown"

7.0.10

  • Video Download Timezone Fix:
    • format_datetime_for_db() was using datetime.now() (local time) instead of UTC
    • Frontend expects UTC timestamps, causing 5-hour offset on Downloads page
    • Changed to datetime.utcnow() for consistency with rest of system
    • Fixed existing records in database to match UTC

7.0.9

  • YouTube Monitor Auto-Start Queue Fix:
    • Manual checks (Check All and single channel) now properly respect auto_start_queue setting
    • Changed from unauthenticated HTTP call to direct queue processor access
    • HTTP call was silently failing with 401 Unauthorized
  • Error Notification Context:
    • Scheduler now sets proper context (platform='system', content_type='error') for error notifications
    • Prevents error notifications from showing stale download metadata
  • Service Documentation:
    • Added complete service reference to CRITICAL_RULES.md
    • Documents active, disabled, and legacy services

7.0.8

  • YouTube Channel Monitor:
    • New feature allowing background monitoring of YouTube channels for videos matching phrases
    • Matching videos automatically added to download queue
    • Management UI in Configuration → Platforms with search and alphabetical sorting
    • Channel names converted to @handle format for consistency
  • Thumbnail Fix (webp → jpg):
    • YouTube's webp thumbnails not available for all videos (especially older ones)
    • Changed youtube_channel_monitor.py to prefer jpg thumbnails
    • Converted 279 existing webp URLs to jpg in database
  • Recycle Bin Metadata Restore:
    • Files restored from recycle bin now correctly restore platform/source metadata
    • Fixed restore_from_recycle_bin() in unified_database.py to update file_inventory
  • Error Notification Context Fix:
    • Error notifications were inheriting stale context from previous download notifications
    • Fixed by clearing _current_notification_context after recording in pushover_notifier.py
  • API Testing Helper Scripts:
    • /scripts/get-api-token.sh - Gets token for claude_test account
    • /scripts/api-call.sh - Makes authenticated API calls using saved token

7.0.5

  • Notifications 503 Fix:
    • Implemented IntersectionObserver-based lazy loading for thumbnails
    • Thumbnails only load when scrolling into view (100px preload margin)
    • Failed thumbnails use inline base64 SVG placeholder instead of network request
    • Eliminates rate limit errors from loading dozens of thumbnails simultaneously
  • Silent Exception Handling:
    • Replaced bare except: with specific exception types throughout codebase
    • Added debug logging to previously silent exception handlers
    • Files fixed: api.py, celebrity.py, scrapers.py, forum_downloader.py
    • Exception types now explicit: ValueError, TypeError, json.JSONDecodeError
  • Proxy Rate Limits:
    • Created new media_downloader nginx rate zone (100r/s vs general 10r/s)
    • Increased burst limit to 500 for md.lic.ad
    • Increased connection limit to 200

7.0.3

  • Celebrity Discovery UX:
    • Default filters: Eva Longoria, Highest Resolution, Not Queued, Unwatched
    • Added Watched/Unwatched bulk action buttons for selected videos
    • Removed redundant Manage Presets dropdown from filter bar
  • Forum Date Fixes:
    • Forum downloads now use correct post date from thread title
    • DateHandler extracts date from thread title before MoveManager processes files
    • File timestamps properly set via os.utime() matching imginn approach
  • Queue Settings Persistence:
    • Fixed backend model missing stop_on_cookie_error, send_cookie_notification, auto_start_on_restart
    • Auto-start on restart now reads from correct settings key (video_queue)
    • All queue settings now properly save and load
  • Bug Fixes:
    • Fixed file-not-found error for video titles with special characters (brackets) using glob.escape()
    • Notification attachments now open for files not in file_inventory (os.path.exists fallback)

7.0.2

  • Anti-Bot Protection:
    • Configurable User-Agent selection (Edge, Chrome, Firefox, Safari, Custom)
    • Rate limiting, throttle detection, sleep between requests
    • Settings UI in Configuration → Downloads
    • User-Agent matches cookie export browser to prevent session invalidation
  • Download Queue Improvements:
    • Sort toggle: Download Order vs Recently Added
    • Queue displays items in actual download order (priority ASC, oldest first)
    • Pending filter now includes downloading items
    • Defaults to Pending filter on load
  • Reliability:
    • Stuck downloads automatically reset to pending on API restart
    • Cookie expiration detection stops queue and sends push notification

7.0.1

  • Download Selected Feature:
    • New "Download Selected" button downloads only checked items
    • Backend endpoint /processor/start-selected with ID filtering
    • Supports pause/resume for selected batch
  • Configurable Download Paths:
    • Base directory now configurable in Configuration → Downloads
    • Stored in settings table, read by universal_video_downloader
    • Platform subdirectories created automatically
  • Bug Fixes:
    • Mobile menu now respects iOS safe-area-inset-top
    • Fixed route ordering conflict in video_queue.py (/settings vs /{queue_id})
    • Fixed NOT NULL constraint on settings.value_type column

7.0.0 (Major Release)

  • Celebrity Discovery System:
    • Automatic YouTube video discovery from celebrity channels
    • Bulk add to download queue with metadata
    • Filtering by celebrity, status, date range
  • iOS Mobile-Friendly UI:
    • Touch-friendly 44px minimum button sizes
    • Bottom-sheet modals for mobile
    • Safe area support for notch devices
    • Responsive layouts across all pages
  • Native Video Player:
    • Privacy-focused yt-dlp streaming
    • Quality selection (360p-1080p)
    • Toggle between YouTube embed and Native
    • 5-hour URL caching
  • Unified Download Queue:
    • Bulk operations (select all, retry, pause, delete)
    • Configurable delay between downloads
    • Cookie support from Scrapers settings
    • Rate limiting protection
  • Memory Optimizations:
    • Lazy loading for face_recognition module
    • Reduced idle memory from 8.5GB to ~100MB

6.52.65

  • Installer Overhaul:
    • Complete rewrite of install.sh to include ALL services (461 lines)
    • Now creates 7 systemd services:
      • media-downloader - Main scheduler service
      • media-downloader-api - Web API (port 8000)
      • media-downloader-frontend - Web UI (port 5173)
      • xvfb-media-downloader - Virtual display for browser automation
      • media-cache-builder - Thumbnail cache builder
      • media-embedding-generator - CLIP embedding generator
      • media-celebrity-enrichment - Celebrity metadata enrichment
    • Creates 3 systemd timers for daily background tasks (3-4 AM)
    • Added FlareSolverr Docker container installation prompt
    • Creates command-line wrapper at /usr/local/bin/media-downloader
    • Comprehensive backup creation during uninstall
  • Uninstaller Updated:
    • Matches install.sh with all 7 services and 3 timers
    • Proper service stop/disable/remove sequence
  • Requirements.txt Updated:
    • Added gallery-dl>=1.30.0 for forum/Erome downloads
    • Added opencv-python-headless>=4.8.0 for image processing
    • Added torch>=2.0.0 and torchvision>=0.15.0 for CLIP
    • Added PyJWT>=2.8.0 for token handling
    • Added cryptography>=41.0.0 for security
    • Added webauthn>=2.0.0 for passkey support
    • Fixed bcrypt version constraint (<5.0 for passlib compatibility)

6.52.64

  • Video Streaming Settings UI:
    • Added comprehensive settings section in Configuration → Downloads tab
    • Default Player setting (Native vs YouTube embed)
    • Default Quality setting (360p-1080p or Best)
    • Cache Duration setting (1-5 hours)
    • Proxy Mode toggle for privacy
  • Bug Fixes:
    • Fixed race condition where quality/player settings weren't honored on load
    • Fixed max_resolution not being passed when adding videos to Download Queue
    • Backfilled resolution data for existing queue items
    • Corrected resolution values storing width instead of height

6.52.63

  • Native Video Player:
    • Privacy-focused alternative to YouTube embed using yt-dlp streaming
    • Backend /api/video/stream-url endpoint with quality selection
    • Custom video controls (play, pause, seek, mute, fullscreen)
    • Audio sync for separate video/audio streams
    • URL caching (5 hours before expiry)

6.52.60

  • Critical Memory Leak Fixes:
    • Fixed OOM kills occurring every 10-15 minutes during Instagram downloads
    • Scheduler was growing from ~800MB to 8GB+ due to unreleased image memory
    • Root cause: PIL Image.open() and cv2.imread() not being properly closed
  • Files Fixed:
    • fastdl_module.py - EXIF update function now uses context managers
    • instagram_repost_detector.py - Hash calculation and OCR functions fixed
    • semantic_search.py - Image embedding generation uses context managers
    • auto_tagger.py - Image embedding generation uses context managers
    • face_recognition_module.py - Thumbnail generation and re-training loops fixed
    • media-downloader.py - Added gc.collect() after each user in FastDL loop
  • Error Banner Fix:
    • Dropdown now shows only errors since last dashboard visit (matches displayed count)
    • Added sinceVisit parameter to getRecentErrors() API call

6.52.59

  • UI Consistency Update:
    • Standardized all page headers with consistent icon + text-3xl font-bold design pattern
    • Updated navigation menu icons in App.tsx to match corresponding page header icons
    • Manual Import page layout fixed (changed from p-6 to space-y-6 for consistent spacing)
    • Celebrity Discovery page simplified with single "Manage Presets" button linking to Search Presets
    • Removed unused searchAllMutation from CelebrityDiscovery.tsx
  • Navigation Icon Updates:
    • Dashboard: LayoutDashboard
    • Downloads: FolderDown
    • Media: GalleryHorizontalEnd
    • Review: ClipboardList
    • Video Downloader: Clapperboard
    • Manual Import: FolderInput
    • Face Recognition: ScanFace
    • Scheduler: CalendarClock
    • Analytics: BarChart3
    • System Health: HeartPulse
    • Monitoring: Gauge
    • Notifications: BellRing
    • Platforms: Layers
    • Scrapers: Bot
    • Logs: ScrollText
    • Configuration: Settings

6.52.58

  • Download Queue Processor:
    • New queue processor with Start/Pause/Stop controls for batch downloads
    • Downloads pending videos sequentially with real-time progress updates
    • Status banner shows current download, completed/failed counts, remaining items
    • Fixed Celebrity Discovery "Add to Queue" inserting into wrong table
    • Fixed Download Queue page calling wrong API endpoints (/queue → /video-queue)
  • Celebrity Discovery Enhancements:
    • Added "Not Queued" status filter (shows new/ignored/watched videos)
    • Resolution badge on thumbnails (e.g., "1080p", "720p")

6.52.57

  • Recycle Bin Filters:
    • Platform and Source filter dropdowns added to Recycle Bin page
    • New /api/recycle/filters endpoint for dynamic filter options
    • Fixed 'fastdl' platform entries corrected to 'instagram'

6.52.55

  • Error Banner Fixes:
    • Removed redundant process_and_store_errors() that was re-adding dismissed errors
    • Errors now only recorded in real-time by universal_logger.py
    • Fixed timestamp format consistency (all ISO format with T separator)
    • Dismissed errors no longer reappear unless they actually recur
  • Instagram Video Downloader Fixes:
    • Fixed nest_asyncio / uvloop conflict preventing downloads
    • Username now extracted from post page (no more "unknown" filenames)
    • Activity status updates to show real username once extracted
  • API Memory Optimization (6.52.54):
    • FaceRecognitionModule cached as singleton (saves ~400MB per request)
    • API memory reduced from 2.1GB to ~113MB

6.52.47

  • Dashboard Error Display Fixed:
    • Errors now show ALL unviewed items, not just since last visit
    • Error count reflects total unviewed regardless of visit time
  • Error Monitoring UI:
    • Settings added to Configuration page Monitoring tab
    • Configurable push alert delay, dashboard banner, retention

6.52.44-6.52.46

  • MAJOR: API Modularization Complete:
    • api.py reduced from 10,465 to 828 lines (92% reduction)
    • 17 modular routers created in web/backend/routers/
    • Shared state via AppState singleton in core/dependencies.py
    • All 170+ endpoints migrated and tested
  • Error monitoring settings API endpoints added
  • Timezone bug fixed in error alert 24-hour check

6.52.49

  • React.lazy() code splitting for all 21 pages (40-50% faster initial load)
  • Reduced Dashboard polling (stats 30s→60s, activity 2s→5s)
  • React Query staleTime/gcTime caching (30s/5min)
  • Memoized filteredMedia with useMemo() in Media.tsx
  • Cache-Control headers on thumbnails/previews (1 hour)
  • Initial bundle reduced from ~600KB to ~50KB app shell

6.52.48

  • Real-time error notifications via WebSocket (error_alert broadcast)
  • ErrorBanner WebSocket listener for instant error updates
  • Error polling reduced from 60s to 15s
  • Scheduler OOM crashes fixed (face recognition disabled in subprocesses)
  • Missing Tuple import in settings_manager.py fixed
  • Added /api/downloads/filters route alias
  • Memory optimization with gc.collect() calls

6.52.53

  • Critical OOM Memory Fix:
    • Fixed memory leak in face recognition causing 9GB+ memory usage and OOM kills
    • Added explicit del and gc.collect() in detect_faces_insightface()
    • Added memory cleanup in detect_faces() fallback method
    • Added gc.collect() after video and image processing in move_module.py
    • Memory now stable at ~743MB (was growing to 9GB+ before fix)
    • Peak memory reduced from 9GB+ to 1.2GB

11.22.1

  • Podcast Appearances Fixes & Enhancements:
    • Fixed episode list endpoint to use query parameters instead of path parameters (handles special characters like ó in Gomez-Rejón)
    • Fixed episode count calculation for podcast appearances (was hardcoded to 1, preventing full episode list display)
    • Modal episode list now excludes current episode to prevent duplication
    • Audio player now preloads metadata to show episode duration before playback starts
    • Mobile app now preserves page state when backgrounding/resuming (changed MemoryRouter to BrowserRouter)
    • RSS feed backfill automatically runs after Podchaser sync for Omny.fm-hosted podcasts (29/32 episodes now have audio URLs)

11.28.0

  • Dashboard New Items Cards:
    • Created dashboard.py router - new /api/dashboard/recent-items endpoint
    • Created RecentItemsCard.tsx component - reusable card with thumbnail grid
    • Three cards: "New in Media", "New in Review", "New in Internet Discovery"
    • Dismissible with localStorage persistence - reappears when new items arrive
    • Hover overlay with circular action buttons matching source page functionality
    • EnhancedLightbox integration for media/review, video preview modal for discovery
    • Session-based tracking removes actioned items from display
    • YouTube link conditional on platform (not shown for Easynews)

6.52.52

  • Instagram Cross-Module Duplicate Detection:
    • Created instagram_utils.py - centralized utility module
    • extract_instagram_media_id() - extracts 18-digit media IDs from filenames
    • record_instagram_download() - centralized database recording with normalized IDs
    • is_instagram_downloaded() - cross-module duplicate check
    • imginn module refactored to use unified_db directly (removed adapter dependency)
    • Removed unused adapters from unified_database.py (~200 lines)
    • Fixed regex pattern for media ID extraction (underscore boundaries)

6.52.47

  • Dashboard now shows ALL unviewed errors (not just since last visit)
  • Error count reflects total unviewed regardless of visit time

6.52.38-6.52.43

  • Phase 1 Refactoring Complete:
    • Custom exception classes (20+ types)
    • Unified configuration manager
    • Standardized response formats
    • ISO 8601 date utilities
    • Router infrastructure created
    • Base Instagram downloader class
  • Technical debt items resolved: TD-002, TD-003, TD-008, TD-015, TD-016
  • All 16 routers enabled and registered

6.52.37

  • Lightbox dimensions fetching implemented
  • Real-time error alerts via WebSocket
  • Thread safety improvements in Playwright
  • Database indexes added for performance

6.52.62

  • Fixed recycle bin restore error (cannot access local variable 'get_app_state')

6.52.61

  • Fixed additional memory leak in face_recognition_module.py add_reference_face
  • Scheduler now stable at ~2GB for 3+ hours (previously crashed within 10-15 minutes)

6.52.60

  • Fixed critical memory leaks causing OOM kills during Instagram downloads
  • Fixed Image.open() leaks in fastdl, repost detector, semantic search, auto_tagger
  • Fixed cv2.imread() leaks in face_recognition_module thumbnail and re-training
  • Added gc.collect() after each user in FastDL download loop
  • Error banner dropdown now shows only errors since last dashboard visit

12.11.0

  • Config Page Redesign: Replaced sidebar + pill tab navigation with horizontal underline tabs matching Configuration.tsx and Paid Content Settings.tsx. Removed sticky header with ArrowLeft back button, replaced with standard text-2xl sm:text-3xl font-bold h1 heading with text-slate-500 icon and text-slate-600 dark:text-slate-400 subtitle. Page root element changed from min-h-screen bg-background > container mx-auto px-4 py-6 to simple space-y-6 div, inheriting padding from parent <main> layout (fixes breadcrumb spacing). Content wrapper changed from max-w-3xl to full-width. Removed ArrowLeft import from lucide-react.
  • Multi-URL Import (Frontend): Added Link icon and urls textarea state to PrivateGalleryUploadModal. URL textarea shown below drop zone when no files selected, with placeholder listing supported hosts. New importUrlsMutation calls api.privateGallery.importUrls() and builds ThumbnailProgressItem[] from URL basenames for progress tracking. handleSubmit now routes: files → upload, URLs → import, existing media → create post. Footer shows "X URLs entered" when URLs present. Submit button shows "Import" with Link icon for URL mode.
  • Multi-URL Import (Backend): Fixed ImportUrlRequest model: tag_ids changed from Field(..., min_items=1) to List[int] = []. Added POST /import-urls endpoint with post creation, job tracking, and background task launch. Added _import_urls_to_gallery_background(): iterates URLs, tries FileHostDownloader.detect_host() first for Bunkr/Pixeldrain/Gofile/Cyberdrop (via asyncio.run()), falls back to direct requests.get() with content-type validation. Each downloaded file: hash → duplicate check → _get_file_info() → date extraction → _generate_thumbnail() → encrypt file + thumb → DB insert with source_type='import' and encrypted source URL. Cleans up empty post if all files fail.
  • API Client: Added importUrls method to privateGallery namespace in api.ts. Updated getJobStatus operation type union to include 'import'.
  • Files Modified: Config.tsx, PrivateGalleryUploadModal.tsx, api.ts, private_gallery.py

12.10.0

  • Slideshow Mode: Added play/pause button, configurable interval selector (3s/5s/8s/10s), and Space keyboard shortcut to both PrivateGalleryBundleLightbox and BundleLightbox. Images auto-advance on timer; videos play to completion via ended event before advancing. Slideshow collapses sidebar and hides metadata panel on start. Manual navigation resets the timer.
  • Shuffle Mode: Fisher-Yates shuffle with cyan toggle indicator. Keeps current item at front of shuffled sequence. Shuffle order rebuilt when toggling on. Position indicator shows "(shuffled)" suffix.
  • Context-Aware Slideshow Button: Gallery view slideshows all filtered media across all posts (flat attachment list with attachmentPostMap for post context). Feed/Social view slideshows only the currently selected post's attachments. Button hidden when 0 or 1 media items.
  • Server-Side total_media: Added total_media count to Private Gallery API (private_gallery.py - counts media items across filtered posts with file_type support) and Paid Content API (paid_content.py + db_adapter.py get_media_count() method). Frontend uses this for accurate lightbox position indicators instead of loaded page count.
  • Filter Bar Labels: Shows "X items" with total_media count in gallery view, "X posts" with total count in feed/social view, for both Private Gallery and Paid Content pages.
  • Gallery Item Ordering Fix: Created galleryLightboxItems useMemo in Private Gallery that follows the same media_date sort order as the gallery grid, fixing position indicator mismatch.
  • Paid Content Gallery Refresh: Added scroll-to-top on filter change (matching Private Gallery). Added data-change watcher effect that auto-loads more pages when gallery content doesn't fill the scroll container after filter change.
  • Infinite Scroll Integration: Lightbox preloads next page when within 5 items of end. Slideshow calls onLoadMore instead of stopping when more pages available. totalCount/hasMore/onLoadMore props added to both lightbox components.
  • Slideshow Controls Visibility: Play/pause, shuffle button, and interval selector all hidden when (totalCount || items.length) <= 1. Position indicator already hid for single items.
  • Files Modified: PrivateGalleryBundleLightbox.tsx (+slideshow/shuffle/infinite scroll), BundleLightbox.tsx (+same), Gallery.tsx (+slideshow button, galleryLightboxItems, totalMediaItems), Feed.tsx (+slideshow button, flat list builder, scroll-to-top, auto-load), api.ts (+total_media type), private_gallery.py (+total_media query), paid_content.py (+total_media response), db_adapter.py (+get_media_count)

12.9.1

  • Reliable 4K HLS Downloads: Replaced ffmpeg HLS networking (stalled at ~1.2GB) with direct segment-by-segment download using aiohttp. Downloads 5 segments concurrently with 120s timeout and 3 retries per segment. ffmpeg only used for local concatenation/remux.
  • Duration Fix: Added -t flag to ffmpeg remux to trim to exact playlist duration, eliminating ~7s padding accumulation from HLS segment boundaries.
  • Quality Recheck Fix: download_pending_for_creator() was breaking because creator_id wasn't registered in active_paid_content_syncs. Fixed by registering before download call.
  • Quality Recheck Upgrade Flow: Changed from in-place URL swap to delete old post/attachment + trigger full Fansly re-sync for clean 4K downloads.
  • Skipped Attachment Fix: upsert_attachment() now handles 'skipped' status like 'failed' - resets to 'pending' with fresh URL on sync.
  • Master Playlist Resolution: Added _resolve_best_hls_variant() to fansly_direct_client.py to select highest resolution variant from HLS master playlists upstream in get_posts().
  • Remux Progress: Download queue shows remux percentage during ffmpeg segment concatenation.
  • Dependency: Added aiohttp>=3.9.0 to requirements.txt (10 modules depend on it).
  • Files Modified: scraper.py (segment download + remux), fansly_direct_client.py (master playlist resolution), db_adapter.py (skipped status), paid_content.py (recheck flow), requirements.txt (aiohttp)

12.9.0

  • ThumbnailProgressModal: New shared component replacing text-based progress displays with visual thumbnail-grid progress tracking. Shows per-file status overlays (pending/processing/success/error/duplicate) with progress bar, status chips, and collapsible error details. Used across Copy to Gallery, Upload, and Manual Import.
  • Async Job Tracking: Private Gallery copy and upload operations now run asynchronously via FastAPI BackgroundTasks. In-memory job dict with threading Lock (mirrors manual_import.py pattern). New GET /private-gallery/job-status/{job_id} endpoint with dual auth.
  • Async Copy: Extracted per-file copy loop into _copy_to_gallery_background(). Endpoint returns {job_id, post_id, status, total_files} immediately. Frontend polls at 500ms for per-file updates.
  • Async Upload: Files saved to temp before responding (UploadFile invalidated after response). Extracted processing loop into _upload_to_gallery_background(). Two-phase frontend: XHR upload progress, then ThumbnailProgressModal for server-side processing.
  • Instagram Filename Patterns: Added 3 new patterns to INSTAGRAM_PATTERNS for browser extension formats: {username}-video-{id}-{YYYYMMDD}_{HHMMSS}_{description}, {username}-photo-{id}-..., {username}-{id}-... (order matters - specific patterns before generic).
  • Video Thumbnail Fix: generate_video_thumbnail() now tries seek at 1s then falls back to 0s for short videos. Moved -ss before -i for faster input seeking.
  • ALLOWED_PATHS Fix: Added /opt/media-downloader/temp/manual_import to allow thumbnail generation for temp files during manual import.
  • ThrottledImage Priority: Added priority prop to bypass IntersectionObserver in nested scroll containers (modal grids).
  • Files Modified: private_gallery.py (+1,896 lines for job tracking and async), ThumbnailProgressModal.tsx (new, 245 lines), CopyToGalleryModal.tsx (polling integration), PrivateGalleryUploadModal.tsx (two-phase progress), ManualImport.tsx (swapped BatchProgressModal), api.ts (getJobStatus, updated return types), filename_parser.py (+3 patterns), core/utils.py (video thumbnail fix + ALLOWED_PATHS)

12.8.0

  • Tag sorting by usage count (most-used first) across paid content and private gallery
  • Copy to Private Gallery from Recycle Bin with original filename preservation
  • Redesigned CopyToGalleryModal: optional tags, TagSearchSelector, thumbnail grid, person default tags
  • Multi-page selection persistence for copy operations
  • EXIF orientation fix for private gallery thumbnails (ImageOps.exif_transpose)
  • Double-submission prevention with ref-based guard in copy modal
  • Nginx proxy timeout fix: dedicated /api/private-gallery/ location with 1h timeout
  • Tag migration fix: syncs ALL paid content tags, not just used ones
  • Feature-gated Copy to Private button (respects Features Management)
  • Fixed Gallery.tsx React Query cache pollution (paidContent.getTags → privateGallery.getTags)

6.52.32

  • N+1 query pattern fixed
  • Batch face lookup implemented
  • Thread-safe WebSocket broadcasting

6.52.29-6.52.31

  • Dimensions captured on file move
  • Recycle bin dimensions support
  • Various lightbox fixes

Appendix: Quick Reference

File Locations

Component Path
Main CLI /opt/media-downloader/media-downloader.py
API Main /opt/media-downloader/web/backend/api.py
API Routers /opt/media-downloader/web/backend/routers/
API Core /opt/media-downloader/web/backend/core/
Frontend /opt/media-downloader/web/frontend/src/
Modules /opt/media-downloader/modules/
Database PostgreSQL media_downloader (via pg_adapter)
Logs /opt/media-downloader/logs/
Config /opt/media-downloader/.env

Key Commands

# Check version
cat /opt/media-downloader/VERSION

# View recent logs
tail -100 /opt/media-downloader/logs/$(date +%Y-%m-%d).log

# Database stats
PGPASSWORD=... psql -h localhost -U media_downloader -d media_downloader -c "SELECT COUNT(*) FROM downloads;"

# Service status
systemctl status media-downloader

# Build frontend
cd /opt/media-downloader/web/frontend && npm run build

# Check API health
curl -s http://localhost:8000/api/health | jq

Document Maintainer: Code Review System Review Schedule: Update after each major version release

Reference in New Issue View Git Blame Copy Permalink