Add Face Reference
setAddPersonName(e.target.value)} onKeyDown={(e) => { if (e.key === 'Enter') confirmAddReference() }} placeholder="Enter person name..." autoFocus />
{platformTabs.map((tab) => (
))}
{/* TikTok settings */}
)}
{/* Instagram */}
{activePlatform === 'instagram' && (
{/* Instagram settings */}
)}
// ... 6 more platforms
```
---
### UI/UX Improvements
**Before**:
- All 8 platforms displayed on one long scrolling page
- Difficult to find specific platform settings
- Visual clutter with all settings visible simultaneously
**After**:
- Clean tab interface at top of Platforms section
- One click to switch between platforms
- Only selected platform's settings are displayed
- Color-coded icons make platforms instantly recognizable
- Reduced cognitive load and improved focus
---
### Impact
- ✓ **Cleaner Configuration Page**: Reduced visual clutter by showing only one platform at a time
- ✓ **Improved Navigation**: Users can quickly switch between platforms with one click
- ✓ **Better Organization**: Platform-specific settings are now isolated and easier to find
- ✓ **Enhanced UX**: Color-coded icons improve platform recognition
- ✓ **Responsive Design**: Tab bar adapts to screen size automatically
---
### Migration
- **Automatic**: No migration required
- **No database changes**: All changes are frontend-only
- **Settings preserved**: All platform configurations remain intact
- **Default tab**: Opens to TikTok tab by default
---
## [6.38.5] - 2025-11-22
### 🔧 UI Bug Fixes: Stop Button, Forum Deduplication, Video Lightbox
This patch release fixes three UI/UX bugs: Dashboard stop button now properly stops the scheduler service, forum thread monitoring filters out duplicate variants, and Downloads page lightbox correctly displays video thumbnails.
---
### Bug Fixes
#### Dashboard Stop Button
- **Problem**: Stop button showed "No active task running" error when clicked
- **Behavior**: Only stopped current download instead of scheduler service
- **User Expectation**: Should immediately stop the entire scheduler service
- **Fixed**: Changed to call `api.stopSchedulerService()` instead of `api.stopCurrentActivity()`
**Changes** (`web/frontend/src/pages/Dashboard.tsx`):
- Line 298-314: Modified `handleStopActivity()` function
- Now calls `stopSchedulerService()` endpoint (`systemctl stop media-downloader.service`)
- Updated success message: "Scheduler Stopped" instead of "Download Stopped"
- Updated error messages to reflect scheduler service control
---
#### Forum Thread Duplication
- **Problem**: Same event appeared multiple times in thread monitoring logs
- **Example**: "Eva Longoria Variety 2025" appeared 4 times (base thread + ADDS + X13 + X25 variants)
- **Root Cause**: Forums create variant threads for same event (original, ADDS, X\d+ numbered posts)
- **Impact**: Redundant downloads and processing of duplicate content
- **Fixed**: Smart filtering to skip variant threads when base thread exists
**Changes** (`modules/scheduler.py`):
- Line 1046-1053: Added `GROUP BY thread_url` with `MAX(thread_id)` for SQL-level deduplication
- Line 1085-1099: Build map of base thread URLs (normalize by removing `-adds`, `-x\d+` suffixes)
- Line 1120-1134: Filter variant threads if base thread exists in monitoring list
- Logs: `"Skipping variant thread (base exists): [title]..."` for filtered duplicates
**Detection Logic**:
```python
# Normalize URL to detect variants
base_url = re.sub(r'-(adds?|x\d+)(?=\.|/|$)', '', thread_url, flags=re.IGNORECASE)
```
**Examples of Filtered Threads**:
- Base: `...2025.396680/` ✓ Checked
- Variant: `...2025-adds.396680/` ✗ Skipped
- Variant: `...2025-x13.396679/` ✗ Skipped
---
#### Downloads Page Video Lightbox
- **Problem**: Clicking video files showed "Failed to load image" error in lightbox
- **Root Cause**: Lightbox tried to access `item.media_type` field that doesn't exist on Download objects
- **Behavior**: Thumbnail API called with wrong media_type, resulting in 404/error
- **Fixed**: Calculate media_type from filename extension before calling thumbnail API
**Changes** (`web/frontend/src/pages/Downloads.tsx`):
- Line 556: Added `.m4v` extension to video detection regex
- Line 672-675: Fixed `getThumbnailUrl` callback to determine media_type from filename
- Line 676: Fixed `isVideo` callback to check filename extension directly
- Supports: `.mp4`, `.mov`, `.webm`, `.avi`, `.mkv`, `.flv`, `.m4v`
**Before** (broken):
```typescript
getThumbnailUrl={(item) => api.getMediaThumbnailUrl(item.file_path, item.media_type)}
// item.media_type is undefined on Download interface
```
**After** (working):
```typescript
getThumbnailUrl={(item) => {
const isVideo = item.filename?.match(/\.(mp4|mov|webm|avi|mkv|flv|m4v)$/i)
return api.getMediaThumbnailUrl(item.file_path, isVideo ? 'video' : 'image')
}}
```
---
### Technical Changes
**Frontend** (`web/frontend/src/pages/Dashboard.tsx`):
```typescript
// Before:
await api.stopCurrentActivity()
notificationManager.success('Download Stopped', 'The current download has been stopped successfully')
// After:
await api.stopSchedulerService()
notificationManager.success('Scheduler Stopped', 'The scheduler service has been stopped successfully')
```
**Backend Scheduler** (`modules/scheduler.py`):
```python
# SQL-level deduplication:
SELECT thread_id, thread_url, thread_title, last_post_date, last_checked
FROM forum_threads
WHERE forum_name = ? AND status = 'active'
AND (monitor_until IS NULL OR monitor_until > datetime('now'))
GROUP BY thread_url
HAVING thread_id = MAX(thread_id)
# Application-level variant filtering:
base_url = re.sub(r'-(adds?|x\d+)(?=\.|/|$)', '', thread_url, flags=re.IGNORECASE)
if thread_url != base_url and base_url in base_thread_urls:
if has_base_thread:
self.log(f"Skipping variant thread (base exists): {thread_title}...", "debug")
filtered_count += 1
continue
```
**Frontend Downloads** (`web/frontend/src/pages/Downloads.tsx`):
```typescript
// Grid view:
const isVideo = download.filename?.match(/\.(mp4|mov|webm|avi|mkv|flv|m4v)$/i)
const mediaType = isImage ? 'image' : isVideo ? 'video' : 'image'
// Lightbox:
getThumbnailUrl={(item) => {
const isVideo = item.filename?.match(/\.(mp4|mov|webm|avi|mkv|flv|m4v)$/i)
return api.getMediaThumbnailUrl(item.file_path, isVideo ? 'video' : 'image')
}}
isVideo={(item) => !!item.filename?.match(/\.(mp4|mov|webm|avi|mkv|flv|m4v)$/i)}
```
---
### Impact
**Dashboard Stop Button**:
- ✓ Stop button now properly stops scheduler service via `systemctl stop`
- ✓ No more "No active task running" errors
- ✓ Immediate scheduler shutdown when needed
- ✓ Clear feedback with "Scheduler Stopped" message
**Forum Thread Monitoring**:
- ✓ Eliminates duplicate thread checking for variants
- ✓ Reduces redundant downloads of same content
- ✓ Logs show filtered threads: `"Skipping variant thread (base exists)"`
- ✓ Monitoring logs cleaner and easier to understand
- ✓ Performance improvement: fewer threads to check
**Video Lightbox**:
- ✓ Videos play correctly in Downloads page lightbox
- ✓ Video thumbnails load properly
- ✓ All video formats supported: MP4, MOV, WebM, AVI, MKV, FLV, M4V
- ✓ Consistent behavior between Downloads and Media pages
- ✓ No more "Failed to load image" errors for videos
---
### Migration
- **Automatic**: No migration required
- **No database changes**: All fixes are code-level
- **Frontend**: Built and deployed automatically
- **Backend**: No service restart needed (scheduler changes apply on next run)
---
## [6.38.4] - 2025-11-22
### 🔧 Instagram Carousel Post Date Fix & Database Backfill
This patch release fixes the root cause of Instagram carousel posts only creating one database entry, and backfills all missing post_dates from historical data.
---
### Bug Fixes
#### ImgInn Carousel URL Hash Collision
- **Root Cause**: All carousel images used the same URL, causing `url_hash` UNIQUE constraint violations
- **Impact**: Only first carousel image got a downloads entry, others silently failed
- **Example**: akil.g.j post with 10 carousel images only had 1 database entry
- **Fixed**: Append `#{filename}` to URL for unique hash per carousel image (same fix as TikTok v6.38.0)
**Locations Fixed (imginn_module.py)**:
- Line 1181: Posts carousel (high-res download)
- Line 1289: Posts carousel (fallback download)
- Line 2222: Tagged carousel (high-res download)
- Line 2323: Tagged carousel (fallback download)
#### Downloads Page Sorting Issue
- **Problem**: Sorting by "Post Date Newest" hid 2077/2517 files (82.5%)
- **Cause**: Files with NULL post_date were pushed to end of results
- **Fixed**: Use `COALESCE(post_date, download_date)` to fall back to download_date
**API Endpoints Fixed (api.py)**:
- Line 1111: `/api/downloads` - Downloads list endpoint
- Line 1317: `/api/downloads/search` - Downloads search endpoint
- Line 3134: `/api/media` - Media gallery endpoint
- Line 3749: `/api/media/review` - Review queue endpoint
#### Non-Deterministic Sorting
- **Problem**: Files with duplicate downloads entries sorted randomly
- **Cause**: `LIMIT 1` without `ORDER BY` picked random row
- **Fixed**: Use `MAX(post_date)` for deterministic sorting
---
### Database Backfill
Created utility scripts to backfill historical data:
#### `/opt/media-downloader/utilities/backfill_instagram_post_dates.py`
- Finds file_inventory entries without corresponding downloads entries
- Extracts post_date from filename format: `username_YYYYMMDD_HHMMSS_...`
- Creates missing downloads entries with proper post_dates
- **Result**: Created 382 new entries (284 final + 98 review)
#### `/opt/media-downloader/utilities/update_null_post_dates.py`
- Finds downloads entries with NULL post_date
- Updates post_date by extracting from filename
- **Result**: Updated 22 entries with post_dates
---
### Technical Changes
**ImgInn Module** (`modules/imginn_module.py`):
```python
# Before (collision):
self.db.mark_downloaded(url=post_url, ...)
# After (unique):
unique_url = f"{post_url}#{filename}"
self.db.mark_downloaded(url=unique_url, ...)
```
**API Backend** (`web/backend/api.py`):
```sql
-- Before (NULL pushed to end):
SELECT (SELECT d.post_date FROM downloads d WHERE d.filename = fi.filename LIMIT 1) as post_date
-- After (fallback to download_date):
SELECT COALESCE((SELECT MAX(d.post_date) FROM downloads d WHERE d.filename = fi.filename), fi.created_date) as post_date
```
---
### Impact
- ✓ **100% Instagram files** now have post_dates (537/537 final, 125/125 review)
- ✓ **Downloads page** sorting shows all files correctly
- ✓ **Media & Review pages** sorting fixed
- ✓ **Future carousel downloads** will create proper entries for each image
- ✓ **No more hidden files** when sorting by post_date
---
### Migration
- **Automatic**: Backfill scripts included with update
- **382 entries created**: With post_dates extracted from filenames
- **22 NULL entries updated**: With post_dates from filenames
- **No user action required**: All fixes applied automatically
---
## [6.38.3] - 2025-11-22
### 🔧 Forum Module Dynamic URL Configuration
This patch release fixes the forum module to accept dynamic forum URLs from database configuration instead of using hardcoded URLs, enabling support for any forum without code changes.
---
### Bug Fixes
#### Forum URL Parameter Error
- **Fixed**: `ForumDownloader.monitor_search() got an unexpected keyword argument 'forum_url'` error
- **Added**: `forum_url` parameter to `monitor_search()` method (forum_downloader.py:1933)
- **Added**: `forum_url` parameter to `_perform_advanced_search()` method (forum_downloader.py:2837)
- **Removed**: Hardcoded forum URL dictionary (lines 2868-2872)
- **Updated**: `_perform_picturepub_search()` to accept and use forum_url parameter (forum_downloader.py:3020)
#### Hardcoded Forum URLs
- **Fixed**: Forums limited to hardcoded URLs (HQCelebCorner, PicturePub only)
- **Now**: Supports any forum URL configured in database settings
- **Impact**: Can add new forums via web UI without code changes
---
### Technical Changes
**`/opt/media-downloader/modules/forum_downloader.py`**:
- Line 1933: Added `forum_url: str = None` parameter to `monitor_search()` signature
- Line 1951: Updated docstring to document forum_url parameter
- Line 1998: Pass forum_url to `_perform_advanced_search()` call
- Line 2837: Added `forum_url: str = None` parameter to `_perform_advanced_search()` signature
- Line 2872-2874: Replaced hardcoded URL dictionary with validation check
- Line 2878: Pass forum_url to `_perform_picturepub_search()` call
- Line 3020: Added `forum_url: str` parameter to `_perform_picturepub_search()` signature
- Line 3031: Use `{forum_url}/search/` instead of hardcoded URL
**`/opt/media-downloader/wrappers/forum_subprocess_wrapper.py`**:
- Line 75: Pass `forum_url=forum_config.get('forum_url')` to monitor_search()
---
### Impact
- **Flexibility**: Forum module now accepts any forum URL from configuration
- **Scalability**: Can add unlimited forums via web UI without modifying code
- **Compatibility**: Works with any XenForo, vBulletin, phpBB, or other supported forum platform
- **Configuration**: Forum URLs sourced from database settings table
---
### Migration
- **Automatic**: No migration required
- **Database**: No schema changes
- **Configuration**: Forum configurations must include `forum_url` field in database settings
- **Action**: Restart service to apply changes
---
## [6.34.9] - 2025-11-16
### 📊 Real-time Progress Tracking for InsightFace Model Switching
This patch release adds real-time progress tracking when switching InsightFace models, providing transparency and user feedback during the re-training process.
---
### New Features
#### Real-time Progress Tracking
- **Progress Bar**: Visual progress bar shows percentage completion (0-100%)
- **Live Updates**: HTTP polling every 500ms for instant UI updates
- **Reference Tracking**: Shows current reference being processed (e.g., "Processing Eva Longoria (23/42)")
- **Elapsed Time**: Real-time timer displays seconds elapsed during re-training
- **Completion Summary**: Final status shows total time (e.g., "42/42 references in 39s")
#### HTTP Polling Architecture
- **Reliable Updates**: Switched from WebSocket to HTTP polling for long-running operations
- **New Endpoint**: GET /api/face/retrain-progress provides progress state
- **Increased Rate Limit**: 300 requests/minute (up from 120) to support polling
---
### Technical Changes
#### Backend Changes
**`/opt/media-downloader/modules/face_recognition_module.py`**:
- `retrain_all_references_with_model()` (line 835-957): Added `progress_callback` parameter
- Callback invoked after each reference is processed with current/total counts
- Provides person_name and success status for each iteration
**`/opt/media-downloader/web/backend/api.py`**:
- `POST /api/face/retrain-references` (line 4383-4449): Added progress state tracking
- Uses `asyncio.to_thread()` for async execution of blocking re-train operation
- Stores progress in `app_state.face_retrain_progress` for polling access
- `GET /api/face/retrain-progress` (line 4451-4468): New polling endpoint
- Rate limit increased from 120/minute to 300/minute (line 48)
#### Frontend Changes
**`/opt/media-downloader/web/frontend/src/lib/api.ts`**:
- `getRetrainProgress()` (line 813-821): New API method for progress polling
**`/opt/media-downloader/web/frontend/src/pages/Configuration.tsx`**:
- Progress UI with polling intervals (line 2896-3070)
- Two intervals: timer (500ms) and progress polling (500ms)
- Shows progress bar, current count, total count, person name
- Displays elapsed time throughout process
#### Utilities
**`/opt/media-downloader/scripts/test_face_recognition_with_model.py`** (new):
- Test script for comparing accuracy between InsightFace models
- Tests different tolerance values (0.15, 0.25, 0.35, 0.45)
- Moved from web/backend to scripts directory for proper organization
---
### Bug Fixes
- **Fixed**: Elapsed time showing 0s on completion
- Now captures final elapsed time from startTime when API completes
- Changed from using state variable to direct calculation
- **Fixed**: WebSocket disconnection during re-training
- Replaced WebSocket broadcasting with HTTP polling
- More reliable for long-running background operations
- **Fixed**: File organization
- Moved test_face_recognition.py from web/backend to scripts directory
---
### Impact
- **User Experience**: Users can now monitor re-training progress in real-time instead of seeing "Initializing..." for minutes
- **Transparency**: Progress shows exactly which reference is being processed, providing visibility into the operation
- **Accuracy**: Completion time now shows actual elapsed seconds
- **Reliability**: HTTP polling more stable than WebSockets for long operations lasting 30-60 seconds
- **Testing**: New test script allows comparing accuracy between different InsightFace models
---
### Migration
- No database migration required
- Frontend rebuild enables new progress tracking UI
- API service restart applies new rate limits
---
## [6.34.3] - 2025-11-16
### 🐛 Critical Bug Fixes - Complete File Inventory Synchronization
This patch release fixes critical bugs where additional file move operations were not updating the file_inventory table, causing database-filesystem inconsistencies.
---
### Fixed Issues
#### File Inventory Not Updated on Manual Moves (3 Operations)
- **Fixed**: Batch move within /opt/immich/md didn't update file_path in file_inventory
- **Fixed**: Move to review queue didn't update location from 'final' to 'review'
- **Fixed**: Add-as-reference (review → final) didn't update location to 'final'
#### Utility Script Using Filesystem Scanning
- **Fixed**: scan_and_hash_files.py using rglob() instead of file_inventory queries
---
### Technical Changes
#### `/opt/media-downloader/web/backend/api.py`
**`process_batch_move_background()` (line 2769)**:
- Added file_inventory path update after shutil.move()
- Files moved within /opt/immich/md now tracked correctly
- Location stays 'final', path updated to new location
**`move_to_review()` (line 2941)**:
- Added file_inventory location update from 'final' to 'review'
- Updates file_path to new location in /opt/immich/review
- Maintains folder structure in database
**`add_review_as_reference()` (line 4020)**:
- Added file_inventory location update from 'review' to 'final'
- Updates file_path when moving file for face reference
- Ensures database reflects final location
#### `/opt/media-downloader/utilities/scan_and_hash_files.py`
**Database-First Query**:
- Queries file_inventory table instead of using rglob()
- Filters by directory path and extensions
- Graceful fallback to filesystem scan if database query fails
- Significantly faster for large file collections
---
### Impact
All file move operations now properly update file_inventory:
- ✅ Batch move within media directories
- ✅ Move from media → review
- ✅ Move from review → final (with face reference)
- ✅ All previously fixed operations (recycle, delete, restore)
Database queries now return 100% accurate results regardless of how files were moved.
---
### Related
- See v6.34.0 for initial File Inventory System implementation
- See v6.34.1 for recycle bin operation fixes
- See v6.34.2 for thumbnail cache builder integration
- This patch achieves **100% file_inventory synchronization** across all operations
---
## [6.34.2] - 2025-11-16
### ⚡ Performance - Thumbnail Cache Builder Database Integration
This patch release completes the file_inventory integration by updating the thumbnail cache builder to use database queries instead of filesystem scanning.
---
### Performance Improvements
#### Thumbnail Cache Builder
- **Query Speed**: Instant database query instead of recursive directory scan
- **Processing Rate**: 261 files/sec (2,515 files in 12 seconds)
- **Consistency**: All file discovery now uses file_inventory table
- **Architecture**: Database-first approach across entire application
---
### Technical Changes
#### `/opt/media-downloader/modules/thumbnail_cache_builder.py`
**`_get_files_from_inventory()` (new method)**:
- Queries file_inventory table for all media files
- Filters by content_type and validates file existence
- Replaces filesystem scanning with indexed database queries
**`_fallback_filesystem_scan()` (new method)**:
- Graceful fallback if database query fails
- Preserves existing rglob logic as safety net
**`scan_and_process()` (updated)**:
- Now calls `_get_files_from_inventory()` instead of `rglob()`
- Logs "database-first architecture" for visibility
- Processes files from file_inventory table
---
### Impact
The thumbnail cache builder now benefits from the same 50-100x performance boost as the rest of the application:
- ✅ No recursive directory scanning
- ✅ Instant file discovery via indexed queries
- ✅ Consistent file tracking across all services
- ✅ Automatically picks up files added via move_module
---
### Related
- See v6.34.0 for initial File Inventory System implementation
- See v6.34.1 for manual file operation fixes
- This patch completes the file_inventory integration across all components
---
## [6.34.1] - 2025-11-16
### 🐛 Bug Fixes - File Inventory Sync for Manual Operations
This patch release fixes critical bugs where manual file operations through the UI were not updating the file_inventory table, causing database-filesystem inconsistencies.
---
### Fixed Issues
#### File Inventory Not Updated on Manual Moves
- **Fixed**: Review "Keep" operation not updating file_inventory when moving files from review to final
- **Fixed**: Recycle bin delete operations not removing entries from file_inventory
- **Fixed**: Restore from recycle bin not re-adding files to file_inventory
- **Fixed**: Batch delete from media gallery not updating file_inventory
#### Metadata Loss on Restore
- **Fixed**: Restoring files from recycle bin lost platform/source/content_type metadata
- **Fixed**: Restored files showing incorrect location ('final' when should be 'review')
---
### Technical Changes
#### `/opt/media-downloader/modules/unified_database.py`
**`move_to_recycle_bin()` (line 1512)**:
- Now queries file_inventory BEFORE deletion to preserve metadata
- Merges file_inventory metadata with passed metadata for complete restoration
- Deletes entry from file_inventory after successful move to recycle
- Stores full metadata in recycle_bin table for restoration
**`restore_from_recycle_bin()` (line 1586)**:
- Re-adds file to file_inventory with original metadata
- Determines correct location based on `deleted_from` field
- Preserves original platform, source, and content_type
- Uses original file modification time as created_date
**`update_file_inventory_location()` (line 2016)**:
- Added optional `new_file_path` parameter for path updates
- Now handles both location-only updates and location + path updates
- Used when moving files between review/final directories
#### `/opt/media-downloader/web/backend/api.py`
**`process_review_keep_background()` (line 3671)**:
- Now calls `update_file_inventory_location()` after moving file
- Updates both location ('review' → 'final') and file_path
- Logs success/failure for debugging
---
### Impact
All manual file operations now properly update the file_inventory table:
- ✅ Review → Keep (move to final)
- ✅ Review → Delete (move to recycle)
- ✅ Media → Delete (move to recycle)
- ✅ Recycle → Restore (restore to original location)
Database queries will now return accurate results for manually moved files, ensuring the dashboard, galleries, and analytics remain synchronized with the filesystem.
---
### Related
- See v6.34.0 for the initial File Inventory System implementation
- This patch completes the file_inventory integration for all file operations
---
## [6.34.0] - 2025-11-16
### 🚀 File Inventory System - Database-First Architecture (50-100x Performance Boost)
This minor release introduces a revolutionary database-first architecture that eliminates filesystem scanning, resulting in 50-100x faster page loads across the entire application.
---
### ⚡ Performance Improvements
#### Page Load Times
- **Media Gallery**: 5-10 seconds → <100ms (100x faster)
- **Downloads Page**: 5-10 seconds → <100ms (100x faster)
- **Review Queue**: 2-5 seconds → <100ms (50x faster)
- **Dashboard Stats**: 10+ seconds → <100ms (100x faster)
All pages now load instantly with indexed database queries instead of scanning 2,499+ files recursively.
---
### 🆕 New Features
#### File Inventory Table
- **Single Source of Truth**: New `file_inventory` table tracks all 2,499+ files with full metadata
- **Real-Time Sync**: Automatically updated by move_module on every file operation
- **Smart Indexing**: 5 optimized indexes for instant queries (platform, source, location, hash, path)
- **Location Tracking**: Files categorized as 'final', 'review', or 'recycle'
#### Database-First Queries
- All API endpoints migrated from `directory.rglob('*')` to indexed SQL queries
- Dashboard widgets use aggregation queries instead of filesystem scans
- Filter options pulled from database instead of directory scanning
#### Intelligent Date Handling
- Files ordered by actual modification time (not backfill timestamp)
- Preserves original download dates when available
- Proper chronological sorting across all pages
#### Platform Source Detection
- **TikTok**: Queries downloads table for actual username (was showing 'TikTok')
- **Coppermine**: Queries downloads table for album source
- **Instagram/Snapchat**: Extracts username from filename pattern
---
### 🐛 Bug Fixes
- **Fixed**: Files showing same created_date (now uses actual file mtime)
- **Fixed**: TikTok files showing source='unknown' (queries downloads table)
- **Fixed**: Coppermine files missing album source (queries downloads table)
- **Fixed**: Latest files not appearing first (proper ORDER BY created_date DESC)
- **Fixed**: Slow page loads from recursive directory scanning (indexed queries)
---
### 📚 Documentation
- **docs/FILE_INVENTORY.md** - Complete architecture documentation (476 lines)
- **utilities/backfill_file_inventory.py** - Backfill script with usage guide
---
### 🔄 Migration
#### Automatic Setup
1. `file_inventory` table created automatically on API startup
2. Run backfill: `python3 utilities/backfill_file_inventory.py --force`
3. All future downloads automatically tracked
#### Backfill Results
- **2,499 files** scanned and indexed
- **Platforms**: Instagram (522), Forums (1,850), TikTok (25), Snapchat (74), Coppermine (22), Review (6)
- **Performance**: 18 seconds to scan and index all files
---
## [6.33.5] - 2025-11-16
### 🐛 Database Corruption Fix - Media Gallery 500 Error
This patch release fixes a critical bug causing media gallery pages to fail with HTTP 500 errors due to corrupted binary data in the database.
---
### 🎯 Bug Fixes
#### Media Gallery / Downloads Page
- **Fixed**: HTTP 500 error when loading media gallery
- **Fixed**: Database corruption handling for binary blob data in face_recognition_scans table
- **Fixed**: UnicodeDecodeError when serializing face recognition results to JSON
- **Improvement**: Added defensive type conversion for database query results
---
### 🔧 Technical Implementation
#### Root Cause
Row ID 4859 in `face_recognition_scans` table had a `confidence` field stored as **binary blob** (`0x80...`) instead of a REAL (float) value. When FastAPI tried to serialize this to JSON for the `/api/media/gallery` endpoint, it attempted UTF-8 decoding and failed:
```
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```
#### Database Repair
```sql
-- Fixed corrupt confidence value
UPDATE face_recognition_scans SET confidence = NULL WHERE id = 4859
```
#### Code Hardening
```python
# modules/unified_database.py:1424-1478
def get_face_recognition_result():
row = cursor.fetchone()
if row:
result = dict(row)
# Convert any bytes to proper types
for key, value in result.items():
if isinstance(value, bytes):
if key == 'confidence':
# Convert bytes to float using struct.unpack
import struct
result[key] = struct.unpack('d', value)[0]
else:
# Try UTF-8 decode for text fields
result[key] = value.decode('utf-8')
return result
```
This defensive handling ensures future blob data is properly converted instead of crashing the API.
---
### 📊 Impact
**Before Fix:**
- Media page: HTTP 500 error, no thumbnails displayed
- Downloads page: HTTP 500 error, no thumbnails displayed
- Console error: "Failed to load resource: the server responded with a status of 500"
**After Fix:**
- All media gallery endpoints working correctly
- Thumbnails loading properly
- Binary data automatically converted to correct types
---
## [6.33.4] - 2025-11-15
### 🐛 Coppermine Cookie Management Fixed
This patch release fixes cookie management in the Coppermine module, eliminating unnecessary FlareSolverr calls and improving download performance.
---
### 🎯 Bug Fixes
#### Coppermine Module Performance
- **Fixed**: Module no longer calls FlareSolverr on every run when cookies are valid
- **Fixed**: Cookie expiration check now only validates critical `cf_clearance` cookie
- **Fixed**: Cookies loaded with proper `secure`, `httpOnly`, and `expires` attributes
- **Improvement**: Ignores short-lived advertising/tracking cookies in expiration logic
- **Performance**: 2+ minute FlareSolverr wait eliminated when cookies are valid (365 days)
---
### 🔧 Technical Implementation
#### Cookie Validation Logic
```python
# modules/coppermine_module.py:142-199
def _cookies_expired():
- Only checks critical cf_clearance cookie (Cloudflare bypass)
- Ignores short-lived tracking cookies (vido_first_impression, etc.)
- Only triggers refresh when cookie actually expired (not "expiring within 7 days")
- Logs cf_clearance validity period for debugging
```
#### Cookie Loading
```python
# modules/coppermine_module.py:106-153
def _load_cookies():
- Sets cookies with proper attributes (secure, httpOnly, expires)
- Validates cf_clearance presence and logs status
- Improved error handling and logging
```
#### Request Testing
```python
# modules/coppermine_module.py:632-655
- Tests existing cookies before calling FlareSolverr
- Logs cookie count and HTTP response status
- Detailed error messages for debugging
```
---
### 📊 Performance Impact
**Before Fix:**
- Every run: 2+ minute wait for FlareSolverr
- Even when cf_clearance cookie valid for 365 days
**After Fix:**
- Instant start when cookies valid
- FlareSolverr only called when actually needed
- ~120 second time savings per run
---
## [6.33.3] - 2025-11-15
### 🔐 2FA Remember Me Functionality Fixed
This patch release fixes Remember Me checkbox functionality for all 2FA authentication methods (Duo, TOTP, Passkey).
---
### 🎯 Bug Fixes
#### 2FA Session Duration
- **Fixed**: Remember Me checkbox now properly creates 30-day sessions for all 2FA methods
- **Fixed**: Duo 2FA respects Remember Me checkbox value from login page
- **Fixed**: TOTP authentication respects Remember Me checkbox value
- **Fixed**: Passkey authentication respects Remember Me checkbox value
- **Fixed**: Session cookies now set correct `max_age` (30 days when Remember Me checked)
- **Fixed**: Duo OAuth state preserves rememberMe value through redirect flow
---
### 🔧 Technical Implementation
#### Backend Changes
```python
# web/backend/twofa_routes.py:238-347
- Added rememberMe field to DuoAuthRequest model
- Added rememberMe field to TOTPLoginVerifyRequest model
- Added rememberMe field to PasskeyVerifyAuthenticationRequest model
- Updated all 2FA callbacks to use verify_data.rememberMe instead of hardcoded False
- Set cookie max_age: 30 * 24 * 60 * 60 seconds (30 days) when rememberMe is true
# web/backend/duo_manager.py:116-178
- generate_state() stores remember_me in state_store
- verify_state() returns (username, remember_me) tuple
- create_auth_url() accepts and passes remember_me parameter
```
#### Frontend Changes
```typescript
// web/frontend/src/lib/api.ts:928-962
- verifyTOTPLogin() accepts rememberMe parameter
- verifyPasskeyAuthentication() accepts rememberMe parameter
- createDuoAuth() accepts rememberMe parameter
// web/frontend/src/pages/TwoFactorAuth.tsx:37-186
- Extracts rememberMe from login navigation state
- Passes rememberMe to TOTP verification
- Passes rememberMe to Passkey verification
- Passes rememberMe to Duo auth initiation
```
---
### 📊 Root Cause
All three 2FA callback handlers (Duo, TOTP, Passkey) were hardcoded to pass `False` for the `remember_me` parameter, causing sessions to always expire after 1 day instead of 30 days when Remember Me was checked.
---
## [6.29.0] - 2025-11-14
### ✨ Analytics Dashboards - Face Recognition & Enhanced Stats
This minor release adds comprehensive analytics dashboards for face recognition and enhanced statistics visualization throughout the web interface.
---
### 🎯 New Features
#### Face Recognition Dashboard (/faces)
- **Total Statistics**: Overview cards showing total scans, matches, no matches, and match rate
- **Matches by Person**: Top 10 matched people with confidence ranges (avg, min, max) displayed as progress bars
- **Confidence Distribution**: Visual breakdown of match confidence levels in buckets (0-20%, 20-40%, etc.)
- **Recent Matches**: Table of last 20 matches with file names, persons, confidence scores, sources, and dates
- **Scans Over Time**: 30-day chart showing daily scan activity and match rates
- **Auto-Refresh**: Dashboard updates every 30 seconds with latest data
#### Enhanced Dashboard Analytics
- **Download Trends**: 14-day bar chart showing daily download counts
- **Storage by Platform**: Progress bars showing actual disk usage per platform with percentages
- **Content Types**: Breakdown of image/video content types with counts and percentages
- **Top Sources**: Top 10 download sources ranked by download count
- **Auto-Refresh**: Analytics widgets update every 60 seconds
#### FlareSolverr Health Monitoring
- **Status Indicator**: Real-time health check with visual status (healthy/unhealthy/offline)
- **Response Time**: Shows FlareSolverr response time in milliseconds
- **Session Count**: Displays number of active browser sessions
- **Manual Refresh**: Click to force immediate health check
- **Auto-Refresh**: Status updates every 30 seconds
---
### 🔧 Technical Implementation
#### Backend API Endpoints
```python
# web/backend/api.py:4499-4623
@app.get("/api/face/dashboard-stats")
- Total scans, matches, no matches, match rate
- Matches grouped by person with confidence statistics
- Confidence distribution in 5 buckets
- Recent 20 matches with file paths and metadata
- Daily scan counts for last 30 days
# web/backend/api.py:5355-5473
@app.get("/api/stats/dashboard")
- Downloads per day (last 30 days)
- Storage by platform using directory sizes (du -sb)
- Content type breakdown
- Top 10 sources by download count
- Growth rate (this week vs last week)
# web/backend/api.py:5477-5552
@app.get("/api/health/flaresolverr")
- Health check with 10-second timeout
- Response time measurement
- Active session count
- Error handling and status reporting
```
#### Frontend Components
```typescript
// web/frontend/src/pages/FaceRecognitionDashboard.tsx
- New dashboard page with TanStack Query for data fetching
- 4 statistics cards with lucide-react icons
- Progress bar visualizations for matches by person
- Gradient progress bars for confidence distribution
- Responsive table for recent matches
- 30-day timeline chart for scans over time
// web/frontend/src/pages/Dashboard.tsx:730-890
- Download trends bar chart (14 days)
- Storage by platform progress bars
- Content types with icons and percentages
- Top sources list with platform tags
// web/frontend/src/components/FlareSolverrStatus.tsx
- Status indicator component with color-coded badges
- Response time display
- Session count display
- Manual refresh button
- Auto-refresh every 30 seconds
// web/frontend/src/App.tsx:64
- Added "Faces" navigation link with Users icon
- Positioned between Analytics and Platforms
```
#### Storage Calculation Enhancement
```python
# Storage now uses actual directory sizes instead of file_size column
dir_mappings = {
'instagram': '/opt/immich/md/social media/instagram', # 1.0 GB
'tiktok': '/opt/immich/md/social media/tiktok', # 153 MB
'snapchat': '/opt/immich/md/social media/snapchat', # 54 MB
'forum': '/opt/immich/md/forums', # 2.4 GB
'coppermine': '/opt/immich/md/gallery' # 25 MB
}
# Uses du -sb for accurate byte counts
subprocess.run(['du', '-sb', dir_path])
```
#### Platform Grouping
- Instagram scrapers (instagram, fastdl, imginn, toolzu) now merged into single "Instagram" entry
- Cleaner statistics without duplicate platform entries
- Accurate total download counts per logical platform
---
### 🐛 Bug Fixes
#### Database Connection Context Manager
**Problem**: API endpoints throwing error `'_GeneratorContextManager' object has no attribute 'cursor'`
**Root Cause**: Incorrect database connection usage
```python
# ❌ Before (incorrect)
conn = app_state.db.get_connection()
cursor = conn.cursor()
# ✅ After (correct)
with app_state.db.get_connection() as conn:
cursor = conn.cursor()
```
**Impact**: Both `/api/face/dashboard-stats` and `/api/stats/dashboard` were returning 500 errors
#### Storage Widget Only Showing Coppermine
**Problem**: Storage by platform only showed coppermine (22 files)
**Root Cause**: Query filtered `WHERE file_size IS NOT NULL`, but only coppermine had file_size populated in database
**Solution**: Changed to use actual directory sizes via `du -sb` command
```python
# Now shows all platforms with accurate disk usage:
# - Forum: 2.4 GB (forums directory)
# - Instagram: 1.0 GB (social media/instagram directory)
# - TikTok: 153 MB (social media/tiktok directory)
# - etc.
```
#### Instagram Stats Split Across Platforms
**Problem**: Instagram downloads were counted separately as instagram, fastdl, imginn, toolzu
**Solution**: Added platform grouping to merge all Instagram scrapers
```python
instagram_platforms = ['instagram', 'fastdl', 'imginn', 'toolzu']
# All now counted under single "Instagram" platform
```
#### Forum Platform Showing 0 Bytes
**Problem**: Forum had 141 downloads but 0 bytes (files deleted from disk)
**Solution**: Filter platforms to only show those with `size_bytes > 0`
---
### 📊 Usage Examples
#### Access Face Recognition Dashboard
```
Navigate to: http://your-server:5173/faces
- Click "Faces" in main navigation menu
- View comprehensive face recognition analytics
- Monitor scan success rates
- Identify most frequently matched people
- Track confidence score distributions
```
#### View Enhanced Dashboard Analytics
```
Navigate to: http://your-server:5173/
- Scroll down to see new analytics widgets
- Download Trends: Last 14 days activity
- Storage by Platform: Disk usage breakdown
- Content Types: Image vs video distribution
- Top Sources: Most active download sources
```
#### Monitor FlareSolverr Health
```
Location: Top right of Dashboard header
- Green = Healthy (< 5s response time)
- Yellow = Slow (5-10s response time)
- Red = Unhealthy/Offline
- Click to manually refresh status
```
---
### 🔄 Migration Guide
#### No Database Changes Required
This release adds new API endpoints but does not modify the database schema.
**Face Recognition Prerequisites:**
- Requires `face_recognition_scans` table (added in v6.5.0)
- If upgrading from pre-v6.5.0, face recognition dashboard will show empty data until scans run
#### Frontend Update
```bash
# Frontend automatically updates on refresh (dev mode)
# For production, rebuild is automatic via systemd service restart
systemctl restart media-downloader-frontend
```
#### API Service Update
```bash
# Backend API needs restart to load new endpoints
systemctl restart media-downloader-api
```
---
### 📝 Files Modified
#### Backend Changes
- `web/backend/api.py` - Added 3 new endpoints (face/dashboard-stats, stats/dashboard, health/flaresolverr)
#### Frontend Changes
- `web/frontend/src/pages/FaceRecognitionDashboard.tsx` - New page (304 lines)
- `web/frontend/src/pages/Dashboard.tsx` - Added analytics widgets (+161 lines)
- `web/frontend/src/components/FlareSolverrStatus.tsx` - New component (152 lines)
- `web/frontend/src/lib/api.ts` - Added API client methods (+69 lines)
- `web/frontend/src/App.tsx` - Added Faces navigation link
#### Documentation
- `README.md` - Updated version to 6.29.0
- `VERSION` - Updated to 6.29.0
- `data/changelog.json` - Added v6.29.0 entry
- `docs/CHANGELOG.md` - Added detailed release notes
---
## [6.28.3] - 2025-11-14
### 🔧 Requirements.txt Update - DeepFace Dependencies
This patch release fixes missing dependencies in requirements.txt. The application switched to DeepFace for face recognition in v6.26.0, but the dependencies were never added to requirements.txt, causing new installations to fail.
---
### 🐛 Problem Fixed
#### Missing Dependencies
- **Issue**: DeepFace and TensorFlow were not listed in requirements.txt
- **History**: Face recognition switched from dlib to DeepFace in v6.26.0
- **Impact**: New installations would fail with "ModuleNotFoundError: No module named 'deepface'"
- **Root Cause**: Dependencies were installed manually but never documented
#### Production Environment
Current system has these installed:
- deepface 0.0.95 ✓
- tensorflow 2.20.0 ✓
- face_recognition 1.3.0 ✓ (fallback)
- dlib 20.0.0 ✓ (for fallback)
- numpy 2.2.6 ✓
---
### ✅ Solution
#### Updated requirements.txt
```python
# Face recognition (v6.5.0, upgraded to DeepFace in v6.26.0)
deepface>=0.0.95 # DeepFace with Facenet512 + RetinaFace (primary, since v6.26.0)
tensorflow>=2.18.0 # TensorFlow for DeepFace models
face-recognition>=1.3.0 # Old face recognition library (fallback only)
dlib>=19.24.0 # Machine learning toolkit (required by face-recognition fallback)
numpy>=1.24.0 # Numerical computing (required by both libraries)
```
#### What Changed
1. **Added** deepface>=0.0.95 (primary face recognition library)
2. **Added** tensorflow>=2.18.0 (required by DeepFace)
3. **Updated** comments to clarify which library is primary
4. **Clarified** face-recognition and dlib are fallback only
---
### 📋 Technical Details
#### Face Recognition Architecture (since v6.26.0)
```python
# modules/face_recognition_module.py
try:
from deepface import DeepFace
DEEPFACE_AVAILABLE = True
except ImportError:
DEEPFACE_AVAILABLE = False
try:
import face_recognition
FACE_RECOGNITION_AVAILABLE = True
except ImportError:
FACE_RECOGNITION_AVAILABLE = False
```
**Priority:**
1. DeepFace (Facenet512 + RetinaFace) - Primary, best accuracy
2. face_recognition (dlib HOG/CNN) - Fallback if DeepFace not available
#### System Dependencies (from installer)
```bash
# scripts/install.sh already installs these for both libraries:
apt-get install -y cmake build-essential libopenblas-dev liblapack-dev ffmpeg
```
These system dependencies support both:
- dlib (legacy face-recognition library)
- TensorFlow (DeepFace)
---
### 🔄 Migration Guide
#### For New Installations
```bash
# Clone repository
cd /opt/media-downloader
# Run installer (will now install deepface and tensorflow automatically)
sudo ./scripts/install.sh
```
#### For Existing Installations
**No action needed** - your system already has all dependencies installed.
To verify:
```bash
source venv/bin/activate
python3 -c "from deepface import DeepFace; import tensorflow; print('✓ All dependencies OK')"
```
#### Manual Installation (if needed)
```bash
source venv/bin/activate
pip install deepface>=0.0.95 tensorflow>=2.18.0
```
---
### 📊 Dependency Versions
#### Minimum Requirements
- deepface >= 0.0.95
- tensorflow >= 2.18.0
- face-recognition >= 1.3.0 (fallback)
- dlib >= 19.24.0 (fallback)
- numpy >= 1.24.0
#### Production Versions (tested)
- deepface 0.0.95 ✓
- tensorflow 2.20.0 ✓
- face_recognition 1.3.0 ✓
- dlib 20.0.0 ✓
- numpy 2.2.6 ✓
---
### 🧪 Testing
Verified all dependencies load correctly:
```python
✓ deepface: OK
✓ tensorflow: OK (version 2.20.0)
✓ face_recognition: OK (fallback)
✓ numpy: OK (version 2.2.6)
✓ All required face recognition dependencies are installed
```
TensorFlow CPU optimization message is normal and expected.
---
### 📝 Why This Matters
**Before this fix:**
- New users running `pip install -r requirements.txt` would get incomplete installation
- Face recognition would fail with import errors
- Manual intervention required to install missing packages
- Documentation didn't match reality
**After this fix:**
- Complete dependency list in requirements.txt
- New installations work out of the box
- Installer automatically handles everything
- Clear documentation of primary vs fallback libraries
---
### 🔗 Related Changes
- **v6.26.0** (2025-11-12): Switched from dlib to DeepFace for face recognition
- **v6.28.2** (2025-11-14): Added separate tolerance settings for images vs videos
- **v6.28.3** (2025-11-14): Fixed missing dependencies in requirements.txt (this release)
---
## [6.28.2] - 2025-11-14
### 🐛 Face Recognition Tolerance Fix - Separate Image & Video Settings
This patch release fixes false positive face recognition matches by implementing separate tolerance settings for images and videos. The previous version (6.28.1) set tolerance to 0.35 globally for videos, which was too lenient for images and caused false positives.
---
### 🐛 Problem Fixed
#### False Positive Matches
- **Issue**: Files from parismatch were incorrectly flagged as Eva Longoria matches
- **Root Cause**: Tolerance 0.35 (set for videos in v6.28.1) was being used for images too
- **Confidence Range**: False positives had 70-85% confidence (below the 85% threshold)
- **Impact**: User had to manually move false positives to recycle bin
---
### ✨ Solution: Separate Tolerances
#### New Architecture
- **Image Tolerance**: 0.15 (strict) - Requires 85%+ confidence
- Prevents false positives
- Recommended range: 0.15-0.20
- **Video Tolerance**: 0.30 (lenient) - Allows 70%+ confidence
- Video frames are harder to match (motion blur, compression)
- Recommended range: 0.25-0.35
#### Configuration Page Updates
- **Image Tolerance** input: Existing field with updated help text
- Help: "Lower = stricter matching. Recommended: 0.15-0.20 for images. Range: 0.0-1.0"
- **Video Tolerance** input: New field in Video Recognition Settings section
- Help: "Video frames are harder to match. Recommended: 0.25-0.35"
- Only visible when "Enable video face recognition" is checked
---
### 📋 Technical Changes
#### Frontend Changes
- `web/frontend/src/pages/Configuration.tsx`
- Line 2933: Updated image tolerance help text
- Lines 2971-2987: Added Video Tolerance input field
- Default video tolerance: 0.30
- Integrated into Video Recognition Settings section
#### Backend Changes
- `modules/move_module.py`
- Lines 118-167: Enhanced `_get_face_recognition_tolerance(is_video=False)`
- New parameter: `is_video` (bool)
- Separate caching for image vs video tolerance
- Reads `video_tolerance` from settings for videos
- Reads `tolerance` from settings for images
- Defaults: 0.15 for images, 0.30 for videos
- Line 542: Videos call `_get_face_recognition_tolerance(is_video=True)`
- Line 545: Added tolerance value to video checking log
- Line 623: Images call `_get_face_recognition_tolerance(is_video=False)`
- Line 624: Added tolerance value to image checking debug log
#### Database Changes
- `database/media_downloader.db`
- Settings table updated:
- `tolerance`: 0.35 → 0.15 (for images)
- `video_tolerance`: 0.30 (new setting)
---
### 🔄 Migration Guide
#### Automatic Changes
1. Image tolerance automatically reset from 0.35 to 0.15
2. New `video_tolerance` setting added with default 0.30
3. Both settings configurable via Configuration page
#### Tuning Recommendations
**If you get false positives (incorrect matches):**
- Lower image tolerance to 0.10-0.12 (very strict, 88-90%+ confidence required)
**If legitimate images aren't matching:**
- Increase image tolerance to 0.18-0.20 (more lenient, 80-82%+ confidence)
**If videos aren't matching:**
- Increase video tolerance to 0.35-0.40 (allows 60-65%+ confidence)
**If videos have false positives:**
- Lower video tolerance to 0.25 (requires 75%+ confidence)
---
### 📊 Expected Results
#### Before (v6.28.1)
- Both images and videos used 0.35 tolerance
- False positive parismatch files at 70-85% confidence
- 2,396 total matches with some incorrect
#### After (v6.28.2)
- Images use 0.15 tolerance (85%+ confidence required)
- Videos use 0.30 tolerance (70%+ confidence required)
- False positives eliminated for images
- Video matching still works (Snapchat videos matched at 67-74%)
---
### 🔍 Testing
#### Face Recognition Statistics (from database)
- **Total Matches**: 2,396 Eva Longoria matches
- **Images**: 1,941 matches (avg confidence: 56.6%)
- **Videos**: 429 matches (avg confidence: 59.6%)
- **Test Person**: 51 matches (avg confidence: 51.0%)
With 0.15 tolerance, only matches with >85% confidence will be accepted for images, eliminating the false positives reported from parismatch.
---
### 💡 How Tolerance Works
**DeepFace Cosine Distance:**
- 0.0 = Perfect match (100% confidence)
- 0.15 = Strong match (~85% confidence) ← Image threshold
- 0.30 = Good match (~70% confidence) ← Video threshold
- 0.40 = Weak match (~60% confidence)
- 1.0 = No match (0% confidence)
**Formula**: `confidence = 1 - (distance / max_distance)`
Lower tolerance = stricter matching = fewer false positives (but might miss some real matches)
Higher tolerance = looser matching = more matches (but might include false positives)
---
## [6.28.1] - 2025-11-13
### 🐛 Critical Bug Fixes & Log Viewer Enhancements
This patch release fixes several critical bugs including the ImgInn tagged posts scraper, improves video face recognition, and enhances the log viewer with better visual feedback for errors and warnings.
---
### 🐛 Bug Fixes
#### ImgInn Tagged Posts Scraping
- **CRITICAL FIX**: Fixed stale element timeout issue preventing tagged posts from being scraped
- Error: "Timeout 30000ms exceeded waiting for locator().nth(49)"
- Root cause: Playwright elements became stale after `page.go_back()` navigation
- Solution: Extract all post URLs before loop, use `page.goto()` instead of `element.click()`
- **ENHANCED**: Added infinite scroll to load up to 50 tagged posts (previously only loaded 12)
- Scrolls up to 10 times with 1.5s wait between scrolls
- Stops when no new posts load or max_posts reached
#### Video Face Recognition
- **FIXED**: Video face recognition tolerance adjusted from 0.15 to 0.35
- 0.15 was too strict for video frames (worked for images)
- All 3 tested Snapchat videos now match successfully (67-74% confidence)
- **FIXED**: `enable_video_recognition` setting now stored as boolean `true` instead of number `1`
- **ADDED**: Video face recognition settings in Configuration page
- Enable/disable video recognition checkbox
- Video frame count setting (defaults to 3 frames at 10%, 50%, 90%)
#### Log Viewer Issues
- **FIXED**: Duplicate "Media Downloader" components appearing in dropdown
- Old naming: `mediadownloader`
- New naming: `media_downloader`
- Renamed 18 existing log files to match new convention
- **FIXED**: Perceptual duplicate detector component name
- Old: `perceptualduplicatedetector`
- New: `perceptual_duplicate_detector`
- **OPTIMIZED**: Empty log files (905 files) no longer appear in component dropdown
- Added size check: `if stat_info.st_size == 0: continue`
#### Systemd Logging
- **FIXED**: web-api.log and web-frontend.log continuing to grow after config update
- Services needed restart to apply systemd configuration changes
- Both services now properly log to systemd journal
- Old flat files can be safely deleted
---
### 🎨 Enhancements
#### Log Viewer Visual Improvements
- **ERROR/CRITICAL**: Red text with bold font + red background + left border
```
bg-red-950/30 hover:bg-red-950/40 border-l-4 border-red-500
text-red-500 dark:text-red-400 font-semibold (ERROR)
text-red-600 dark:text-red-300 font-bold (CRITICAL)
```
- **WARNING**: Amber text with medium weight + amber background + left border
```
bg-amber-950/20 hover:bg-amber-950/30 border-l-4 border-amber-500
text-amber-500 dark:text-amber-300 font-medium
```
#### Component Name Consistency
- All log components now use standardized naming with underscores
- Consistent with Python module naming conventions
- Easier to read and maintain
---
### 📋 Technical Changes
#### Modified Files
- `modules/imginn_module.py`
- Lines 1944-1971: Added infinite scroll logic
- Lines 1996-2034: Fixed stale element issue with URL extraction
- `modules/unified_database.py`
- Line 2014: Added `content_type` parameter to `mark_downloaded()`
- `web/frontend/src/pages/Logs.tsx`
- Enhanced `getLogColor()` with font weights
- Added background highlighting with `border-l-4`
- `web/frontend/src/pages/Configuration.tsx`
- Added video recognition settings section
- `database/media_downloader.db`
- Updated `face_recognition.enable_video_recognition`: `1` → `true`
- Updated `face_recognition.tolerance`: `0.15` → `0.35`
- `media-downloader.py`
- Lines 201, 2549: Logger name `'MediaDownloader'` → `'Media_Downloader'`
- `modules/instagram_perceptual_duplicate_detector.py`
- Line 63: Logger name `'PerceptualDuplicateDetector'` → `'Perceptual_Duplicate_Detector'`
- `web/backend/api.py`
- Lines 3475-3512: Added empty log file filtering
- `/etc/systemd/system/media-downloader-api.service`
- Changed `StandardOutput` and `StandardError` to `journal`
- `/etc/systemd/system/media-downloader-frontend.service`
- Changed `StandardOutput` and `StandardError` to `journal`
#### File Operations
- Renamed 18 log files: `*_mediadownloader.log` → `*_media_downloader.log`
- Renamed perceptual duplicate detector logs with underscores
- Removed deprecated `unified.db` (0 bytes, unused)
---
### 🔄 Migration Notes
1. **Log File Naming**: Old log files were automatically renamed to match new convention
- `20251113_*_mediadownloader.log` → `20251113_*_media_downloader.log`
2. **Deprecated Logs**: Can safely delete:
- `web-api.log`
- `web-frontend.log`
3. **Face Recognition**: Video tolerance now separate from image tolerance (0.35 vs 0.15)
4. **Systemd Services**: API and frontend services restarted to apply journal logging
---
### 📊 Testing Results
#### ImgInn Tagged Posts
- **Before**: 12 posts found, then timeout error
- **After**: 48 posts loaded successfully via infinite scroll
- **Test User**: Eva Longoria (@evalongoria)
#### Video Face Recognition
- **Test Set**: 3 Snapchat videos from 2025-11-13
- **Tolerance 0.15**: 0/3 matched (too strict)
- **Tolerance 0.35**: 3/3 matched (67-74% confidence)
#### Log Component Cleanup
- **Empty Files Removed**: 905 components
- **Duplicate Components Fixed**: Media Downloader (2 → 1)
- **Naming Standardized**: All components use underscore format
---
## [6.28.0] - 2025-11-13
### 🎯 Universal Logging System & Comprehensive Cleanup
This major release introduces a comprehensive universal logging system that standardizes logging across all 27 modules and components. All logs now use datetime-stamped filenames with consistent formatting, automatic 7-day retention, and a professional web interface for viewing and filtering. The application has undergone a thorough cleanup, removing unused files and standardizing component names throughout the UI.
---
### 🎯 Key Features
#### Universal Logging System
- **Datetime-Stamped Log Files**: All logs now use format `YYYYMMDD_HHMMSS_component.log`
- Matches the existing media-downloader.py pattern
- Easy chronological sorting and identification
- No more date-based rotation conflicts
- **7-Day Automatic Retention**: Old logs automatically deleted on application startup
- Keeps disk usage under control
- Configurable retention period in UniversalLogger class
- **Consistent Format**: All log entries follow standard format
- `2025-11-13 11:24:40 [MediaDownloader.Component] [Module] [LEVEL] message`
- Color-coded levels in terminal output
- Structured parsing in web UI
- **Universal Integration**: All 27 modules now use the same logger
- modules/universal_logger.py - Core logging module
- 43 logging calls updated in API server
- 100+ calls updated in scheduler
- 172 calls updated in main application
- All 8 download modules integrated
- All 15+ utility modules integrated
#### Professional Logs Page
- **Component Selector Dropdown**: Switch between different component logs
- API Server, Scheduler, Media Downloader, Database, etc.
- Automatically discovers available components from log files
- Sorted alphabetically with clean display names
- Defaults to Media Downloader logs on page load
- **Parsed Log Display**: Structured log format with color-coding
- Timestamp, component, module, level, and message sections
- Color-coded level badges (ERROR: red, WARNING: amber, INFO: blue, etc.)
- Monospace font for readability
- Syntax highlighting for different log sections
- **Level Filtering**: Filter logs by level (ERROR, WARNING, INFO, SUCCESS, DEBUG)
- Multi-select checkbox interface
- Real-time filtering without page reload
- Reset filters button
- Persistent filter state during session
- **Stats Dashboard**: Real-time statistics
- Total logs, Errors, Warnings, Success, Filtered
- Color-coded stats cards
- Updates automatically as new logs arrive
- **iOS Optimizations**: Mobile-friendly interface
- 44px minimum touch targets (iOS standard)
- Collapsible filter section for mobile
- WebkitOverflowScrolling for native feel
- Responsive grid layout (3 cols mobile, 5 cols desktop)
- Touch-manipulation CSS for better responsiveness
- **Auto-scroll Control**: Toggle automatic scrolling
- Manual scroll detection
- Auto-scroll checkbox in header
- Maintains position when disabled
#### Component Name Standardization
- **Clean Display Names**: All component names now show without hyphens/underscores
- API Server (not "api" or "web-api")
- Settings Manager (not "settingsmanager")
- Activity Monitor (not "activitystatus")
- File Manager (not "movemanager")
- **Comprehensive Mapping**: 30+ component name mappings
- Core services, main components, download modules
- Analysis modules, utilities, legacy logs
- Intelligent fallback for unmapped names
- **Legacy Log Handling**: Old log files marked as "(Legacy)"
- web-api.log → "API Server (Legacy)"
- web-frontend.log → "Frontend Server (Legacy)"
- service.log → "Service (Legacy)"
---
### 📋 Technical Changes
#### New Files
- `modules/universal_logger.py` - Universal logging module
- UniversalLogger class with datetime-stamped files
- Automatic 7-day cleanup
- Console and file handlers with separate log levels
- Module-based tagging for filtering
- get_logger() factory function for easy integration
#### Updated Files
- `web/backend/api.py` - API server logging integration (43 calls)
- Replaced all logging calls with universal logger
- Updated /api/logs endpoint for datetime-stamped files
- Added component discovery and filtering
- `modules/scheduler.py` - Scheduler logging integration (100+ calls)
- Integrated universal logger throughout
- Maintained log_callback compatibility
- `modules/move_module.py` - Fixed Move Manager logging
- log() method now properly calls universal logger
- Maintains backwards compatibility with log_callback
- `modules/thumbnail_cache_builder.py` - Cache builder integration (21 calls)
- `media-downloader.py` - Main application integration (172 calls)
- `web/frontend/src/pages/Logs.tsx` - Complete logs page redesign
- Component selector dropdown (lines 200-230)
- Enhanced log parsing (lines 53-66)
- Component name mapping (lines 129-183)
- iOS optimizations throughout
- Stats dashboard (lines 232-280)
- Level filtering (lines 242-285)
- `web/frontend/package.json` - Updated version to 6.28.0
- `web/frontend/VERSION` - Updated to 6.28.0
- `README.md` - Updated version references
- `VERSION` - Updated to 6.28.0
#### Download Modules Updated (8 modules, 990 logging calls)
- `modules/forum_downloader.py` → get_logger('Forum')
- `modules/instaloader_module.py` → get_logger('Instagram')
- `modules/fastdl_module.py` → get_logger('Instagram')
- `modules/imginn_module.py` → get_logger('Instagram')
- `modules/toolzu_module.py` → get_logger('Instagram')
- `modules/tiktok_module.py` → get_logger('TikTok')
- `modules/snapchat_module.py` → get_logger('Snapchat')
- `modules/coppermine_module.py` → get_logger('Coppermine')
#### Utility Modules Updated (15+ modules)
- `modules/settings_manager.py`
- `modules/activity_status.py`
- `modules/pushover_notifier.py`
- `modules/face_recognition_module.py`
- `modules/unified_database.py`
- `modules/download_manager.py`
- `modules/date_utils.py`
- `modules/service_health_monitor.py`
- `modules/dependency_updater.py`
- And more...
---
### 🐛 Bug Fixes
1. **Move Manager Logging**: Fixed Move Manager not actually using universal logger
- Was importing get_logger() but not calling it
- log() method now properly calls logger.log()
- Location: modules/move_module.py:341-355
2. **Logs Page Default Component**: Fixed page not defaulting to Media Downloader
- Changed from 'mediadownloader' to 'media_downloader'
- Now correctly shows main application logs on page load
- Location: web/frontend/src/pages/Logs.tsx:9
3. **Component Name Display**: Fixed hyphens/underscores in component dropdown
- Added comprehensive name mapping
- Intelligent fallback for unmapped names
- Location: web/frontend/src/pages/Logs.tsx:129-183
4. **Frontend Version Sync**: Fixed VERSION file out of sync
- Was 6.4.4, now 6.28.0
- Matches application version
- Location: web/frontend/VERSION
5. **Log File Discovery**: Fixed component list not showing all available logs
- Updated endpoint to parse datetime-stamped filenames
- Extracts component name from filename
- Location: web/backend/api.py:4580-4653
6. **Mobile Touch Targets**: Fixed touch targets too small for iOS
- Increased to 44px minimum (Apple standard)
- Larger checkboxes on mobile (20px vs 16px)
- Location: web/frontend/src/pages/Logs.tsx (multiple locations)
---
### 🧹 Cleanup & Organization
1. **Documentation Organization**
- Moved `UNIVERSAL_LOGGING_IMPLEMENTATION.txt` to docs/
- Moved `VERSION_6.27.0_RELEASE_SUMMARY.txt` to docs/
- All documentation now in proper docs/ folder
2. **Code Quality**
- All Python modules compile successfully (syntax check)
- No unused temporary or backup files
- Consistent logging patterns across all modules
3. **Version Management**
- Updated all version references to 6.28.0
- README.md, package.json, VERSION files synchronized
- Changelog updated with comprehensive release notes
---
### 📊 Statistics
- **27 modules** integrated with universal logger
- **43 logging calls** updated in API server
- **100+ logging calls** updated in scheduler
- **172 logging calls** updated in main application
- **990 logging calls** updated across 8 download modules
- **30+ component name mappings** created
- **7-day retention** automatically enforced
- **2 documentation files** moved to docs folder
---
### 🚀 Migration Notes
1. **Log File Format Change**: Old log files still accessible
- Legacy logs show as "(Legacy)" in dropdown
- New logs use YYYYMMDD_HHMMSS_component.log format
- Old logs will be cleaned up manually if needed
2. **Automatic Cleanup**: Logs older than 7 days deleted automatically
- Runs on application startup
- Configurable in universal_logger.py
- Applies to all datetime-stamped log files
3. **Component Discovery**: New components appear automatically
- Download modules (Instagram, TikTok, etc.) appear after first run
- Move Manager appears after first file move
- No manual configuration needed
---
### 📝 Breaking Changes
None. All changes are backwards compatible.
---
### ⚡ Performance
- **Indexed Log Discovery**: O(1) component lookups
- **Client-side Filtering**: Real-time without backend calls
- **Efficient Parsing**: Regex-based log structure extraction
- **Lazy Loading**: Stats calculated on-demand
- **Optimized Cleanup**: Single pass through log directory
---
## [6.27.0] - 2025-11-13
### ✨ Lightbox Metadata Enhancements & Face Recognition Improvements
This release enhances the lightbox viewer with comprehensive metadata display including resolution and face recognition status. The metadata panel now features a beautiful card design with smooth animations. Face recognition has been improved with better path handling for files containing special characters, and the Configuration page now displays useful statistics about reference faces.
---
### 🎯 Key Features
#### Lightbox Enhancements
- **Resolution Display**: Shows image/video dimensions (width x height) in Details panel
- Pulled from media_metadata.db cache (2,743+ cached items)
- Displays for all media across Review, Media, Downloads, and Dashboard pages
- Auto-populated by metadata cache builder service
- **Face Recognition Status**: Complete face match information in lightbox
- Shows matched person name with confidence percentage
- Displays "No match" for unmatched faces
- Color-coded: green for matches, gray for no match
- **Sliding Metadata Card**: Beautiful card design with animations
- Gradient background (gray-900 to black)
- Smooth slide-in animation with fade effect
- Rounded corners and elevated shadow
- Toggle button with proper click handling
#### Configuration Page Improvements
- **Reference Face Statistics**: Live display of face recognition references
- Shows total reference count (currently 39 references)
- Breakdown by person name with individual counts
- First and last added dates for each person
- Auto-refreshes every 30 seconds
- **API Integration**: New endpoint for reference stats
- GET /api/face/reference-stats returns structured data
- Groups references by person name
- Includes timestamp information
#### Face Recognition Fixes
- **Path Special Character Handling**: Fixed DeepFace errors with special characters
- Handles paths with spaces and non-breaking spaces (Unicode \xa0)
- Creates temporary files with clean names for processing
- Automatically cleans up temp files after processing
- Fixes "Add Reference" feature for all file paths
- **Import Error Prevention**: Made face_recognition library import optional
- No longer crashes when old face_recognition library not installed
- Graceful fallback with FACE_RECOGNITION_AVAILABLE flag
- Prevents ModuleNotFoundError on API startup
---
### 📋 Technical Changes
**Frontend - Lightbox Component**:
- Lines 562-665: Added metadata panel card with gradient styling and animations
- Lines 574-594: Added resolution display section with ImageIcon
- Lines 639-660: Face recognition display section (already present, verified working)
- Lines 666-679: Added metadata toggle button with e.stopPropagation() to prevent lightbox closure
- Metadata panel now uses card design: `from-gray-900/95 to-black/95 backdrop-blur-xl rounded-2xl`
**Backend - API Endpoints**:
- Lines 3299-3330: Added width/height metadata to get_media_gallery endpoint
- Lines 3701-3731: Added width/height metadata to get_review_queue endpoint
- Lines 1044-1077: Added width/height metadata to get_downloads_from_filesystem endpoint
- Lines 3348, 4067: Fixed face_recognition field name (person → person_name) in 2 locations
- Lines 4390-4422: Added GET /api/face/reference-stats endpoint with grouping logic
**Frontend - API Client**:
- Lines 718-729: Added getFaceReferenceStats() method with TypeScript types
- Lines 71-97: Enhanced error handling in GET requests to extract detail from response body
**Frontend - Configuration Page**:
- Lines 2586-2635: Added ReferenceStatsSection component with useQuery hook
- Lines 2893-2894: Integrated reference stats in Face Recognition settings section
- Component shows loading state, error state, and empty state appropriately
**Backend - Face Recognition Module**:
- Lines 7-29: Made face_recognition import optional with try/except and FACE_RECOGNITION_AVAILABLE flag
- Lines 311-359: Added temp file workaround in detect_faces_deepface() for paths with special characters
- Checks for spaces, non-breaking spaces (\xa0), and non-ASCII characters
- Uses tempfile.NamedTemporaryFile for clean temporary paths
---
### 🐛 Bug Fixes
1. **Lightbox Metadata Not Showing**:
- Fixed resolution not displaying (was missing from API responses)
- Fixed face recognition data not showing (field name mismatch: person vs person_name)
- All three endpoints now return width/height from metadata cache
2. **Metadata Panel Toggle Issue**:
- Fixed toggle button closing entire lightbox
- Added e.stopPropagation() to button onClick handler
- Panel now slides in/out correctly without affecting lightbox
3. **Face Recognition Path Errors**:
- Fixed DeepFace failing on paths with spaces
- Fixed handling of non-breaking spaces (Unicode \xa0) in forum downloads
- Fixed "Add Reference" feature for files with special characters
4. **API Import Error**:
- Fixed ModuleNotFoundError when face_recognition library not installed
- Made old library import optional since DeepFace is now primary
5. **API Error Messages**:
- Fixed truncated error messages in frontend
- Now extracts full detail from backend response
- Better debugging information for users
6. **Metadata Panel Styling**:
- Changed from plain panel to beautiful card design
- Added gradient background and proper spacing
- Better visual hierarchy with rounded corners
---
### 📊 Statistics
- **Reference Faces**: 39 Eva Longoria references active
- **Metadata Cache**: 2,743+ media items with cached dimensions
- **Code Quality**: All face recognition methods consistently use DeepFace
- **API Endpoints**: 3 endpoints updated with metadata support
- **Frontend Components**: 1 new component (ReferenceStatsSection)
---
### 🔧 Configuration
No configuration changes required. The reference statistics section automatically appears in Configuration → Downloads → Face Recognition when face recognition is enabled.
---
### ⚠️ Breaking Changes
None. This is a fully backward-compatible enhancement release.
---
### 📝 Notes
- All endpoints now consistently return width/height metadata from media_metadata.db cache
- Face recognition statistics auto-refresh every 30 seconds for live updates
- Metadata panel features smooth 300ms slide-in animation with opacity fade
- DeepFace path handling now works with any Unicode characters in file paths
- Reference statistics show useful information: total count, per-person breakdown, and dates
---
## [6.24.0] - 2025-11-11
### ✨ Perceptual Hashing Enhancements & UI Improvements
This release enhances the perceptual duplicate detection system with better OCR accuracy using EasyOCR and automatic hash recording for future use. The UI improvements make the system more responsive with better feedback and control. The perceptual hash database now builds automatically even when detection is disabled, enabling instant activation when needed.
---
### 🎯 Key Features
#### Perceptual Hashing Improvements
- **EasyOCR Integration**: Better text overlay detection for Instagram duplicates
- Uses EasyOCR as primary OCR engine with Tesseract as fallback
- More accurate detection of text overlays and watermarks
- Better handling of stylized text on social media posts
- **Always-On Hash Recording**: Build historical database automatically
- Perceptual hashes now recorded even when detection is disabled
- Enables instant activation without backfill when detection is enabled
- Database grows organically over time with all downloads
- **Schema Integration**: Perceptual hashes table added to unified database schema
- Automatic table creation on fresh installs
- Proper indexes for fast lookups by platform/source
- Support for UPDATE operations on re-processed files
#### UI Enhancements
- **Scheduler Start Button**: Start scheduler service directly from Dashboard
- Green "Start" button on Scheduler Disabled card
- Success/error notifications for feedback
- Auto-refresh after starting
- **Stop Button Feedback**: Clear notifications when stopping downloads
- Success notification on successful stop
- Error notification with details if stop fails
- Better user feedback for download control
- **Improved Toast Position**: Notifications appear lower and less intrusive
- Mobile: Moved from 80px to 128px (48px lower)
- Desktop: Moved from 16px to 80px (64px lower)
- Better visibility without blocking content
---
### 📋 Technical Changes
**Backend - Perceptual Duplicate Detector**:
- Lines 35-39, 63-76: Added EasyOCR import and initialization with error handling
- Lines 88-164: Modified `check_and_handle_duplicate()` to always record hashes before checking if detection enabled
- Lines 298-387: Rewrote `_detect_text_overlays()` to use EasyOCR as primary with Tesseract fallback
- Lines 592-676: Added UPDATE support in `_store_perceptual_hash()` to prevent duplicate key errors on re-processing
**Backend - Database Schema**:
- Lines 338-358: Added `instagram_perceptual_hashes` table to UnifiedDatabase schema
- Lines 410-412: Added indexes for fast lookups (platform_source, file_path, perceptual_hash)
**Frontend - Configuration Page**:
- Lines 24-105: Added global notification state (`globalSuccess`, `globalError`) to prevent duplicate notifications
- Lines 787-815, 2586-2614, 3381-3408: Updated PlatformsTab, DownloadsTab, DependenciesTab to use parent notification callbacks
- Removed local success/error states from tabs to use shared notification system
**Frontend - Dashboard Page**:
- Lines 90, 254-271: Added scheduler start handler (`handleStartScheduler`) and state (`startingScheduler`)
- Lines 237-254: Added success/error notifications to stop handler (`handleStopActivity`)
- Lines 467-493: Added Start button to Scheduler Disabled card with loading state
**Frontend - Notification Component**:
- Line 82: Adjusted toast position from `top-20 md:top-4` to `top-32 md:top-20`
---
### 🐛 Bug Fixes
- **Fixed Configuration page showing 2 save notifications**: Moved notification state to parent component
- **Fixed Stop button not showing feedback**: Added success/error notifications with proper error handling
- **Fixed toast notifications appearing too high**: Adjusted positioning for better visibility
---
### 🔄 Breaking Changes
None.
---
## [6.23.0] - 2025-11-11
### ✨ Downloads & Dashboard Lightbox UI Enhancement
This release extends the enhanced lightbox UI from the Media and Review pages to the Downloads and Dashboard pages, providing a consistent user experience across all pages. Users can now Move to Review, Add Reference (with custom name), Quick Add Face, and Delete directly from lightbox viewers on any page.
---
### 🎯 Key Features
#### Downloads Page Lightbox Enhancement
- **Four Action Buttons**: Comprehensive controls in lightbox viewer
- **Move to Review**: Move items while viewing
- **Add Reference**: Prompt for person name and add face reference
- **Quick Add Face**: Instantly add face reference with default name
- **Delete**: Remove files while viewing
- **Single-Line Layout**: All buttons display on one line
- **Consistent Design**: Matches Media and Review page button styling exactly
#### Dashboard (Recent Downloads) Lightbox Enhancement
- **Four Action Buttons**: Same comprehensive controls in Dashboard lightbox
- **Move to Review**: Move items to review queue
- **Add Reference**: Add face reference with person name prompt
- **Quick Add Face**: Instantly add face reference with default name
- **Delete**: Remove files directly
- **Better Organization**: Buttons above file info box
- **Layout Fix**: Buttons outside constrained container for proper single-line display
#### Add Reference Modal (Both Pages)
- **Person Name Input**: Enter custom person name for face reference
- **Default Name**: Pre-filled with "Eva Longoria" for quick entry
- **Keyboard Support**: Press Enter to submit
- **Cancel/Confirm**: Clear action buttons
---
### 📋 Technical Changes
**Frontend - Downloads.tsx**:
- Lines 3: Added Eye, Trash2 icon imports
- Lines 65-67: Added `showAddModal`, `addPersonName`, `actioningFile` state variables
- Lines 180-229: Added `addReferenceMutation`, `moveToReviewMutation`, `batchDeleteMutation`
- Lines 231-252: Added handler functions (`handleSingleMoveToReview`, `handleAddReference`, `handleDelete`, `confirmAddReference`)
- Lines 711-778: Replaced single button with four action buttons in lightbox
- Lines 783-831: Added Add Reference modal with person name input
**Frontend - Dashboard.tsx**:
- Lines 4-22: Added Eye, Trash2 icon imports
- Lines 84-86: Added `showAddModal`, `addPersonName`, `actioningFile` state variables
- Lines 137-177: Added `addReferenceMutation`, `moveToReviewMutation`, `batchDeleteMutation`
- Lines 253-278: Added handler functions (`handleSingleMoveToReview`, `handleAddReference`, `handleDelete`, `confirmAddReference`)
- Lines 739-776: Replaced single button with four action buttons in lightbox
- Lines 814-862: Added Add Reference modal with person name input
**Changes**:
- Buttons placed outside `max-w-2xl` container to prevent wrapping
- Consistent button layout: `flex items-center space-x-4` (no `justify-center`, no `flex-wrap`)
- All pages now have consistent lightbox UI (Media, Review, Downloads, Dashboard)
---
### 🔄 Breaking Changes
None.
---
## [6.22.0] - 2025-11-11
### ✨ Media Page UI Enhancement - Button Overlays & Lightbox Controls
This release enhances the Media page UI to match the Review page design, providing consistent button overlays on thumbnails and comprehensive action buttons in the lightbox viewer. Users can now Review, Add Reference, and Delete directly from thumbnails, plus Quick Add Face in the lightbox.
---
### 🎯 Key Features
#### Button Overlays on Thumbnails
- **Three Action Buttons**: Appear on hover, replacing the old icon overlays
- **Review**: Quickly move items to review queue
- **Add Reference**: Add face reference with person name prompt
- **Delete**: Remove files directly
- **Consistent Design**: Matches Review page button styling exactly
- **Full-Width Buttons**: Better touch targets and clearer labels
#### Enhanced Lightbox Controls
- **Four Action Buttons**: Comprehensive controls in lightbox viewer
- **Move to Review**: Move items while viewing
- **Add Reference**: Prompt for person name and add face reference
- **Quick Add Face**: Instantly add face reference with default name
- **Delete**: Remove files while viewing
- **Single-Line Layout**: All buttons display on one line
- **Better Organization**: Buttons above file info box
#### Add Reference Modal
- **Person Name Input**: Enter custom person name for face reference
- **Default Name**: Pre-filled with "Eva Longoria" for quick entry
- **Keyboard Support**: Press Enter to submit
- **Cancel/Confirm**: Clear action buttons
---
### 🐛 Bug Fixes
1. **Button Wrapping**: Fixed buttons wrapping to multiple lines
- Moved buttons outside constrained container
- Removed `justify-center` and `flex-wrap` classes
- Matched Review page layout structure exactly
---
### 📋 Technical Changes
**Frontend - Media.tsx**:
- Lines 80-82: Added `showAddModal`, `addPersonName`, `actioningFile` state variables
- Lines 209-226: Added `addReferenceMutation` for face reference handling with person name
- Lines 410-425: Added `handleAddReference`, `handleDelete`, `confirmAddReference` functions
- Lines 854-899: Replaced icon overlays with button overlays (Review, Add Reference, Delete)
- Lines 991-1024: Added four action buttons to lightbox (Move to Review, Add Reference, Quick Add Face, Delete)
- Lines 1107-1154: Added Add Reference modal with person name input and validation
**Changes**:
- Removed circular icon overlays (Eye and Trash2 icons)
- Replaced with full-width button overlays matching Review page
- Moved lightbox buttons outside `max-w-2xl` container for proper layout
- Added modal dialog for face reference person name input
---
### 🔄 Breaking Changes
None.
---
## [6.21.0] - 2025-11-11
### ✨ Review Queue, iOS Improvements, Enhanced Face Recognition & Maintenance
This release adds Review Queue functionality allowing users to move media items for review, significantly improves the mobile experience with iOS-optimized lightboxes and swipe gestures, enhances face recognition accuracy by re-enabling the CNN model with smart fallback, and fixes TikTok video downloads. Project maintenance includes cleanup of cache files, old logs, and better documentation organization.
---
### 🎯 Key Features
#### Review Queue Functionality
- **Move to Review**: Added Review button in three locations on Media page
- Thumbnail overlay button for quick single-item moves
- Batch operations bar for multi-item moves
- Lightbox viewer button for in-place review decisions
- **Backend Integration**: Files moved from `/opt/immich/md/` to `/opt/immich/review/`
- **Auto-Sync**: Automatically triggers Immich scan after move operations
- **Status Notifications**: Success/error feedback for all operations
#### iOS-Friendly Lightboxes
- **Swipe Gestures**: Navigate media with intuitive swipe gestures
- Swipe left/right: Previous/next item
- Swipe down: Close lightbox
- **Touch-Compliant Buttons**: All interactive elements meet iOS 44×44px minimum size
- **Safe Area Support**: Proper positioning around notches and rounded corners
- **Reusable Hook**: Created `useSwipeGestures.ts` hook used across 6 pages
#### Enhanced Face Recognition
- **CNN Model Re-enabled**: More accurate face detection with intelligent fallback
- HOG model first (fast, 100ms per image)
- CNN model if no faces found (accurate, 500ms per image)
- **Improved Video Sampling**: Increased from 3 to 5 frames per video
- **Expected Results**: 15-20% improvement in face detection accuracy
- **Optimized for SSD**: Takes advantage of MX500 upgrade (from BX500)
#### TikTok Download Fixes
- **Video Format**: Explicitly request best video+audio format with `--format best`
- **Photo Post Handling**: Detect and skip photo-only posts (download videos only)
- **Dependencies**: Installed curl-cffi for better TikTok API access
---
### 🐛 Bug Fixes
1. **TikTok Downloads**: Fixed downloads only getting audio/thumbnails instead of videos
- Added `--format best` to yt-dlp command (tiktok_module.py:269, 433)
- Installed curl-cffi dependency for better API access
2. **iOS Button Cutoff**: Fixed navigation buttons cut off on both sides on iOS
- Added responsive sizing (50px mobile, 60px desktop)
- Implemented iOS safe area insets: `env(safe-area-inset-left/right)`
3. **Face Recognition Accuracy**: Improved detection with CNN model and better sampling
- Re-enabled CNN model with HOG fallback (face_recognition_module.py:250-262)
- Increased video frame sampling from 3 to 5 frames (line 154)
---
### 🧹 Maintenance & Cleanup
- **Python Cache**: Removed 10,373 cache files (__pycache__ directories and .pyc files)
- **Log Cleanup**: Deleted 242 log files older than 30 days (saved 14MB, 127MB → 113MB)
- **Documentation**: Moved REPOST_DETECTION_QUICKSTART.md to docs/ directory
- **Code Quality**: Verified no syntax errors across all Python files
- **Build Verification**: Frontend builds successfully with no errors
---
### 📋 Technical Changes
**Frontend**:
- `web/frontend/src/lib/api.ts:625-633` - Added moveToReview API function
- `web/frontend/src/pages/Media.tsx:205-220` - Added Review mutation with Immich sync
- `web/frontend/src/lib/useSwipeGestures.ts` - Created reusable swipe gesture hook
- `web/frontend/src/pages/*.tsx` - Updated 6 pages with iOS-friendly lightboxes
- Media.tsx, Review.tsx, RecycleBin.tsx
- Dashboard.tsx, Downloads.tsx, Notifications.tsx
**Backend**:
- `web/backend/api.py:3131-3187` - Added /api/media/move-to-review endpoint
- Validates file paths are within media_base
- Preserves directory structure in review queue
- Returns detailed success/error information
- Broadcasts WebSocket event on completion
**Modules**:
- `modules/tiktok_module.py:269,433` - Added --format best for video downloads
- `modules/tiktok_module.py:429-436` - Added photo post detection and skipping
- `modules/face_recognition_module.py:250-262` - Re-enabled CNN with HOG fallback
- `modules/face_recognition_module.py:154` - Increased video sampling to 5 frames
---
### 🔄 Breaking Changes
None.
---
### 📦 Dependencies
- Installed: curl-cffi (for improved TikTok downloads)
---
## [6.20.2] - 2025-11-09
### 🔧 CRITICAL FIX: Authentication Cookie Support Missing
**PROBLEM**: All POST/PUT/DELETE requests failing with 403 Forbidden errors, even after implementing CSRF fixes. Users could not delete files, update preferences, or perform any mutations.
**ROOT CAUSE**: The `get_current_user()` authentication dependency only checked Authorization headers, completely ignoring the `auth_token` cookie. When users logged in, the backend set the auth_token cookie (v6.19.0), but ALL protected endpoints failed to read it.
**IMPACT**:
- ❌ Review page Keep/Delete buttons non-functional (403 Forbidden)
- ❌ Batch delete operations failing (403 Forbidden)
- ❌ User preferences updates failing (403 Forbidden)
- ❌ All POST/PUT/DELETE endpoints blocked (403 Forbidden)
- ❌ Only endpoints using custom auth (like `get_current_user_media`) worked
---
### 🎯 The Actual Problem
**What We Thought**: CSRF protection or missing `credentials: 'include'` in frontend
**What It Actually Was**: Authentication function ignoring cookies entirely
**Evidence**:
```python
# BEFORE (broken) - api.py:147-160
async def get_current_user(
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security)
) -> Dict:
if not credentials:
raise HTTPException(status_code=401, detail="Not authenticated")
token = credentials.credentials # Only checks Authorization header
payload = app_state.auth.verify_session(token)
...
```
This meant:
1. Login sets `auth_token` cookie ✅
2. Frontend sends cookie with requests ✅
3. **Backend ignores the cookie completely** ❌
4. Backend returns 403 Forbidden ❌
---
### ✅ The Fix
**Modified `get_current_user()` to support BOTH auth methods** (api.py:147-169):
```python
# AFTER (working)
async def get_current_user(
request: Request, # NEW: Need request to access cookies
credentials: Optional[HTTPAuthorizationCredentials] = Depends(security)
) -> Dict:
auth_token = None
# Try Authorization header first (preferred)
if credentials:
auth_token = credentials.credentials
# Try cookie second (backward compatible + iOS support)
elif 'auth_token' in request.cookies:
auth_token = request.cookies.get('auth_token')
if not auth_token:
raise HTTPException(status_code=401, detail="Not authenticated")
payload = app_state.auth.verify_session(auth_token)
...
```
**This fix makes authentication work with**:
- ✅ Authorization header (Bearer token)
- ✅ Cookie-based auth (existing sessions)
- ✅ Old frontend code (cached in browser)
- ✅ New frontend code (with credentials: 'include')
- ✅ iOS/Mobile apps using cookies
- ✅ Desktop browsers with any cache state
---
### 🔄 CSRF Protection Status
**Temporarily disabled** during troubleshooting to isolate the authentication issue.
**Re-enable when**:
1. Users clear browser cache to get new frontend code
2. Or after sufficient time for natural cache expiration
3. Frontend code with `credentials: 'include'` is loaded
**To re-enable**: Uncomment lines 367-382 in `web/backend/api.py`
---
### 📋 Technical Details
**File Modified**:
- `web/backend/api.py:147-169` - Fixed `get_current_user()` function
**Change Summary**:
- Added `request: Request` parameter to access cookies
- Check Authorization header first (preferred method)
- Fall back to `auth_token` cookie (backward compatible)
- Works with ANY frontend version (old or new)
**Authentication Priority**:
1. Authorization header (Bearer token) - Most secure
2. Cookie (`auth_token`) - Backward compatible, iOS support
**Affected Endpoints** (all now working):
- ✅ `/api/review/keep` - Review keep
- ✅ `/api/review/delete` - Review delete (batch operations)
- ✅ `/api/auth/preferences` - User preferences
- ✅ `/api/downloads/{id}` - Delete downloads
- ✅ `/api/media/batch-move` - Batch file operations
- ✅ `/api/media/batch-download` - Batch downloads
- ✅ ALL POST/PUT/DELETE endpoints
---
### 🧪 Testing
**Test Case 1: Cookie-based Auth**
```bash
# User logged in, has auth_token cookie
curl -X DELETE https://md.lic.ad/api/review/delete \
-H "Content-Type: application/json" \
--cookie "auth_token=