media-downloader/docs/FACE_RECOGNITION.md

# Face Recognition System

**Version:** 6.5.1
**Status:** Production Ready
**Last Updated:** 2025-11-01

## Overview

The Media Downloader now includes an automated face recognition system that analyzes downloaded media (images and videos) and routes them based on whether they match reference faces in the database.

## Features

- **Automatic Face Detection**: Scans all downloaded images and videos
- **Video Support**: Extracts frames from videos for face analysis
- **Smart Routing**: Matched media → final destination, unmatched → review queue
- **Web UI Review Queue**: Manual review interface with batch operations
- **Reference Training**: Build face database from known good images
- **Configurable Tolerance**: Adjustable matching sensitivity

## Architecture

### Components

1. **Face Recognition Module** (`modules/face_recognition_module.py`)
   - Face detection using `face_recognition` library (dlib HOG model)
   - Face encoding (128-dimensional vectors)
   - Reference face database management
   - Video frame extraction via ffmpeg

2. **Move Module Integration** (`modules/move_module.py`)
   - Integrated into file move workflow
   - Checks after duplicate detection
   - Routes to review queue on no-match

3. **Review API** (`web/backend/api.py`)
   - `/api/review/list` - List review queue
   - `/api/review/keep` - Move to destination
   - `/api/review/delete` - Delete from queue
   - `/api/review/add-reference` - Add as reference face

4. **Review UI** (`web/frontend/src/pages/Review.tsx`)
   - Gallery view of unmatched media
   - Single-file and batch operations
   - Lightbox preview
   - Action buttons: Keep, Add Reference, Delete

### Database Schema

```sql
CREATE TABLE face_recognition_references (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    person_name TEXT NOT NULL,
    encoding_data TEXT NOT NULL,  -- Base64 encoded pickle of numpy array
    reference_image_path TEXT,
    is_active INTEGER DEFAULT 1,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE INDEX idx_face_ref_person ON face_recognition_references(person_name, is_active);
```

## Workflow

```
┌─────────────────────┐
│  Download Media     │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│  Calculate Hash     │ ──── Duplicate? ──→ Skip
└──────────┬──────────┘
           │ New file
           ▼
┌─────────────────────┐
│  Detect Faces       │
│  (Image: direct)    │
│  (Video: extract    │
│   frame @ 1s)       │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│  Match Against      │
│  Reference Faces    │
│  (tolerance: 0.6)   │
└──────────┬──────────┘
           │
           ├─── Match (>60%) ──→ Move to Final Destination
           │
           └─── No Match ─────→ Move to /opt/immich/review
                                         │
                                         ▼
                                ┌─────────────────────┐
                                │  Review Queue (UI)  │
                                │  - Keep             │
                                │  - Add Reference    │
                                │  - Delete           │
                                └─────────────────────┘
```

## Configuration

### Settings (Web UI)

Face recognition settings can be configured via the **Configuration → Downloads** page in the web UI:

1. Navigate to http://your-server:5173/configuration
2. Click the **Downloads** tab
3. Scroll to the **Face Recognition** section
4. Configure settings:
   - **Enabled**: Toggle face recognition on/off
   - **Person Name**: Name used for matching reference faces (e.g., "Eva Longoria")
   - **Tolerance**: Match sensitivity 0.0-1.0 (default: 0.6, lower = stricter)
   - **Review Queue Path**: Directory for unmatched media (default: /opt/immich/review)
5. Click **Save Download Settings**

### Settings (Database)

Settings are stored in the database `settings` table:

```json
{
  "enabled": true,
  "tolerance": 0.6,
  "person_name": "Eva Longoria",
  "review_path": "/opt/immich/review"
}
```

**Parameters:**
- `enabled` (boolean): Enable/disable face recognition
- `tolerance` (float 0.0-1.0): Lower = stricter matching (default: 0.6)
- `person_name` (string): Default person name for references
- `review_path` (string): Directory for unmatched media

**Direct Database Access:**
```bash
sqlite3 /opt/media-downloader/database/media_downloader.db "SELECT key, value FROM settings WHERE key = 'face_recognition'"
```

### Supported Formats

**Images:**
- .jpg, .jpeg, .png, .gif, .bmp, .webp, .heic

**Videos:**
- .mp4, .mov, .avi, .mkv, .webm, .flv, .m4v

## Usage

### Training Reference Faces

Add reference faces from known good images:

```bash
/opt/media-downloader/venv/bin/python3 /opt/media-downloader/scripts/add_reference_face.py "Person Name" "/path/to/image.jpg"
```

**Best Practices:**
- Use 5-10 reference images per person
- Include variety: different angles, lighting, expressions
- Use high-quality, clear face images
- Avoid group photos (will use first detected face)

### Testing Face Recognition

Test an image/video against reference database:

```bash
/opt/media-downloader/venv/bin/python3 /opt/media-downloader/scripts/test_face_recognition.py "/path/to/test.jpg" [tolerance]
```

### Managing Review Queue

**Via Web UI:**
1. Navigate to `/review` page
2. View unmatched media in gallery
3. For each item or batch selection:
   - **Keep**: Move to destination without adding as reference
   - **Add Reference**: Add face to database + move to destination
   - **Delete**: Remove from review queue

**Via CLI:**
```bash
# List review queue
ls -lh /opt/immich/review/

# Move to final destination manually
mv /opt/immich/review/file.jpg /opt/immich/md/destination/

# Add as reference then move
venv/bin/python3 scripts/add_reference_face.py "Name" "/opt/immich/review/file.jpg"
mv /opt/immich/review/file.jpg /opt/immich/md/destination/
```

### Batch Operations

**Web UI:**
1. Click "Select Multiple" button
2. Click images to select (blue ring + checkbox)
3. Use "Select All" for all items
4. Choose batch action:
   - **Keep Selected** - Bulk move to destination
   - **Add as Reference** - Bulk add faces + move
   - **Delete Selected** - Bulk delete

## Performance

### Speed
- **Image Detection**: ~0.5-2s per image (HOG model)
- **Video Detection**: ~2-5s per video (frame extraction + detection)
- **Matching**: <0.1s per face against all references

### Accuracy
- **Same Person, Same Conditions**: 90-100% confidence
- **Same Person, Different Conditions**: 50-80% confidence
- **Different Person**: <40% confidence
- **Threshold**: 60% (tolerance: 0.6)

### Resource Usage
- **CPU**: Moderate (HOG model is CPU-based)
- **Memory**: ~200MB additional for face_recognition library
- **Disk**: Minimal (encodings are ~1KB each)
- **Temp Files**: Video frames auto-deleted after processing

## Troubleshooting

### No Faces Detected

**Causes:**
- Face too small in image/video
- Face obscured or at extreme angle
- Poor image quality

**Solutions:**
- Use higher quality source images
- For videos, try different timestamp (currently fixed at 1s)
- Check image isn't corrupted: `file /path/to/image.jpg`

### Low Confidence Matches

**Causes:**
- Insufficient reference faces
- References don't match current conditions (age, lighting, angle)
- Tolerance too strict

**Solutions:**
- Add more reference faces (5-10 recommended)
- Add references from similar conditions to target media
- Increase tolerance in settings (0.6 → 0.65)

### False Positives

**Causes:**
- Tolerance too loose
- Similar-looking people
- Insufficient reference diversity

**Solutions:**
- Decrease tolerance (0.6 → 0.55)
- Add negative examples to recognize differences
- Review reference faces for quality

### Video Frame Extraction Fails

**Causes:**
- ffmpeg not installed
- Video codec not supported
- Video shorter than 1 second

**Solutions:**
```bash
# Check ffmpeg
which ffmpeg

# Test frame extraction manually
ffmpeg -ss 1 -i video.mp4 -frames:v 1 test_frame.jpg

# Check video duration
ffmpeg -i video.mp4 2>&1 | grep Duration
```

## API Reference

### Face Recognition Module

```python
from modules.face_recognition_module import FaceRecognitionModule
from modules.unified_database import UnifiedDatabase

# Initialize
db = UnifiedDatabase()
face_module = FaceRecognitionModule(unified_db=db)

# Add reference face
face_module.add_reference_face("Person Name", "/path/to/image.jpg")

# Check image
result = face_module.check_image("/path/to/test.jpg", tolerance=0.6, is_video=False)
# Returns: {'has_match': bool, 'person_name': str, 'confidence': float, 'face_count': int, 'faces': list}

# Check video
result = face_module.check_image("/path/to/video.mp4", tolerance=0.6, is_video=True)

# Get reference faces
refs = face_module.get_reference_faces()
```

### Review API Endpoints

```javascript
// List review queue
GET /api/review/list?limit=50&offset=0

// Keep image (move to destination)
POST /api/review/keep
Body: { file_path: "/opt/immich/review/file.jpg", destination: "social media/instagram/posts" }

// Delete from review queue
DELETE /api/review/delete
Body: { file_path: "/opt/immich/review/file.jpg" }

// Add as reference + move
POST /api/review/add-reference
Body: {
  file_path: "/opt/immich/review/file.jpg",
  person_name: "Person Name",
  destination: "social media/instagram/posts"
}
```

## Maintenance

### Regular Tasks

1. **Review Queue Cleanup** (Weekly)
   - Process items in /opt/immich/review
   - Keep: items that should have matched
   - Delete: irrelevant items
   - Add Reference: good quality faces to improve matching

2. **Reference Database Audit** (Monthly)
   - Remove poor quality references
   - Add new references from recent media
   - Check reference count per person

3. **Performance Monitoring**
   - Check review queue size: `ls /opt/immich/review | wc -l`
   - Monitor match rate in logs
   - Adjust tolerance if needed

### Database Queries

```sql
-- Count active references by person
SELECT person_name, COUNT(*) as count
FROM face_recognition_references
WHERE is_active = 1
GROUP BY person_name;

-- View recent references
SELECT person_name, reference_image_path, created_at
FROM face_recognition_references
WHERE is_active = 1
ORDER BY created_at DESC
LIMIT 10;

-- Disable a reference
UPDATE face_recognition_references
SET is_active = 0
WHERE id = ?;
```

## Security & Privacy

- **Face Encodings**: Stored as 128-dimensional vectors (not original images)
- **Local Processing**: All face detection happens locally, no cloud services
- **Access Control**: Review queue API requires authentication
- **Data Retention**: Reference faces kept indefinitely until manually removed
- **Audit Trail**: created_at/updated_at timestamps track reference changes

## Dependencies

- **face_recognition** (1.3.0): Face detection and recognition
- **dlib** (20.0.0): Machine learning toolkit (face detection models)
- **numpy** (2.3.4): Numerical computing (face encoding vectors)
- **ffmpeg**: Video frame extraction (system package)

## Future Enhancements

- [ ] Multi-person recognition (tag all people in image)
- [ ] Confidence threshold per person
- [ ] Face clustering for unknown faces
- [ ] GPU acceleration (dlib CNN model)
- [ ] Multiple frame extraction for videos
- [ ] Face detection quality scoring
- [ ] Auto-training from high-confidence matches
- [ ] REST API for external integrations

## Version History

**6.5.1** (2025-11-01)
- Added face recognition settings to Configuration page (Web UI)
- Settings now editable via Configuration → Downloads tab
- Real-time settings updates without editing database directly

**6.5.0** (2025-10-31)
- Initial face recognition implementation
- Image and video support
- Review queue with batch operations
- Reference face training
- Web UI integration