# Face Recognition - Immich Integration Plan

**Created**: 2025-10-31
**Status**: Planning Phase - Immich Integration Approach
**Target Version**: 6.5.0

---

## 🎯 Overview

**NEW APPROACH**: Instead of building face recognition from scratch, integrate with Immich's existing face recognition system. Immich already processes faces, we just need to read its data and use it for auto-sorting.

---

## 💡 Why Use Immich's Face Data?

### Advantages
✅ **Already processed** - Immich has already detected faces in your photos
✅ **No duplicate processing** - Don't waste CPU doing the same work twice
✅ **Consistent** - Same face recognition across Immich and Media Downloader
✅ **Centralized management** - Manage people in one place (Immich UI)
✅ **Better accuracy** - Immich uses machine learning models that improve over time
✅ **GPU accelerated** - Immich can use GPU for faster processing
✅ **No new dependencies** - Don't need to install face_recognition library

### Architecture
```
Downloads → Immich Scan → Immich Face Recognition → Media Downloader Reads Data
                                                              ↓
                                                    Auto-Sort by Person Name
```

---

## 🗄️ Immich Database Structure

### Understanding Immich's Face Tables

Immich stores face data in PostgreSQL database. Key tables:

#### 1. `person` table
Stores information about identified people:
```sql
SELECT * FROM person;

Columns:
- id (uuid)
- name (text) - Person's name
- thumbnailPath (text)
- isHidden (boolean)
- birthDate (date)
- createdAt, updatedAt
```

#### 2. `asset_faces` table
Links faces to assets (photos):
```sql
SELECT * FROM asset_faces;

Columns:
- id (uuid)
- assetId (uuid) - References the photo
- personId (uuid) - References the person (if identified)
- embedding (vector) - Face encoding data
- imageWidth, imageHeight
- boundingBoxX1, boundingBoxY1, boundingBoxX2, boundingBoxY2
```

#### 3. `assets` table
Photo metadata:
```sql
SELECT * FROM assets;

Columns:
- id (uuid)
- originalPath (text) - File path on disk
- originalFileName (text)
- type (enum) - IMAGE, VIDEO
- ownerId (uuid)
- libraryId (uuid)
- checksum (bytea) - File hash
```

### Key Relationships
```
assets (photos)
  ↓ (1 photo can have many faces)
asset_faces (detected faces)
  ↓ (each face can be linked to a person)
person (identified people)
```

---

## 🔌 Integration Architecture

### High-Level Flow

```
┌──────────────────────┐
│  1. Image Downloaded │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  2. Immich Scans     │ ◄── Existing Immich process
│     (Auto/Manual)     │     Detects faces, creates embeddings
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  3. User Identifies  │ ◄── Done in Immich UI
│     Faces (Immich)   │     Assigns names to faces
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│ 4. Media Downloader  │ ◄── NEW: Our integration
│    Reads Immich DB   │     Query PostgreSQL
└──────────┬───────────┘
           │
           ├─── Person identified? ──► Auto-sort to /faces/{person_name}/
           │
           └─── Not identified ──────► Leave in original location
```

### Implementation Options

#### Option A: Direct Database Integration (Recommended)
**Read Immich's PostgreSQL database directly**

Pros:
- Real-time access to face data
- No API dependencies
- Fast queries
- Can join tables for complex queries

Cons:
- Couples to Immich's database schema (may break on updates)
- Requires PostgreSQL connection

#### Option B: Immich API Integration
**Use Immich's REST API**

Pros:
- Stable interface (less likely to break)
- Official supported method
- Can work with remote Immich instances

Cons:
- Slower (HTTP overhead)
- May require multiple API calls
- Need to handle API authentication

**Recommendation**: Start with **Option A** (direct database), add Option B later if needed.

---

## 💾 Database Integration Implementation

### Step 1: Connect to Immich PostgreSQL

```python
import psycopg2
from psycopg2.extras import RealDictCursor

class ImmichFaceDB:
    """Read face recognition data from Immich database"""

    def __init__(self, config):
        self.config = config
        self.conn = None

        # Immich DB connection details
        self.db_config = {
            'host': config.get('immich', {}).get('db_host', 'localhost'),
            'port': config.get('immich', {}).get('db_port', 5432),
            'database': config.get('immich', {}).get('db_name', 'immich'),
            'user': config.get('immich', {}).get('db_user', 'postgres'),
            'password': config.get('immich', {}).get('db_password', '')
        }

    def connect(self):
        """Connect to Immich database"""
        try:
            self.conn = psycopg2.connect(**self.db_config)
            return True
        except Exception as e:
            logging.error(f"Failed to connect to Immich DB: {e}")
            return False

    def get_faces_for_file(self, file_path: str) -> list:
        """
        Get all identified faces for a specific file

        Args:
            file_path: Full path to the image file

        Returns:
            list of dicts: [{
                'person_id': str,
                'person_name': str,
                'confidence': float,
                'bounding_box': dict
            }]
        """
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                # Query to get faces and their identified people
                query = """
                    SELECT
                        p.id as person_id,
                        p.name as person_name,
                        af.id as face_id,
                        af."boundingBoxX1" as bbox_x1,
                        af."boundingBoxY1" as bbox_y1,
                        af."boundingBoxX2" as bbox_x2,
                        af."boundingBoxY2" as bbox_y2,
                        a."originalPath" as file_path,
                        a."originalFileName" as filename
                    FROM assets a
                    JOIN asset_faces af ON a.id = af."assetId"
                    LEFT JOIN person p ON af."personId" = p.id
                    WHERE a."originalPath" = %s
                        AND a.type = 'IMAGE'
                        AND p.name IS NOT NULL  -- Only identified faces
                        AND p."isHidden" = false
                """

                cursor.execute(query, (file_path,))
                results = cursor.fetchall()

                faces = []
                for row in results:
                    faces.append({
                        'person_id': str(row['person_id']),
                        'person_name': row['person_name'],
                        'bounding_box': {
                            'x1': row['bbox_x1'],
                            'y1': row['bbox_y1'],
                            'x2': row['bbox_x2'],
                            'y2': row['bbox_y2']
                        }
                    })

                return faces

        except Exception as e:
            logging.error(f"Error querying faces for {file_path}: {e}")
            return []

    def get_all_people(self) -> list:
        """Get list of all identified people in Immich"""
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                query = """
                    SELECT
                        id,
                        name,
                        "thumbnailPath",
                        "createdAt",
                        (SELECT COUNT(*) FROM asset_faces WHERE "personId" = person.id) as face_count
                    FROM person
                    WHERE name IS NOT NULL
                        AND "isHidden" = false
                    ORDER BY name
                """

                cursor.execute(query)
                return cursor.fetchall()

        except Exception as e:
            logging.error(f"Error getting people list: {e}")
            return []

    def get_unidentified_faces(self, limit=100) -> list:
        """
        Get faces that haven't been identified yet

        Returns:
            list of dicts with file_path, face_id, bounding_box
        """
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                query = """
                    SELECT
                        a."originalPath" as file_path,
                        a."originalFileName" as filename,
                        af.id as face_id,
                        af."boundingBoxX1" as bbox_x1,
                        af."boundingBoxY1" as bbox_y1,
                        af."boundingBoxX2" as bbox_x2,
                        af."boundingBoxY2" as bbox_y2,
                        a."createdAt" as created_at
                    FROM asset_faces af
                    JOIN assets a ON af."assetId" = a.id
                    WHERE af."personId" IS NULL
                        AND a.type = 'IMAGE'
                    ORDER BY a."createdAt" DESC
                    LIMIT %s
                """

                cursor.execute(query, (limit,))
                return cursor.fetchall()

        except Exception as e:
            logging.error(f"Error getting unidentified faces: {e}")
            return []

    def close(self):
        """Close database connection"""
        if self.conn:
            self.conn.close()
```

---

## 🔄 Auto-Sort Implementation

### Core Auto-Sort Module

```python
#!/usr/bin/env python3
"""
Immich Face-Based Auto-Sorter
Reads face data from Immich and sorts images by person
"""

import os
import shutil
import logging
from pathlib import Path
from datetime import datetime

logger = logging.getLogger(__name__)


class ImmichFaceSorter:
    """Auto-sort images based on Immich face recognition"""

    def __init__(self, config, immich_db):
        self.config = config
        self.immich_db = immich_db

        # Configuration
        self.enabled = config.get('face_sorting', {}).get('enabled', False)
        self.base_dir = config.get('face_sorting', {}).get('base_directory',
                                                            '/mnt/storage/Downloads/faces')
        self.min_faces_to_sort = config.get('face_sorting', {}).get('min_faces_to_sort', 1)
        self.single_person_only = config.get('face_sorting', {}).get('single_person_only', True)
        self.move_or_copy = config.get('face_sorting', {}).get('move_or_copy', 'copy')  # 'move' or 'copy'

        # Create base directory
        os.makedirs(self.base_dir, exist_ok=True)

    def process_downloaded_file(self, file_path: str) -> dict:
        """
        Process a newly downloaded file

        Args:
            file_path: Full path to the downloaded image

        Returns:
            dict: {
                'status': 'success'|'skipped'|'error',
                'action': 'sorted'|'copied'|'skipped',
                'person_name': str or None,
                'faces_found': int,
                'message': str
            }
        """
        if not self.enabled:
            return {'status': 'skipped', 'message': 'Face sorting disabled'}

        if not os.path.exists(file_path):
            return {'status': 'error', 'message': 'File not found'}

        # Only process images
        ext = os.path.splitext(file_path)[1].lower()
        if ext not in ['.jpg', '.jpeg', '.png', '.heic', '.heif']:
            return {'status': 'skipped', 'message': 'Not an image file'}

        # Wait for Immich to process (if needed)
        # This could be a configurable delay or check if file is in Immich DB
        import time
        time.sleep(2)  # Give Immich time to scan new file

        # Get faces from Immich
        faces = self.immich_db.get_faces_for_file(file_path)

        if not faces:
            logger.debug(f"No identified faces in {file_path}")
            return {
                'status': 'skipped',
                'action': 'skipped',
                'faces_found': 0,
                'message': 'No identified faces found'
            }

        # Handle multiple faces
        if len(faces) > 1 and self.single_person_only:
            logger.info(f"Multiple faces ({len(faces)}) in {file_path}, skipping")
            return {
                'status': 'skipped',
                'action': 'skipped',
                'faces_found': len(faces),
                'message': f'Multiple faces found ({len(faces)}), single_person_only=true'
            }

        # Sort to first person's directory (or implement multi-person logic)
        primary_face = faces[0]
        person_name = primary_face['person_name']

        return self._sort_to_person(file_path, person_name, len(faces))

    def _sort_to_person(self, file_path: str, person_name: str, faces_count: int) -> dict:
        """Move or copy file to person's directory"""

        # Create person directory (sanitize name)
        person_dir_name = self._sanitize_directory_name(person_name)
        person_dir = os.path.join(self.base_dir, person_dir_name)
        os.makedirs(person_dir, exist_ok=True)

        # Determine target path
        filename = os.path.basename(file_path)
        target_path = os.path.join(person_dir, filename)

        # Handle duplicates
        if os.path.exists(target_path):
            base, ext = os.path.splitext(filename)
            timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
            filename = f"{base}_{timestamp}{ext}"
            target_path = os.path.join(person_dir, filename)

        try:
            # Move or copy
            if self.move_or_copy == 'move':
                shutil.move(file_path, target_path)
                action = 'sorted'
                logger.info(f"Moved {filename} to {person_name}/")
            else:  # copy
                shutil.copy2(file_path, target_path)
                action = 'copied'
                logger.info(f"Copied {filename} to {person_name}/")

            return {
                'status': 'success',
                'action': action,
                'person_name': person_name,
                'faces_found': faces_count,
                'target_path': target_path,
                'message': f'{"Moved" if action == "sorted" else "Copied"} to {person_name}/'
            }

        except Exception as e:
            logger.error(f"Error sorting {file_path}: {e}")
            return {'status': 'error', 'message': str(e)}

    def _sanitize_directory_name(self, name: str) -> str:
        """Convert person name to safe directory name"""
        # Replace spaces with underscores, remove special chars
        import re
        safe_name = re.sub(r'[^\w\s-]', '', name)
        safe_name = re.sub(r'[-\s]+', '_', safe_name)
        return safe_name.lower()

    def batch_sort_existing(self, source_dir: str = None, limit: int = None) -> dict:
        """
        Batch sort existing files that are already in Immich

        Args:
            source_dir: Directory to process (None = all Immich files)
            limit: Max files to process (None = all)

        Returns:
            dict: Statistics of operation
        """
        stats = {
            'processed': 0,
            'sorted': 0,
            'skipped': 0,
            'errors': 0
        }

        # Query Immich for all files with identified faces
        # This would require additional query method in ImmichFaceDB

        logger.info(f"Batch sorting from {source_dir or 'all Immich files'}")

        # Implementation here...

        return stats
```

---

## ⚙️ Configuration

### Add to `config.json`:

```json
{
  "immich": {
    "enabled": true,
    "url": "http://localhost:2283",
    "api_key": "your-immich-api-key",
    "db_host": "localhost",
    "db_port": 5432,
    "db_name": "immich",
    "db_user": "postgres",
    "db_password": "your-postgres-password"
  },
  "face_sorting": {
    "enabled": true,
    "base_directory": "/mnt/storage/Downloads/faces",
    "min_faces_to_sort": 1,
    "single_person_only": true,
    "move_or_copy": "copy",
    "process_delay_seconds": 5,
    "sync_with_immich_scan": true,
    "create_person_subdirs": true,
    "handle_multiple_faces": "skip"
  }
}
```

---

## 🔄 Integration Points

### 1. Post-Download Hook

Add face sorting after download completes:

```python
def on_download_complete(file_path: str, download_id: int):
    """Called when download completes"""

    # Existing tasks
    update_database(download_id)
    send_notification(download_id)

    # Trigger Immich scan (if not automatic)
    if config.get('immich', {}).get('trigger_scan', True):
        trigger_immich_library_scan()

    # Wait for Immich to process
    delay = config.get('face_sorting', {}).get('process_delay_seconds', 5)
    time.sleep(delay)

    # Sort by faces
    if config.get('face_sorting', {}).get('enabled', False):
        immich_db = ImmichFaceDB(config)
        sorter = ImmichFaceSorter(config, immich_db)
        result = sorter.process_downloaded_file(file_path)
        logger.info(f"Face sort result: {result}")
        immich_db.close()
```

### 2. Trigger Immich Library Scan

```python
def trigger_immich_library_scan():
    """Trigger Immich to scan for new files"""
    import requests

    immich_url = config.get('immich', {}).get('url')
    api_key = config.get('immich', {}).get('api_key')

    if not immich_url or not api_key:
        return

    try:
        response = requests.post(
            f"{immich_url}/api/library/scan",
            headers={'x-api-key': api_key}
        )
        if response.status_code == 201:
            logger.info("Triggered Immich library scan")
        else:
            logger.warning(f"Immich scan trigger failed: {response.status_code}")
    except Exception as e:
        logger.error(f"Error triggering Immich scan: {e}")
```

---

## 📊 Database Schema (Simplified)

Since we're reading from Immich, we only need minimal tracking:

```sql
-- Track what we've sorted
CREATE TABLE face_sort_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    download_id INTEGER,
    original_path TEXT NOT NULL,
    sorted_path TEXT NOT NULL,
    person_name TEXT NOT NULL,
    person_id TEXT,  -- Immich person UUID
    faces_count INTEGER DEFAULT 1,
    action TEXT,  -- 'moved' or 'copied'
    sorted_at TEXT,
    FOREIGN KEY (download_id) REFERENCES downloads(id)
);

CREATE INDEX idx_face_sort_person ON face_sort_history(person_name);
CREATE INDEX idx_face_sort_date ON face_sort_history(sorted_at);
```

---

## 🎨 Web UI (Simplified)

### Dashboard Page

```
┌─────────────────────────────────────────────┐
│ Face-Based Sorting (Powered by Immich)     │
├─────────────────────────────────────────────┤
│                                             │
│ Status: [✓ Enabled] [⚙️ Configure]         │
│                                             │
│ Connected to Immich: ✓                     │
│ People in Immich: 12                        │
│ Images Sorted: 145                          │
│                                             │
│ ┌───────────────────────────────────────┐  │
│ │ Recent Activity                       │  │
│ │                                       │  │
│ │ • 14:23 - Sorted to "John" (3 images)│  │
│ │ • 14:20 - Sorted to "Sarah" (1 image)│  │
│ │ • 14:18 - Skipped (multiple faces)   │  │
│ └───────────────────────────────────────┘  │
│                                             │
│ [View People] [Sort History] [Settings]    │
│                                             │
│ 💡 Manage people and faces in Immich UI    │
└─────────────────────────────────────────────┘
```

### People List (Read from Immich)

```
┌─────────────────────────────────────────────┐
│ People (from Immich)                        │
├─────────────────────────────────────────────┤
│                                             │
│ 👤 John Doe                                 │
│    Faces in Immich: 25                      │
│    Sorted by us: 42 images                  │
│    Directory: /faces/john_doe/              │
│    [View in Immich]                         │
│                                             │
│ 👤 Sarah Smith                              │
│    Faces in Immich: 18                      │
│    Sorted by us: 28 images                  │
│    Directory: /faces/sarah_smith/           │
│    [View in Immich]                         │
│                                             │
│ 💡 Add/edit people in Immich interface      │
└─────────────────────────────────────────────┘
```

---

## 🚀 Implementation Phases

### Phase 1: Basic Integration (Week 1)
- [ ] Install psycopg2 (PostgreSQL client)
- [ ] Create ImmichFaceDB class
- [ ] Test connection to Immich database
- [ ] Query faces for a test file
- [ ] List all people from Immich

### Phase 2: Auto-Sort Logic (Week 2)
- [ ] Create ImmichFaceSorter class
- [ ] Implement single-person sorting
- [ ] Handle move vs copy logic
- [ ] Add post-download hook integration
- [ ] Test with new downloads

### Phase 3: Configuration & Control (Week 3)
- [ ] Add configuration options
- [ ] Create enable/disable mechanism
- [ ] Add delay/timing controls
- [ ] Implement error handling
- [ ] Add logging

### Phase 4: Web UI (Week 4)
- [ ] Dashboard page (stats, enable/disable)
- [ ] People list (read from Immich)
- [ ] Sort history page
- [ ] Configuration interface

### Phase 5: Advanced Features (Week 5)
- [ ] Multi-face handling options
- [ ] Batch sort existing files
- [ ] Immich API integration (fallback)
- [ ] Statistics and reporting

### Phase 6: Polish (Week 6)
- [ ] Performance optimization
- [ ] Documentation
- [ ] Testing
- [ ] Error recovery

---

## 📝 API Endpoints (New)

```python
# Face Sorting Status
GET  /api/face-sort/status
POST /api/face-sort/enable
POST /api/face-sort/disable

# People (Read from Immich)
GET  /api/face-sort/people          # List people from Immich
GET  /api/face-sort/people/{id}     # Get person details

# History
GET  /api/face-sort/history         # Our sorting history
GET  /api/face-sort/stats           # Statistics

# Operations
POST /api/face-sort/batch           # Batch sort existing files
GET  /api/face-sort/batch/status    # Check batch progress

# Immich Connection
GET  /api/face-sort/immich/status   # Test Immich connection
POST /api/face-sort/immich/scan     # Trigger Immich library scan
```

---

## 🔧 Installation & Setup

### Step 1: Install PostgreSQL Client

```bash
pip3 install psycopg2-binary
```

### Step 2: Get Immich Database Credentials

```bash
# If Immich is running in Docker
docker exec -it immich_postgres env | grep POSTGRES

# Get credentials from Immich's docker-compose.yml or .env file
```

### Step 3: Test Connection

```python
import psycopg2

try:
    conn = psycopg2.connect(
        host="localhost",
        port=5432,
        database="immich",
        user="postgres",
        password="your-password"
    )
    print("✓ Connected to Immich database!")
    conn.close()
except Exception as e:
    print(f"✗ Connection failed: {e}")
```

### Step 4: Configure

Add Immich settings to `config.json`:

```json
{
  "immich": {
    "db_host": "localhost",
    "db_port": 5432,
    "db_name": "immich",
    "db_user": "postgres",
    "db_password": "your-password"
  },
  "face_sorting": {
    "enabled": true,
    "base_directory": "/mnt/storage/Downloads/faces"
  }
}
```

---

## ⚡ Performance Considerations

### Efficiency Gains
- **No duplicate processing** - Immich already did the heavy lifting
- **Fast queries** - Direct database access (milliseconds)
- **No ML overhead** - No face detection/recognition on our end
- **Scalable** - Works with thousands of photos

### Timing
- Database query: ~10-50ms per file
- File operation (move/copy): ~100-500ms
- Total per image: <1 second

---

## 🔒 Security Considerations

1. **Database Access** - Store PostgreSQL credentials securely
2. **Read-Only** - Only read from Immich DB, never write
3. **Connection Pooling** - Reuse connections efficiently
4. **Error Handling** - Don't crash if Immich DB is unavailable

---

## 🎯 Comparison: Standalone vs Immich Integration

| Feature | Standalone | Immich Integration |
|---------|-----------|-------------------|
| Setup Complexity | High (install dlib, face_recognition) | Low (just psycopg2) |
| Processing Speed | 1-2 sec/image | <1 sec/image |
| Duplicate Work | Yes (re-process all faces) | No (use existing) |
| Face Management | Custom UI needed | Use Immich UI |
| Accuracy | 85-92% | Same as Immich (90-95%) |
| Dependencies | Heavy (dlib, face_recognition) | Light (psycopg2) |
| Maintenance | High (our code) | Low (leverage Immich) |
| Learning | From our reviews | From Immich reviews |

**Winner**: **Immich Integration** ✅

---

## 💡 Best Practices

### 1. Let Immich Process First
```python
# After download, wait for Immich to scan
time.sleep(5)  # Or check if file is in Immich DB
```

### 2. Use Copy Instead of Move
```json
"move_or_copy": "copy"
```
This keeps originals in place, sorted copies in /faces/

### 3. Single Person Per Image
```json
"single_person_only": true
```
Skip images with multiple faces (let user review in Immich)

### 4. Monitor Immich Connection
```python
# Periodically check if Immich DB is available
# Fall back gracefully if not
```

---

## 🚀 Quick Start (30 Minutes)

### 1. Install PostgreSQL Client (5 min)
```bash
pip3 install psycopg2-binary
```

### 2. Get Immich DB Credentials (5 min)
```bash
# Find in Immich's docker-compose.yml or .env
grep POSTGRES immich/.env
```

### 3. Test Connection (5 min)
```python
# Use test script from above
python3 test_immich_connection.py
```

### 4. Add Configuration (5 min)
```bash
nano config.json
# Add immich and face_sorting sections
```

### 5. Test with One File (10 min)
```python
# Use basic test script
python3 test_immich_face_sort.py /path/to/image.jpg
```

---

## 📚 Resources

- [Immich Database Schema](https://github.com/immich-app/immich/tree/main/server/src/infra/migrations)
- [Immich API Docs](https://immich.app/docs/api)
- [PostgreSQL Python Client](https://www.psycopg.org/docs/)

---

## ✅ Success Checklist

- [ ] Connected to Immich PostgreSQL database
- [ ] Can query people list from Immich
- [ ] Can get faces for a specific file
- [ ] Tested sorting logic with sample files
- [ ] Configuration added to config.json
- [ ] Post-download hook integrated
- [ ] Web UI shows Immich connection status

---

**Status**: Ready for implementation
**Next Step**: Install psycopg2 and test Immich database connection
**Advantage**: Much simpler than standalone, leverages existing Immich infrastructure

---

**Last Updated**: 2025-10-31