media-downloader/docs/archive/AI_FACE_RECOGNITION_IMMICH_INTEGRATION.md

Face Recognition - Immich Integration Plan

Created: 2025-10-31 Status: Planning Phase - Immich Integration Approach Target Version: 6.5.0

---

🎯 Overview

NEW APPROACH: Instead of building face recognition from scratch, integrate with Immich's existing face recognition system. Immich already processes faces, we just need to read its data and use it for auto-sorting.

---

💡 Why Use Immich's Face Data?

Advantages

✅ Already processed - Immich has already detected faces in your photos ✅ No duplicate processing - Don't waste CPU doing the same work twice ✅ Consistent - Same face recognition across Immich and Media Downloader ✅ Centralized management - Manage people in one place (Immich UI) ✅ Better accuracy - Immich uses machine learning models that improve over time ✅ GPU accelerated - Immich can use GPU for faster processing ✅ No new dependencies - Don't need to install face_recognition library

Architecture

Downloads → Immich Scan → Immich Face Recognition → Media Downloader Reads Data
                                                              ↓
                                                    Auto-Sort by Person Name

---

🗄️ Immich Database Structure

Understanding Immich's Face Tables

Immich stores face data in PostgreSQL database. Key tables:

1. person table

Stores information about identified people:

SELECT * FROM person;

Columns:
- id (uuid)
- name (text) - Person's name
- thumbnailPath (text)
- isHidden (boolean)
- birthDate (date)
- createdAt, updatedAt

2. asset_faces table

Links faces to assets (photos):

SELECT * FROM asset_faces;

Columns:
- id (uuid)
- assetId (uuid) - References the photo
- personId (uuid) - References the person (if identified)
- embedding (vector) - Face encoding data
- imageWidth, imageHeight
- boundingBoxX1, boundingBoxY1, boundingBoxX2, boundingBoxY2

3. assets table

Photo metadata:

SELECT * FROM assets;

Columns:
- id (uuid)
- originalPath (text) - File path on disk
- originalFileName (text)
- type (enum) - IMAGE, VIDEO
- ownerId (uuid)
- libraryId (uuid)
- checksum (bytea) - File hash

Key Relationships

assets (photos)
  ↓ (1 photo can have many faces)
asset_faces (detected faces)
  ↓ (each face can be linked to a person)
person (identified people)

---

🔌 Integration Architecture

High-Level Flow

┌──────────────────────┐
│  1. Image Downloaded │
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  2. Immich Scans     │ ◄── Existing Immich process
│     (Auto/Manual)     │     Detects faces, creates embeddings
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│  3. User Identifies  │ ◄── Done in Immich UI
│     Faces (Immich)   │     Assigns names to faces
└──────────┬───────────┘
           │
           ▼
┌──────────────────────┐
│ 4. Media Downloader  │ ◄── NEW: Our integration
│    Reads Immich DB   │     Query PostgreSQL
└──────────┬───────────┘
           │
           ├─── Person identified? ──► Auto-sort to /faces/{person_name}/
           │
           └─── Not identified ──────► Leave in original location

Implementation Options

Option A: Direct Database Integration (Recommended)

Read Immich's PostgreSQL database directly

Pros:

• Real-time access to face data
• No API dependencies
• Fast queries
• Can join tables for complex queries

Cons:

• Couples to Immich's database schema (may break on updates)
• Requires PostgreSQL connection

Option B: Immich API Integration

Use Immich's REST API

Pros:

• Stable interface (less likely to break)
• Official supported method
• Can work with remote Immich instances

Cons:

• Slower (HTTP overhead)
• May require multiple API calls
• Need to handle API authentication

Recommendation: Start with Option A (direct database), add Option B later if needed.

---

💾 Database Integration Implementation

Step 1: Connect to Immich PostgreSQL

import psycopg2
from psycopg2.extras import RealDictCursor

class ImmichFaceDB:
    """Read face recognition data from Immich database"""

    def __init__(self, config):
        self.config = config
        self.conn = None

        # Immich DB connection details
        self.db_config = {
            'host': config.get('immich', {}).get('db_host', 'localhost'),
            'port': config.get('immich', {}).get('db_port', 5432),
            'database': config.get('immich', {}).get('db_name', 'immich'),
            'user': config.get('immich', {}).get('db_user', 'postgres'),
            'password': config.get('immich', {}).get('db_password', '')
        }

    def connect(self):
        """Connect to Immich database"""
        try:
            self.conn = psycopg2.connect(**self.db_config)
            return True
        except Exception as e:
            logging.error(f"Failed to connect to Immich DB: {e}")
            return False

    def get_faces_for_file(self, file_path: str) -> list:
        """
        Get all identified faces for a specific file

        Args:
            file_path: Full path to the image file

        Returns:
            list of dicts: [{
                'person_id': str,
                'person_name': str,
                'confidence': float,
                'bounding_box': dict
            }]
        """
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                # Query to get faces and their identified people
                query = """
                    SELECT
                        p.id as person_id,
                        p.name as person_name,
                        af.id as face_id,
                        af."boundingBoxX1" as bbox_x1,
                        af."boundingBoxY1" as bbox_y1,
                        af."boundingBoxX2" as bbox_x2,
                        af."boundingBoxY2" as bbox_y2,
                        a."originalPath" as file_path,
                        a."originalFileName" as filename
                    FROM assets a
                    JOIN asset_faces af ON a.id = af."assetId"
                    LEFT JOIN person p ON af."personId" = p.id
                    WHERE a."originalPath" = %s
                        AND a.type = 'IMAGE'
                        AND p.name IS NOT NULL  -- Only identified faces
                        AND p."isHidden" = false
                """

                cursor.execute(query, (file_path,))
                results = cursor.fetchall()

                faces = []
                for row in results:
                    faces.append({
                        'person_id': str(row['person_id']),
                        'person_name': row['person_name'],
                        'bounding_box': {
                            'x1': row['bbox_x1'],
                            'y1': row['bbox_y1'],
                            'x2': row['bbox_x2'],
                            'y2': row['bbox_y2']
                        }
                    })

                return faces

        except Exception as e:
            logging.error(f"Error querying faces for {file_path}: {e}")
            return []

    def get_all_people(self) -> list:
        """Get list of all identified people in Immich"""
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                query = """
                    SELECT
                        id,
                        name,
                        "thumbnailPath",
                        "createdAt",
                        (SELECT COUNT(*) FROM asset_faces WHERE "personId" = person.id) as face_count
                    FROM person
                    WHERE name IS NOT NULL
                        AND "isHidden" = false
                    ORDER BY name
                """

                cursor.execute(query)
                return cursor.fetchall()

        except Exception as e:
            logging.error(f"Error getting people list: {e}")
            return []

    def get_unidentified_faces(self, limit=100) -> list:
        """
        Get faces that haven't been identified yet

        Returns:
            list of dicts with file_path, face_id, bounding_box
        """
        if not self.conn:
            self.connect()

        try:
            with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
                query = """
                    SELECT
                        a."originalPath" as file_path,
                        a."originalFileName" as filename,
                        af.id as face_id,
                        af."boundingBoxX1" as bbox_x1,
                        af."boundingBoxY1" as bbox_y1,
                        af."boundingBoxX2" as bbox_x2,
                        af."boundingBoxY2" as bbox_y2,
                        a."createdAt" as created_at
                    FROM asset_faces af
                    JOIN assets a ON af."assetId" = a.id
                    WHERE af."personId" IS NULL
                        AND a.type = 'IMAGE'
                    ORDER BY a."createdAt" DESC
                    LIMIT %s
                """

                cursor.execute(query, (limit,))
                return cursor.fetchall()

        except Exception as e:
            logging.error(f"Error getting unidentified faces: {e}")
            return []

    def close(self):
        """Close database connection"""
        if self.conn:
            self.conn.close()

---

🔄 Auto-Sort Implementation

Core Auto-Sort Module

#!/usr/bin/env python3
"""
Immich Face-Based Auto-Sorter
Reads face data from Immich and sorts images by person
"""

import os
import shutil
import logging
from pathlib import Path
from datetime import datetime

logger = logging.getLogger(__name__)


class ImmichFaceSorter:
    """Auto-sort images based on Immich face recognition"""

    def __init__(self, config, immich_db):
        self.config = config
        self.immich_db = immich_db

        # Configuration
        self.enabled = config.get('face_sorting', {}).get('enabled', False)
        self.base_dir = config.get('face_sorting', {}).get('base_directory',
                                                            '/mnt/storage/Downloads/faces')
        self.min_faces_to_sort = config.get('face_sorting', {}).get('min_faces_to_sort', 1)
        self.single_person_only = config.get('face_sorting', {}).get('single_person_only', True)
        self.move_or_copy = config.get('face_sorting', {}).get('move_or_copy', 'copy')  # 'move' or 'copy'

        # Create base directory
        os.makedirs(self.base_dir, exist_ok=True)

    def process_downloaded_file(self, file_path: str) -> dict:
        """
        Process a newly downloaded file

        Args:
            file_path: Full path to the downloaded image

        Returns:
            dict: {
                'status': 'success'|'skipped'|'error',
                'action': 'sorted'|'copied'|'skipped',
                'person_name': str or None,
                'faces_found': int,
                'message': str
            }
        """
        if not self.enabled:
            return {'status': 'skipped', 'message': 'Face sorting disabled'}

        if not os.path.exists(file_path):
            return {'status': 'error', 'message': 'File not found'}

        # Only process images
        ext = os.path.splitext(file_path)[1].lower()
        if ext not in ['.jpg', '.jpeg', '.png', '.heic', '.heif']:
            return {'status': 'skipped', 'message': 'Not an image file'}

        # Wait for Immich to process (if needed)
        # This could be a configurable delay or check if file is in Immich DB
        import time
        time.sleep(2)  # Give Immich time to scan new file

        # Get faces from Immich
        faces = self.immich_db.get_faces_for_file(file_path)

        if not faces:
            logger.debug(f"No identified faces in {file_path}")
            return {
                'status': 'skipped',
                'action': 'skipped',
                'faces_found': 0,
                'message': 'No identified faces found'
            }

        # Handle multiple faces
        if len(faces) > 1 and self.single_person_only:
            logger.info(f"Multiple faces ({len(faces)}) in {file_path}, skipping")
            return {
                'status': 'skipped',
                'action': 'skipped',
                'faces_found': len(faces),
                'message': f'Multiple faces found ({len(faces)}), single_person_only=true'
            }

        # Sort to first person's directory (or implement multi-person logic)
        primary_face = faces[0]
        person_name = primary_face['person_name']

        return self._sort_to_person(file_path, person_name, len(faces))

    def _sort_to_person(self, file_path: str, person_name: str, faces_count: int) -> dict:
        """Move or copy file to person's directory"""

        # Create person directory (sanitize name)
        person_dir_name = self._sanitize_directory_name(person_name)
        person_dir = os.path.join(self.base_dir, person_dir_name)
        os.makedirs(person_dir, exist_ok=True)

        # Determine target path
        filename = os.path.basename(file_path)
        target_path = os.path.join(person_dir, filename)

        # Handle duplicates
        if os.path.exists(target_path):
            base, ext = os.path.splitext(filename)
            timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
            filename = f"{base}_{timestamp}{ext}"
            target_path = os.path.join(person_dir, filename)

        try:
            # Move or copy
            if self.move_or_copy == 'move':
                shutil.move(file_path, target_path)
                action = 'sorted'
                logger.info(f"Moved {filename} to {person_name}/")
            else:  # copy
                shutil.copy2(file_path, target_path)
                action = 'copied'
                logger.info(f"Copied {filename} to {person_name}/")

            return {
                'status': 'success',
                'action': action,
                'person_name': person_name,
                'faces_found': faces_count,
                'target_path': target_path,
                'message': f'{"Moved" if action == "sorted" else "Copied"} to {person_name}/'
            }

        except Exception as e:
            logger.error(f"Error sorting {file_path}: {e}")
            return {'status': 'error', 'message': str(e)}

    def _sanitize_directory_name(self, name: str) -> str:
        """Convert person name to safe directory name"""
        # Replace spaces with underscores, remove special chars
        import re
        safe_name = re.sub(r'[^\w\s-]', '', name)
        safe_name = re.sub(r'[-\s]+', '_', safe_name)
        return safe_name.lower()

    def batch_sort_existing(self, source_dir: str = None, limit: int = None) -> dict:
        """
        Batch sort existing files that are already in Immich

        Args:
            source_dir: Directory to process (None = all Immich files)
            limit: Max files to process (None = all)

        Returns:
            dict: Statistics of operation
        """
        stats = {
            'processed': 0,
            'sorted': 0,
            'skipped': 0,
            'errors': 0
        }

        # Query Immich for all files with identified faces
        # This would require additional query method in ImmichFaceDB

        logger.info(f"Batch sorting from {source_dir or 'all Immich files'}")

        # Implementation here...

        return stats

---

⚙️ Configuration

Add to config.json:

{
  "immich": {
    "enabled": true,
    "url": "http://localhost:2283",
    "api_key": "your-immich-api-key",
    "db_host": "localhost",
    "db_port": 5432,
    "db_name": "immich",
    "db_user": "postgres",
    "db_password": "your-postgres-password"
  },
  "face_sorting": {
    "enabled": true,
    "base_directory": "/mnt/storage/Downloads/faces",
    "min_faces_to_sort": 1,
    "single_person_only": true,
    "move_or_copy": "copy",
    "process_delay_seconds": 5,
    "sync_with_immich_scan": true,
    "create_person_subdirs": true,
    "handle_multiple_faces": "skip"
  }
}

---

🔄 Integration Points

1. Post-Download Hook

Add face sorting after download completes:

def on_download_complete(file_path: str, download_id: int):
    """Called when download completes"""

    # Existing tasks
    update_database(download_id)
    send_notification(download_id)

    # Trigger Immich scan (if not automatic)
    if config.get('immich', {}).get('trigger_scan', True):
        trigger_immich_library_scan()

    # Wait for Immich to process
    delay = config.get('face_sorting', {}).get('process_delay_seconds', 5)
    time.sleep(delay)

    # Sort by faces
    if config.get('face_sorting', {}).get('enabled', False):
        immich_db = ImmichFaceDB(config)
        sorter = ImmichFaceSorter(config, immich_db)
        result = sorter.process_downloaded_file(file_path)
        logger.info(f"Face sort result: {result}")
        immich_db.close()

2. Trigger Immich Library Scan

def trigger_immich_library_scan():
    """Trigger Immich to scan for new files"""
    import requests

    immich_url = config.get('immich', {}).get('url')
    api_key = config.get('immich', {}).get('api_key')

    if not immich_url or not api_key:
        return

    try:
        response = requests.post(
            f"{immich_url}/api/library/scan",
            headers={'x-api-key': api_key}
        )
        if response.status_code == 201:
            logger.info("Triggered Immich library scan")
        else:
            logger.warning(f"Immich scan trigger failed: {response.status_code}")
    except Exception as e:
        logger.error(f"Error triggering Immich scan: {e}")

---

📊 Database Schema (Simplified)

Since we're reading from Immich, we only need minimal tracking:

-- Track what we've sorted
CREATE TABLE face_sort_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    download_id INTEGER,
    original_path TEXT NOT NULL,
    sorted_path TEXT NOT NULL,
    person_name TEXT NOT NULL,
    person_id TEXT,  -- Immich person UUID
    faces_count INTEGER DEFAULT 1,
    action TEXT,  -- 'moved' or 'copied'
    sorted_at TEXT,
    FOREIGN KEY (download_id) REFERENCES downloads(id)
);

CREATE INDEX idx_face_sort_person ON face_sort_history(person_name);
CREATE INDEX idx_face_sort_date ON face_sort_history(sorted_at);

---

🎨 Web UI (Simplified)

Dashboard Page

┌─────────────────────────────────────────────┐
│ Face-Based Sorting (Powered by Immich)     │
├─────────────────────────────────────────────┤
│                                             │
│ Status: [✓ Enabled] [⚙️ Configure]         │
│                                             │
│ Connected to Immich: ✓                     │
│ People in Immich: 12                        │
│ Images Sorted: 145                          │
│                                             │
│ ┌───────────────────────────────────────┐  │
│ │ Recent Activity                       │  │
│ │                                       │  │
│ │ • 14:23 - Sorted to "John" (3 images)│  │
│ │ • 14:20 - Sorted to "Sarah" (1 image)│  │
│ │ • 14:18 - Skipped (multiple faces)   │  │
│ └───────────────────────────────────────┘  │
│                                             │
│ [View People] [Sort History] [Settings]    │
│                                             │
│ 💡 Manage people and faces in Immich UI    │
└─────────────────────────────────────────────┘

People List (Read from Immich)

┌─────────────────────────────────────────────┐
│ People (from Immich)                        │
├─────────────────────────────────────────────┤
│                                             │
│ 👤 John Doe                                 │
│    Faces in Immich: 25                      │
│    Sorted by us: 42 images                  │
│    Directory: /faces/john_doe/              │
│    [View in Immich]                         │
│                                             │
│ 👤 Sarah Smith                              │
│    Faces in Immich: 18                      │
│    Sorted by us: 28 images                  │
│    Directory: /faces/sarah_smith/           │
│    [View in Immich]                         │
│                                             │
│ 💡 Add/edit people in Immich interface      │
└─────────────────────────────────────────────┘

---

🚀 Implementation Phases

Phase 1: Basic Integration (Week 1)

• Install psycopg2 (PostgreSQL client)
• Create ImmichFaceDB class
• Test connection to Immich database
• Query faces for a test file
• List all people from Immich

Phase 2: Auto-Sort Logic (Week 2)

• Create ImmichFaceSorter class
• Implement single-person sorting
• Handle move vs copy logic
• Add post-download hook integration
• Test with new downloads

Phase 3: Configuration & Control (Week 3)

• Add configuration options
• Create enable/disable mechanism
• Add delay/timing controls
• Implement error handling
• Add logging

Phase 4: Web UI (Week 4)

• Dashboard page (stats, enable/disable)
• People list (read from Immich)
• Sort history page
• Configuration interface

Phase 5: Advanced Features (Week 5)

• Multi-face handling options
• Batch sort existing files
• Immich API integration (fallback)
• Statistics and reporting

Phase 6: Polish (Week 6)

• Performance optimization
• Documentation
• Testing
• Error recovery

---

📝 API Endpoints (New)

# Face Sorting Status
GET  /api/face-sort/status
POST /api/face-sort/enable
POST /api/face-sort/disable

# People (Read from Immich)
GET  /api/face-sort/people          # List people from Immich
GET  /api/face-sort/people/{id}     # Get person details

# History
GET  /api/face-sort/history         # Our sorting history
GET  /api/face-sort/stats           # Statistics

# Operations
POST /api/face-sort/batch           # Batch sort existing files
GET  /api/face-sort/batch/status    # Check batch progress

# Immich Connection
GET  /api/face-sort/immich/status   # Test Immich connection
POST /api/face-sort/immich/scan     # Trigger Immich library scan

---

🔧 Installation & Setup

Step 1: Install PostgreSQL Client

pip3 install psycopg2-binary

Step 2: Get Immich Database Credentials

# If Immich is running in Docker
docker exec -it immich_postgres env | grep POSTGRES

# Get credentials from Immich's docker-compose.yml or .env file

Step 3: Test Connection

import psycopg2

try:
    conn = psycopg2.connect(
        host="localhost",
        port=5432,
        database="immich",
        user="postgres",
        password="your-password"
    )
    print("✓ Connected to Immich database!")
    conn.close()
except Exception as e:
    print(f"✗ Connection failed: {e}")

Step 4: Configure

Add Immich settings to config.json:

{
  "immich": {
    "db_host": "localhost",
    "db_port": 5432,
    "db_name": "immich",
    "db_user": "postgres",
    "db_password": "your-password"
  },
  "face_sorting": {
    "enabled": true,
    "base_directory": "/mnt/storage/Downloads/faces"
  }
}

---

⚡ Performance Considerations

Efficiency Gains

• No duplicate processing - Immich already did the heavy lifting
• Fast queries - Direct database access (milliseconds)
• No ML overhead - No face detection/recognition on our end
• Scalable - Works with thousands of photos

Timing

• Database query: ~10-50ms per file
• File operation (move/copy): ~100-500ms
• Total per image: <1 second

---

🔒 Security Considerations

1. Database Access - Store PostgreSQL credentials securely
2. Read-Only - Only read from Immich DB, never write
3. Connection Pooling - Reuse connections efficiently
4. Error Handling - Don't crash if Immich DB is unavailable

---

🎯 Comparison: Standalone vs Immich Integration

Feature	Standalone	Immich Integration
Setup Complexity	High (install dlib, face_recognition)	Low (just psycopg2)
Processing Speed	1-2 sec/image	<1 sec/image
Duplicate Work	Yes (re-process all faces)	No (use existing)
Face Management	Custom UI needed	Use Immich UI
Accuracy	85-92%	Same as Immich (90-95%)
Dependencies	Heavy (dlib, face_recognition)	Light (psycopg2)
Maintenance	High (our code)	Low (leverage Immich)
Learning	From our reviews	From Immich reviews

Winner: Immich Integration ✅

---

💡 Best Practices

1. Let Immich Process First

# After download, wait for Immich to scan
time.sleep(5)  # Or check if file is in Immich DB

2. Use Copy Instead of Move

"move_or_copy": "copy"

This keeps originals in place, sorted copies in /faces/

3. Single Person Per Image

"single_person_only": true

Skip images with multiple faces (let user review in Immich)

4. Monitor Immich Connection

# Periodically check if Immich DB is available
# Fall back gracefully if not

---

🚀 Quick Start (30 Minutes)

1. Install PostgreSQL Client (5 min)

pip3 install psycopg2-binary

2. Get Immich DB Credentials (5 min)

# Find in Immich's docker-compose.yml or .env
grep POSTGRES immich/.env

3. Test Connection (5 min)

# Use test script from above
python3 test_immich_connection.py

4. Add Configuration (5 min)

nano config.json
# Add immich and face_sorting sections

5. Test with One File (10 min)

# Use basic test script
python3 test_immich_face_sort.py /path/to/image.jpg

---

📚 Resources

• Immich Database Schema
• Immich API Docs
• PostgreSQL Python Client

---

✅ Success Checklist

• Connected to Immich PostgreSQL database
• Can query people list from Immich
• Can get faces for a specific file
• Tested sorting logic with sample files
• Configuration added to config.json
• Post-download hook integrated
• Web UI shows Immich connection status

---

Status: Ready for implementation Next Step: Install psycopg2 and test Immich database connection Advantage: Much simpler than standalone, leverages existing Immich infrastructure

---

Last Updated: 2025-10-31