LIComputerGuy/media-downloader
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
523f91788e158e9f7a2c84306499bc94ad1f3bc5
media-downloader/docs/archive/AI_FACE_RECOGNITION_IMMICH_INTEGRATION.md
Todd 0d7b2b1aab Initial commit 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-29 22:42:55 -04:00

933 lines
28 KiB
Markdown
Raw Blame History

# Face Recognition - Immich Integration Plan
**Created**: 2025-10-31
**Status**: Planning Phase - Immich Integration Approach
**Target Version**: 6.5.0
---
## 🎯 Overview
**NEW APPROACH**: Instead of building face recognition from scratch, integrate with Immich's existing face recognition system. Immich already processes faces, we just need to read its data and use it for auto-sorting.
---
## 💡 Why Use Immich's Face Data?
### Advantages
✅ **Already processed** - Immich has already detected faces in your photos
✅ **No duplicate processing** - Don't waste CPU doing the same work twice
✅ **Consistent** - Same face recognition across Immich and Media Downloader
✅ **Centralized management** - Manage people in one place (Immich UI)
✅ **Better accuracy** - Immich uses machine learning models that improve over time
✅ **GPU accelerated** - Immich can use GPU for faster processing
✅ **No new dependencies** - Don't need to install face_recognition library
### Architecture
```
Downloads → Immich Scan → Immich Face Recognition → Media Downloader Reads Data
↓
Auto-Sort by Person Name
```
---
## 🗄️ Immich Database Structure
### Understanding Immich's Face Tables
Immich stores face data in PostgreSQL database. Key tables:
#### 1. `person` table
Stores information about identified people:
```sql
SELECT * FROM person;
Columns:
- id (uuid)
- name (text) - Person's name
- thumbnailPath (text)
- isHidden (boolean)
- birthDate (date)
- createdAt, updatedAt
```
#### 2. `asset_faces` table
Links faces to assets (photos):
```sql
SELECT * FROM asset_faces;
Columns:
- id (uuid)
- assetId (uuid) - References the photo
- personId (uuid) - References the person (if identified)
- embedding (vector) - Face encoding data
- imageWidth, imageHeight
- boundingBoxX1, boundingBoxY1, boundingBoxX2, boundingBoxY2
```
#### 3. `assets` table
Photo metadata:
```sql
SELECT * FROM assets;
Columns:
- id (uuid)
- originalPath (text) - File path on disk
- originalFileName (text)
- type (enum) - IMAGE, VIDEO
- ownerId (uuid)
- libraryId (uuid)
- checksum (bytea) - File hash
```
### Key Relationships
```
assets (photos)
↓ (1 photo can have many faces)
asset_faces (detected faces)
↓ (each face can be linked to a person)
person (identified people)
```
---
## 🔌 Integration Architecture
### High-Level Flow
```
┌──────────────────────┐
│ 1. Image Downloaded │
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 2. Immich Scans │ ◄── Existing Immich process
│ (Auto/Manual) │ Detects faces, creates embeddings
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 3. User Identifies │ ◄── Done in Immich UI
│ Faces (Immich) │ Assigns names to faces
└──────────┬───────────┘
│
▼
┌──────────────────────┐
│ 4. Media Downloader │ ◄── NEW: Our integration
│ Reads Immich DB │ Query PostgreSQL
└──────────┬───────────┘
│
├─── Person identified? ──► Auto-sort to /faces/{person_name}/
│
└─── Not identified ──────► Leave in original location
```
### Implementation Options
#### Option A: Direct Database Integration (Recommended)
**Read Immich's PostgreSQL database directly**
Pros:
- Real-time access to face data
- No API dependencies
- Fast queries
- Can join tables for complex queries
Cons:
- Couples to Immich's database schema (may break on updates)
- Requires PostgreSQL connection
#### Option B: Immich API Integration
**Use Immich's REST API**
Pros:
- Stable interface (less likely to break)
- Official supported method
- Can work with remote Immich instances
Cons:
- Slower (HTTP overhead)
- May require multiple API calls
- Need to handle API authentication
**Recommendation**: Start with **Option A** (direct database), add Option B later if needed.
---
## 💾 Database Integration Implementation
### Step 1: Connect to Immich PostgreSQL
```python
import psycopg2
from psycopg2.extras import RealDictCursor
class ImmichFaceDB:
"""Read face recognition data from Immich database"""
def __init__(self, config):
self.config = config
self.conn = None
# Immich DB connection details
self.db_config = {
'host': config.get('immich', {}).get('db_host', 'localhost'),
'port': config.get('immich', {}).get('db_port', 5432),
'database': config.get('immich', {}).get('db_name', 'immich'),
'user': config.get('immich', {}).get('db_user', 'postgres'),
'password': config.get('immich', {}).get('db_password', '')
}
def connect(self):
"""Connect to Immich database"""
try:
self.conn = psycopg2.connect(**self.db_config)
return True
except Exception as e:
logging.error(f"Failed to connect to Immich DB: {e}")
return False
def get_faces_for_file(self, file_path: str) -> list:
"""
Get all identified faces for a specific file
Args:
file_path: Full path to the image file
Returns:
list of dicts: [{
'person_id': str,
'person_name': str,
'confidence': float,
'bounding_box': dict
}]
"""
if not self.conn:
self.connect()
try:
with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
# Query to get faces and their identified people
query = """
SELECT
p.id as person_id,
p.name as person_name,
af.id as face_id,
af."boundingBoxX1" as bbox_x1,
af."boundingBoxY1" as bbox_y1,
af."boundingBoxX2" as bbox_x2,
af."boundingBoxY2" as bbox_y2,
a."originalPath" as file_path,
a."originalFileName" as filename
FROM assets a
JOIN asset_faces af ON a.id = af."assetId"
LEFT JOIN person p ON af."personId" = p.id
WHERE a."originalPath" = %s
AND a.type = 'IMAGE'
AND p.name IS NOT NULL -- Only identified faces
AND p."isHidden" = false
"""
cursor.execute(query, (file_path,))
results = cursor.fetchall()
faces = []
for row in results:
faces.append({
'person_id': str(row['person_id']),
'person_name': row['person_name'],
'bounding_box': {
'x1': row['bbox_x1'],
'y1': row['bbox_y1'],
'x2': row['bbox_x2'],
'y2': row['bbox_y2']
}
})
return faces
except Exception as e:
logging.error(f"Error querying faces for {file_path}: {e}")
return []
def get_all_people(self) -> list:
"""Get list of all identified people in Immich"""
if not self.conn:
self.connect()
try:
with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
query = """
SELECT
id,
name,
"thumbnailPath",
"createdAt",
(SELECT COUNT(*) FROM asset_faces WHERE "personId" = person.id) as face_count
FROM person
WHERE name IS NOT NULL
AND "isHidden" = false
ORDER BY name
"""
cursor.execute(query)
return cursor.fetchall()
except Exception as e:
logging.error(f"Error getting people list: {e}")
return []
def get_unidentified_faces(self, limit=100) -> list:
"""
Get faces that haven't been identified yet
Returns:
list of dicts with file_path, face_id, bounding_box
"""
if not self.conn:
self.connect()
try:
with self.conn.cursor(cursor_factory=RealDictCursor) as cursor:
query = """
SELECT
a."originalPath" as file_path,
a."originalFileName" as filename,
af.id as face_id,
af."boundingBoxX1" as bbox_x1,
af."boundingBoxY1" as bbox_y1,
af."boundingBoxX2" as bbox_x2,
af."boundingBoxY2" as bbox_y2,
a."createdAt" as created_at
FROM asset_faces af
JOIN assets a ON af."assetId" = a.id
WHERE af."personId" IS NULL
AND a.type = 'IMAGE'
ORDER BY a."createdAt" DESC
LIMIT %s
"""
cursor.execute(query, (limit,))
return cursor.fetchall()
except Exception as e:
logging.error(f"Error getting unidentified faces: {e}")
return []
def close(self):
"""Close database connection"""
if self.conn:
self.conn.close()
```
---
## 🔄 Auto-Sort Implementation
### Core Auto-Sort Module
```python
#!/usr/bin/env python3
"""
Immich Face-Based Auto-Sorter
Reads face data from Immich and sorts images by person
"""
import os
import shutil
import logging
from pathlib import Path
from datetime import datetime
logger = logging.getLogger(__name__)
class ImmichFaceSorter:
"""Auto-sort images based on Immich face recognition"""
def __init__(self, config, immich_db):
self.config = config
self.immich_db = immich_db
# Configuration
self.enabled = config.get('face_sorting', {}).get('enabled', False)
self.base_dir = config.get('face_sorting', {}).get('base_directory',
'/mnt/storage/Downloads/faces')
self.min_faces_to_sort = config.get('face_sorting', {}).get('min_faces_to_sort', 1)
self.single_person_only = config.get('face_sorting', {}).get('single_person_only', True)
self.move_or_copy = config.get('face_sorting', {}).get('move_or_copy', 'copy') # 'move' or 'copy'
# Create base directory
os.makedirs(self.base_dir, exist_ok=True)
def process_downloaded_file(self, file_path: str) -> dict:
"""
Process a newly downloaded file
Args:
file_path: Full path to the downloaded image
Returns:
dict: {
'status': 'success'|'skipped'|'error',
'action': 'sorted'|'copied'|'skipped',
'person_name': str or None,
'faces_found': int,
'message': str
}
"""
if not self.enabled:
return {'status': 'skipped', 'message': 'Face sorting disabled'}
if not os.path.exists(file_path):
return {'status': 'error', 'message': 'File not found'}
# Only process images
ext = os.path.splitext(file_path)[1].lower()
if ext not in ['.jpg', '.jpeg', '.png', '.heic', '.heif']:
return {'status': 'skipped', 'message': 'Not an image file'}
# Wait for Immich to process (if needed)
# This could be a configurable delay or check if file is in Immich DB
import time
time.sleep(2) # Give Immich time to scan new file
# Get faces from Immich
faces = self.immich_db.get_faces_for_file(file_path)
if not faces:
logger.debug(f"No identified faces in {file_path}")
return {
'status': 'skipped',
'action': 'skipped',
'faces_found': 0,
'message': 'No identified faces found'
}
# Handle multiple faces
if len(faces) > 1 and self.single_person_only:
logger.info(f"Multiple faces ({len(faces)}) in {file_path}, skipping")
return {
'status': 'skipped',
'action': 'skipped',
'faces_found': len(faces),
'message': f'Multiple faces found ({len(faces)}), single_person_only=true'
}
# Sort to first person's directory (or implement multi-person logic)
primary_face = faces[0]
person_name = primary_face['person_name']
return self._sort_to_person(file_path, person_name, len(faces))
def _sort_to_person(self, file_path: str, person_name: str, faces_count: int) -> dict:
"""Move or copy file to person's directory"""
# Create person directory (sanitize name)
person_dir_name = self._sanitize_directory_name(person_name)
person_dir = os.path.join(self.base_dir, person_dir_name)
os.makedirs(person_dir, exist_ok=True)
# Determine target path
filename = os.path.basename(file_path)
target_path = os.path.join(person_dir, filename)
# Handle duplicates
if os.path.exists(target_path):
base, ext = os.path.splitext(filename)
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
filename = f"{base}_{timestamp}{ext}"
target_path = os.path.join(person_dir, filename)
try:
# Move or copy
if self.move_or_copy == 'move':
shutil.move(file_path, target_path)
action = 'sorted'
logger.info(f"Moved {filename} to {person_name}/")
else: # copy
shutil.copy2(file_path, target_path)
action = 'copied'
logger.info(f"Copied {filename} to {person_name}/")
return {
'status': 'success',
'action': action,
'person_name': person_name,
'faces_found': faces_count,
'target_path': target_path,
'message': f'{"Moved" if action == "sorted" else "Copied"} to {person_name}/'
}
except Exception as e:
logger.error(f"Error sorting {file_path}: {e}")
return {'status': 'error', 'message': str(e)}
def _sanitize_directory_name(self, name: str) -> str:
"""Convert person name to safe directory name"""
# Replace spaces with underscores, remove special chars
import re
safe_name = re.sub(r'[^\w\s-]', '', name)
safe_name = re.sub(r'[-\s]+', '_', safe_name)
return safe_name.lower()
def batch_sort_existing(self, source_dir: str = None, limit: int = None) -> dict:
"""
Batch sort existing files that are already in Immich
Args:
source_dir: Directory to process (None = all Immich files)
limit: Max files to process (None = all)
Returns:
dict: Statistics of operation
"""
stats = {
'processed': 0,
'sorted': 0,
'skipped': 0,
'errors': 0
}
# Query Immich for all files with identified faces
# This would require additional query method in ImmichFaceDB
logger.info(f"Batch sorting from {source_dir or 'all Immich files'}")
# Implementation here...
return stats
```
---
## ⚙️ Configuration
### Add to `config.json`:
```json
{
"immich": {
"enabled": true,
"url": "http://localhost:2283",
"api_key": "your-immich-api-key",
"db_host": "localhost",
"db_port": 5432,
"db_name": "immich",
"db_user": "postgres",
"db_password": "your-postgres-password"
},
"face_sorting": {
"enabled": true,
"base_directory": "/mnt/storage/Downloads/faces",
"min_faces_to_sort": 1,
"single_person_only": true,
"move_or_copy": "copy",
"process_delay_seconds": 5,
"sync_with_immich_scan": true,
"create_person_subdirs": true,
"handle_multiple_faces": "skip"
}
}
```
---
## 🔄 Integration Points
### 1. Post-Download Hook
Add face sorting after download completes:
```python
def on_download_complete(file_path: str, download_id: int):
"""Called when download completes"""
# Existing tasks
update_database(download_id)
send_notification(download_id)
# Trigger Immich scan (if not automatic)
if config.get('immich', {}).get('trigger_scan', True):
trigger_immich_library_scan()
# Wait for Immich to process
delay = config.get('face_sorting', {}).get('process_delay_seconds', 5)
time.sleep(delay)
# Sort by faces
if config.get('face_sorting', {}).get('enabled', False):
immich_db = ImmichFaceDB(config)
sorter = ImmichFaceSorter(config, immich_db)
result = sorter.process_downloaded_file(file_path)
logger.info(f"Face sort result: {result}")
immich_db.close()
```
### 2. Trigger Immich Library Scan
```python
def trigger_immich_library_scan():
"""Trigger Immich to scan for new files"""
import requests
immich_url = config.get('immich', {}).get('url')
api_key = config.get('immich', {}).get('api_key')
if not immich_url or not api_key:
return
try:
response = requests.post(
f"{immich_url}/api/library/scan",
headers={'x-api-key': api_key}
)
if response.status_code == 201:
logger.info("Triggered Immich library scan")
else:
logger.warning(f"Immich scan trigger failed: {response.status_code}")
except Exception as e:
logger.error(f"Error triggering Immich scan: {e}")
```
---
## 📊 Database Schema (Simplified)
Since we're reading from Immich, we only need minimal tracking:
```sql
-- Track what we've sorted
CREATE TABLE face_sort_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
download_id INTEGER,
original_path TEXT NOT NULL,
sorted_path TEXT NOT NULL,
person_name TEXT NOT NULL,
person_id TEXT, -- Immich person UUID
faces_count INTEGER DEFAULT 1,
action TEXT, -- 'moved' or 'copied'
sorted_at TEXT,
FOREIGN KEY (download_id) REFERENCES downloads(id)
);
CREATE INDEX idx_face_sort_person ON face_sort_history(person_name);
CREATE INDEX idx_face_sort_date ON face_sort_history(sorted_at);
```
---
## 🎨 Web UI (Simplified)
### Dashboard Page
```
┌─────────────────────────────────────────────┐
│ Face-Based Sorting (Powered by Immich) │
├─────────────────────────────────────────────┤
│ │
│ Status: [✓ Enabled] [⚙️ Configure] │
│ │
│ Connected to Immich: ✓ │
│ People in Immich: 12 │
│ Images Sorted: 145 │
│ │
│ ┌───────────────────────────────────────┐ │
│ │ Recent Activity │ │
│ │ │ │
│ │ • 14:23 - Sorted to "John" (3 images)│ │
│ │ • 14:20 - Sorted to "Sarah" (1 image)│ │
│ │ • 14:18 - Skipped (multiple faces) │ │
│ └───────────────────────────────────────┘ │
│ │
│ [View People] [Sort History] [Settings] │
│ │
│ 💡 Manage people and faces in Immich UI │
└─────────────────────────────────────────────┘
```
### People List (Read from Immich)
```
┌─────────────────────────────────────────────┐
│ People (from Immich) │
├─────────────────────────────────────────────┤
│ │
│ 👤 John Doe │
│ Faces in Immich: 25 │
│ Sorted by us: 42 images │
│ Directory: /faces/john_doe/ │
│ [View in Immich] │
│ │
│ 👤 Sarah Smith │
│ Faces in Immich: 18 │
│ Sorted by us: 28 images │
│ Directory: /faces/sarah_smith/ │
│ [View in Immich] │
│ │
│ 💡 Add/edit people in Immich interface │
└─────────────────────────────────────────────┘
```
---
## 🚀 Implementation Phases
### Phase 1: Basic Integration (Week 1)
- [ ] Install psycopg2 (PostgreSQL client)
- [ ] Create ImmichFaceDB class
- [ ] Test connection to Immich database
- [ ] Query faces for a test file
- [ ] List all people from Immich
### Phase 2: Auto-Sort Logic (Week 2)
- [ ] Create ImmichFaceSorter class
- [ ] Implement single-person sorting
- [ ] Handle move vs copy logic
- [ ] Add post-download hook integration
- [ ] Test with new downloads
### Phase 3: Configuration & Control (Week 3)
- [ ] Add configuration options
- [ ] Create enable/disable mechanism
- [ ] Add delay/timing controls
- [ ] Implement error handling
- [ ] Add logging
### Phase 4: Web UI (Week 4)
- [ ] Dashboard page (stats, enable/disable)
- [ ] People list (read from Immich)
- [ ] Sort history page
- [ ] Configuration interface
### Phase 5: Advanced Features (Week 5)
- [ ] Multi-face handling options
- [ ] Batch sort existing files
- [ ] Immich API integration (fallback)
- [ ] Statistics and reporting
### Phase 6: Polish (Week 6)
- [ ] Performance optimization
- [ ] Documentation
- [ ] Testing
- [ ] Error recovery
---
## 📝 API Endpoints (New)
```python
# Face Sorting Status
GET /api/face-sort/status
POST /api/face-sort/enable
POST /api/face-sort/disable
# People (Read from Immich)
GET /api/face-sort/people # List people from Immich
GET /api/face-sort/people/{id} # Get person details
# History
GET /api/face-sort/history # Our sorting history
GET /api/face-sort/stats # Statistics
# Operations
POST /api/face-sort/batch # Batch sort existing files
GET /api/face-sort/batch/status # Check batch progress
# Immich Connection
GET /api/face-sort/immich/status # Test Immich connection
POST /api/face-sort/immich/scan # Trigger Immich library scan
```
---
## 🔧 Installation & Setup
### Step 1: Install PostgreSQL Client
```bash
pip3 install psycopg2-binary
```
### Step 2: Get Immich Database Credentials
```bash
# If Immich is running in Docker
docker exec -it immich_postgres env | grep POSTGRES
# Get credentials from Immich's docker-compose.yml or .env file
```
### Step 3: Test Connection
```python
import psycopg2
try:
conn = psycopg2.connect(
host="localhost",
port=5432,
database="immich",
user="postgres",
password="your-password"
)
print("✓ Connected to Immich database!")
conn.close()
except Exception as e:
print(f"✗ Connection failed: {e}")
```
### Step 4: Configure
Add Immich settings to `config.json`:
```json
{
"immich": {
"db_host": "localhost",
"db_port": 5432,
"db_name": "immich",
"db_user": "postgres",
"db_password": "your-password"
},
"face_sorting": {
"enabled": true,
"base_directory": "/mnt/storage/Downloads/faces"
}
}
```
---
## ⚡ Performance Considerations
### Efficiency Gains
- **No duplicate processing** - Immich already did the heavy lifting
- **Fast queries** - Direct database access (milliseconds)
- **No ML overhead** - No face detection/recognition on our end
- **Scalable** - Works with thousands of photos
### Timing
- Database query: ~10-50ms per file
- File operation (move/copy): ~100-500ms
- Total per image: <1 second
---
## 🔒 Security Considerations
1. **Database Access** - Store PostgreSQL credentials securely
2. **Read-Only** - Only read from Immich DB, never write
3. **Connection Pooling** - Reuse connections efficiently
4. **Error Handling** - Don't crash if Immich DB is unavailable
---
## 🎯 Comparison: Standalone vs Immich Integration
| Feature | Standalone | Immich Integration |
|---------|-----------|-------------------|
| Setup Complexity | High (install dlib, face_recognition) | Low (just psycopg2) |
| Processing Speed | 1-2 sec/image | <1 sec/image |
| Duplicate Work | Yes (re-process all faces) | No (use existing) |
| Face Management | Custom UI needed | Use Immich UI |
| Accuracy | 85-92% | Same as Immich (90-95%) |
| Dependencies | Heavy (dlib, face_recognition) | Light (psycopg2) |
| Maintenance | High (our code) | Low (leverage Immich) |
| Learning | From our reviews | From Immich reviews |
**Winner**: **Immich Integration** ✅
---
## 💡 Best Practices
### 1. Let Immich Process First
```python
# After download, wait for Immich to scan
time.sleep(5) # Or check if file is in Immich DB
```
### 2. Use Copy Instead of Move
```json
"move_or_copy": "copy"
```
This keeps originals in place, sorted copies in /faces/
### 3. Single Person Per Image
```json
"single_person_only": true
```
Skip images with multiple faces (let user review in Immich)
### 4. Monitor Immich Connection
```python
# Periodically check if Immich DB is available
# Fall back gracefully if not
```
---
## 🚀 Quick Start (30 Minutes)
### 1. Install PostgreSQL Client (5 min)
```bash
pip3 install psycopg2-binary
```
### 2. Get Immich DB Credentials (5 min)
```bash
# Find in Immich's docker-compose.yml or .env
grep POSTGRES immich/.env
```
### 3. Test Connection (5 min)
```python
# Use test script from above
python3 test_immich_connection.py
```
### 4. Add Configuration (5 min)
```bash
nano config.json
# Add immich and face_sorting sections
```
### 5. Test with One File (10 min)
```python
# Use basic test script
python3 test_immich_face_sort.py /path/to/image.jpg
```
---
## 📚 Resources
- [Immich Database Schema](https://github.com/immich-app/immich/tree/main/server/src/infra/migrations)
- [Immich API Docs](https://immich.app/docs/api)
- [PostgreSQL Python Client](https://www.psycopg.org/docs/)
---
## ✅ Success Checklist
- [ ] Connected to Immich PostgreSQL database
- [ ] Can query people list from Immich
- [ ] Can get faces for a specific file
- [ ] Tested sorting logic with sample files
- [ ] Configuration added to config.json
- [ ] Post-download hook integrated
- [ ] Web UI shows Immich connection status
---
**Status**: Ready for implementation
**Next Step**: Install psycopg2 and test Immich database connection
**Advantage**: Much simpler than standalone, leverages existing Immich infrastructure
---
**Last Updated**: 2025-10-31
Reference in New Issue View Git Blame Copy Permalink