Todd
278
docs/CACHE_BUILDER.md
Normal file
278
docs/CACHE_BUILDER.md
Normal file
@@ -0,0 +1,278 @@
|
||||
# Media Cache Builder
|
||||
|
||||
## Overview
|
||||
|
||||
The Media Cache Builder is a background service that pre-generates thumbnails and caches metadata for all media files in the system. This significantly improves performance by:
|
||||
|
||||
- **Pre-generating thumbnails**: Thumbnails are created in advance rather than on-demand when viewing media
|
||||
- **Caching metadata**: Resolution, file size, duration, and format information is extracted and cached
|
||||
- **Reducing API latency**: Media gallery and downloads pages load much faster with cached data
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Background Worker Script
|
||||
|
||||
**Location**: `/opt/media-downloader/modules/thumbnail_cache_builder.py`
|
||||
|
||||
This Python script scans all media files in `/opt/immich/md` and:
|
||||
- Generates 300x300 pixel thumbnails for images and videos
|
||||
- Extracts metadata (width, height, duration, format)
|
||||
- Stores thumbnails in `/opt/media-downloader/database/thumbnails.db`
|
||||
- Stores metadata in `/opt/media-downloader/database/media_metadata.db`
|
||||
- Skips files that are already cached and haven't been modified
|
||||
- Runs with low priority (Nice=19, IOSchedulingClass=idle) to avoid impacting system performance
|
||||
|
||||
### 2. Systemd Service
|
||||
|
||||
**Location**: `/etc/systemd/system/media-cache-builder.service`
|
||||
|
||||
A oneshot systemd service that runs the cache builder script.
|
||||
|
||||
**Resource Limits**:
|
||||
- CPU quota: 50% (limited to prevent high CPU usage)
|
||||
- I/O scheduling: idle priority
|
||||
- Nice level: 19 (lowest CPU priority)
|
||||
|
||||
### 3. Systemd Timer
|
||||
|
||||
**Location**: `/etc/systemd/system/media-cache-builder.timer`
|
||||
|
||||
Automatically runs the cache builder daily at 3:00 AM with a randomized delay of up to 30 minutes.
|
||||
|
||||
**Schedule**:
|
||||
- Daily at 3:00 AM
|
||||
- Persistent (runs missed timers on boot)
|
||||
- Random delay: 0-30 minutes
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### Get Cached Metadata
|
||||
|
||||
```
|
||||
GET /api/media/metadata?file_path=/path/to/file
|
||||
```
|
||||
|
||||
Returns cached metadata for a media file:
|
||||
```json
|
||||
{
|
||||
"file_path": "/opt/immich/md/instagram/user/image.jpg",
|
||||
"width": 1920,
|
||||
"height": 1080,
|
||||
"file_size": 245678,
|
||||
"duration": null,
|
||||
"format": "JPEG",
|
||||
"cached": true,
|
||||
"cached_at": "2025-10-30T22:36:45.123"
|
||||
}
|
||||
```
|
||||
|
||||
### Trigger Cache Rebuild
|
||||
|
||||
```
|
||||
POST /api/media/cache/rebuild
|
||||
```
|
||||
|
||||
Manually triggers a cache rebuild in the background:
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"message": "Cache rebuild started in background"
|
||||
}
|
||||
```
|
||||
|
||||
### Get Cache Statistics
|
||||
|
||||
```
|
||||
GET /api/media/cache/stats
|
||||
```
|
||||
|
||||
Returns statistics about the cache:
|
||||
```json
|
||||
{
|
||||
"thumbnails": {
|
||||
"exists": true,
|
||||
"count": 2126,
|
||||
"size_bytes": 52428800
|
||||
},
|
||||
"metadata": {
|
||||
"exists": true,
|
||||
"count": 2126,
|
||||
"size_bytes": 204800
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Manual Usage
|
||||
|
||||
### Run Cache Builder Manually
|
||||
|
||||
```bash
|
||||
# Run directly
|
||||
sudo /usr/bin/python3 /opt/media-downloader/modules/thumbnail_cache_builder.py
|
||||
|
||||
# Or via systemd
|
||||
sudo systemctl start media-cache-builder.service
|
||||
```
|
||||
|
||||
### Check Service Status
|
||||
|
||||
```bash
|
||||
# Check if timer is active
|
||||
sudo systemctl status media-cache-builder.timer
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u media-cache-builder.service -f
|
||||
|
||||
# Check when next run is scheduled
|
||||
systemctl list-timers media-cache-builder.timer
|
||||
```
|
||||
|
||||
### Enable/Disable Automatic Runs
|
||||
|
||||
```bash
|
||||
# Disable daily automatic runs
|
||||
sudo systemctl stop media-cache-builder.timer
|
||||
sudo systemctl disable media-cache-builder.timer
|
||||
|
||||
# Re-enable daily automatic runs
|
||||
sudo systemctl enable media-cache-builder.timer
|
||||
sudo systemctl start media-cache-builder.timer
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Thumbnails Database
|
||||
|
||||
**Location**: `/opt/media-downloader/database/thumbnails.db`
|
||||
|
||||
```sql
|
||||
CREATE TABLE thumbnails (
|
||||
file_hash TEXT PRIMARY KEY,
|
||||
file_path TEXT NOT NULL,
|
||||
thumbnail_data BLOB NOT NULL,
|
||||
created_at TEXT,
|
||||
file_mtime REAL
|
||||
);
|
||||
CREATE INDEX idx_file_path ON thumbnails(file_path);
|
||||
```
|
||||
|
||||
### Metadata Database
|
||||
|
||||
**Location**: `/opt/media-downloader/database/media_metadata.db`
|
||||
|
||||
```sql
|
||||
CREATE TABLE media_metadata (
|
||||
file_hash TEXT PRIMARY KEY,
|
||||
file_path TEXT NOT NULL,
|
||||
width INTEGER,
|
||||
height INTEGER,
|
||||
file_size INTEGER,
|
||||
duration REAL,
|
||||
format TEXT,
|
||||
created_at TEXT,
|
||||
file_mtime REAL
|
||||
);
|
||||
CREATE INDEX idx_meta_file_path ON media_metadata(file_path);
|
||||
```
|
||||
|
||||
## Performance
|
||||
|
||||
### Typical Performance
|
||||
|
||||
- **Processing rate**: 15-25 files/second (varies by file size and type)
|
||||
- **Memory usage**: ~900MB - 1GB during operation
|
||||
- **CPU usage**: Limited to 50% of one core
|
||||
- **I/O priority**: Idle (won't interfere with normal operations)
|
||||
|
||||
### For 2,000 files:
|
||||
- **Time**: ~2-3 minutes
|
||||
- **Thumbnail cache size**: ~50-100MB
|
||||
- **Metadata cache size**: ~200-500KB
|
||||
|
||||
## Logs
|
||||
|
||||
**Location**: `/opt/media-downloader/logs/thumbnail_cache_builder.log`
|
||||
|
||||
The cache builder logs detailed progress information:
|
||||
- Total files processed
|
||||
- Thumbnails created
|
||||
- Metadata cached
|
||||
- Files skipped (already cached)
|
||||
- Errors encountered
|
||||
- Processing rate and ETA
|
||||
|
||||
**View logs**:
|
||||
```bash
|
||||
# Live tail
|
||||
tail -f /opt/media-downloader/logs/thumbnail_cache_builder.log
|
||||
|
||||
# Via systemd journal
|
||||
sudo journalctl -u media-cache-builder.service -f
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service Fails to Start
|
||||
|
||||
Check logs:
|
||||
```bash
|
||||
sudo journalctl -xeu media-cache-builder.service
|
||||
```
|
||||
|
||||
Common issues:
|
||||
- Missing dependencies (PIL/Pillow, ffmpeg)
|
||||
- Permission issues accessing media directory
|
||||
- Database corruption
|
||||
|
||||
### Thumbnails Not Appearing
|
||||
|
||||
1. Check if cache builder has run:
|
||||
```bash
|
||||
sudo systemctl status media-cache-builder.service
|
||||
```
|
||||
|
||||
2. Manually trigger rebuild:
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/media/cache/rebuild
|
||||
```
|
||||
|
||||
3. Check cache stats:
|
||||
```bash
|
||||
curl http://localhost:8000/api/media/cache/stats
|
||||
```
|
||||
|
||||
### High Memory Usage
|
||||
|
||||
The cache builder can use 900MB-1GB of RAM during operation. This is normal due to image processing. The systemd service runs with low priority and won't impact other services.
|
||||
|
||||
To reduce memory usage, you can:
|
||||
- Reduce the batch size (modify script)
|
||||
- Run manually during off-peak hours instead of using timer
|
||||
|
||||
### Corrupted or Invalid Images
|
||||
|
||||
Some files may fail to process (shown in error logs). This is normal for:
|
||||
- Corrupted downloads
|
||||
- Unsupported formats
|
||||
- Incomplete files
|
||||
|
||||
These errors don't stop the cache builder from processing other files.
|
||||
|
||||
## Integration with Frontend
|
||||
|
||||
The frontend automatically:
|
||||
- Uses cached thumbnails when available
|
||||
- Falls back to on-demand generation if cache miss
|
||||
- Shows resolution from cache in lightbox (no need to load image first)
|
||||
|
||||
No frontend changes are required - caching is transparent to users.
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
Potential improvements:
|
||||
- Progressive thumbnail generation (prioritize recently viewed files)
|
||||
- Cleanup of thumbnails for deleted files
|
||||
- Configurable thumbnail sizes
|
||||
- Batch processing with configurable batch sizes
|
||||
- Real-time generation triggered by downloads
|
||||
- Cache warming based on user access patterns
|
||||
Reference in New Issue
Block a user