279 lines
6.6 KiB
Markdown
279 lines
6.6 KiB
Markdown
# Media Cache Builder
|
|
|
|
## Overview
|
|
|
|
The Media Cache Builder is a background service that pre-generates thumbnails and caches metadata for all media files in the system. This significantly improves performance by:
|
|
|
|
- **Pre-generating thumbnails**: Thumbnails are created in advance rather than on-demand when viewing media
|
|
- **Caching metadata**: Resolution, file size, duration, and format information is extracted and cached
|
|
- **Reducing API latency**: Media gallery and downloads pages load much faster with cached data
|
|
|
|
## Components
|
|
|
|
### 1. Background Worker Script
|
|
|
|
**Location**: `/opt/media-downloader/modules/thumbnail_cache_builder.py`
|
|
|
|
This Python script scans all media files in `/opt/immich/md` and:
|
|
- Generates 300x300 pixel thumbnails for images and videos
|
|
- Extracts metadata (width, height, duration, format)
|
|
- Stores thumbnails in `/opt/media-downloader/database/thumbnails.db`
|
|
- Stores metadata in `/opt/media-downloader/database/media_metadata.db`
|
|
- Skips files that are already cached and haven't been modified
|
|
- Runs with low priority (Nice=19, IOSchedulingClass=idle) to avoid impacting system performance
|
|
|
|
### 2. Systemd Service
|
|
|
|
**Location**: `/etc/systemd/system/media-cache-builder.service`
|
|
|
|
A oneshot systemd service that runs the cache builder script.
|
|
|
|
**Resource Limits**:
|
|
- CPU quota: 50% (limited to prevent high CPU usage)
|
|
- I/O scheduling: idle priority
|
|
- Nice level: 19 (lowest CPU priority)
|
|
|
|
### 3. Systemd Timer
|
|
|
|
**Location**: `/etc/systemd/system/media-cache-builder.timer`
|
|
|
|
Automatically runs the cache builder daily at 3:00 AM with a randomized delay of up to 30 minutes.
|
|
|
|
**Schedule**:
|
|
- Daily at 3:00 AM
|
|
- Persistent (runs missed timers on boot)
|
|
- Random delay: 0-30 minutes
|
|
|
|
## API Endpoints
|
|
|
|
### Get Cached Metadata
|
|
|
|
```
|
|
GET /api/media/metadata?file_path=/path/to/file
|
|
```
|
|
|
|
Returns cached metadata for a media file:
|
|
```json
|
|
{
|
|
"file_path": "/opt/immich/md/instagram/user/image.jpg",
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"file_size": 245678,
|
|
"duration": null,
|
|
"format": "JPEG",
|
|
"cached": true,
|
|
"cached_at": "2025-10-30T22:36:45.123"
|
|
}
|
|
```
|
|
|
|
### Trigger Cache Rebuild
|
|
|
|
```
|
|
POST /api/media/cache/rebuild
|
|
```
|
|
|
|
Manually triggers a cache rebuild in the background:
|
|
```json
|
|
{
|
|
"success": true,
|
|
"message": "Cache rebuild started in background"
|
|
}
|
|
```
|
|
|
|
### Get Cache Statistics
|
|
|
|
```
|
|
GET /api/media/cache/stats
|
|
```
|
|
|
|
Returns statistics about the cache:
|
|
```json
|
|
{
|
|
"thumbnails": {
|
|
"exists": true,
|
|
"count": 2126,
|
|
"size_bytes": 52428800
|
|
},
|
|
"metadata": {
|
|
"exists": true,
|
|
"count": 2126,
|
|
"size_bytes": 204800
|
|
}
|
|
}
|
|
```
|
|
|
|
## Manual Usage
|
|
|
|
### Run Cache Builder Manually
|
|
|
|
```bash
|
|
# Run directly
|
|
sudo /usr/bin/python3 /opt/media-downloader/modules/thumbnail_cache_builder.py
|
|
|
|
# Or via systemd
|
|
sudo systemctl start media-cache-builder.service
|
|
```
|
|
|
|
### Check Service Status
|
|
|
|
```bash
|
|
# Check if timer is active
|
|
sudo systemctl status media-cache-builder.timer
|
|
|
|
# View logs
|
|
sudo journalctl -u media-cache-builder.service -f
|
|
|
|
# Check when next run is scheduled
|
|
systemctl list-timers media-cache-builder.timer
|
|
```
|
|
|
|
### Enable/Disable Automatic Runs
|
|
|
|
```bash
|
|
# Disable daily automatic runs
|
|
sudo systemctl stop media-cache-builder.timer
|
|
sudo systemctl disable media-cache-builder.timer
|
|
|
|
# Re-enable daily automatic runs
|
|
sudo systemctl enable media-cache-builder.timer
|
|
sudo systemctl start media-cache-builder.timer
|
|
```
|
|
|
|
## Database Schema
|
|
|
|
### Thumbnails Database
|
|
|
|
**Location**: `/opt/media-downloader/database/thumbnails.db`
|
|
|
|
```sql
|
|
CREATE TABLE thumbnails (
|
|
file_hash TEXT PRIMARY KEY,
|
|
file_path TEXT NOT NULL,
|
|
thumbnail_data BLOB NOT NULL,
|
|
created_at TEXT,
|
|
file_mtime REAL
|
|
);
|
|
CREATE INDEX idx_file_path ON thumbnails(file_path);
|
|
```
|
|
|
|
### Metadata Database
|
|
|
|
**Location**: `/opt/media-downloader/database/media_metadata.db`
|
|
|
|
```sql
|
|
CREATE TABLE media_metadata (
|
|
file_hash TEXT PRIMARY KEY,
|
|
file_path TEXT NOT NULL,
|
|
width INTEGER,
|
|
height INTEGER,
|
|
file_size INTEGER,
|
|
duration REAL,
|
|
format TEXT,
|
|
created_at TEXT,
|
|
file_mtime REAL
|
|
);
|
|
CREATE INDEX idx_meta_file_path ON media_metadata(file_path);
|
|
```
|
|
|
|
## Performance
|
|
|
|
### Typical Performance
|
|
|
|
- **Processing rate**: 15-25 files/second (varies by file size and type)
|
|
- **Memory usage**: ~900MB - 1GB during operation
|
|
- **CPU usage**: Limited to 50% of one core
|
|
- **I/O priority**: Idle (won't interfere with normal operations)
|
|
|
|
### For 2,000 files:
|
|
- **Time**: ~2-3 minutes
|
|
- **Thumbnail cache size**: ~50-100MB
|
|
- **Metadata cache size**: ~200-500KB
|
|
|
|
## Logs
|
|
|
|
**Location**: `/opt/media-downloader/logs/thumbnail_cache_builder.log`
|
|
|
|
The cache builder logs detailed progress information:
|
|
- Total files processed
|
|
- Thumbnails created
|
|
- Metadata cached
|
|
- Files skipped (already cached)
|
|
- Errors encountered
|
|
- Processing rate and ETA
|
|
|
|
**View logs**:
|
|
```bash
|
|
# Live tail
|
|
tail -f /opt/media-downloader/logs/thumbnail_cache_builder.log
|
|
|
|
# Via systemd journal
|
|
sudo journalctl -u media-cache-builder.service -f
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### Service Fails to Start
|
|
|
|
Check logs:
|
|
```bash
|
|
sudo journalctl -xeu media-cache-builder.service
|
|
```
|
|
|
|
Common issues:
|
|
- Missing dependencies (PIL/Pillow, ffmpeg)
|
|
- Permission issues accessing media directory
|
|
- Database corruption
|
|
|
|
### Thumbnails Not Appearing
|
|
|
|
1. Check if cache builder has run:
|
|
```bash
|
|
sudo systemctl status media-cache-builder.service
|
|
```
|
|
|
|
2. Manually trigger rebuild:
|
|
```bash
|
|
curl -X POST http://localhost:8000/api/media/cache/rebuild
|
|
```
|
|
|
|
3. Check cache stats:
|
|
```bash
|
|
curl http://localhost:8000/api/media/cache/stats
|
|
```
|
|
|
|
### High Memory Usage
|
|
|
|
The cache builder can use 900MB-1GB of RAM during operation. This is normal due to image processing. The systemd service runs with low priority and won't impact other services.
|
|
|
|
To reduce memory usage, you can:
|
|
- Reduce the batch size (modify script)
|
|
- Run manually during off-peak hours instead of using timer
|
|
|
|
### Corrupted or Invalid Images
|
|
|
|
Some files may fail to process (shown in error logs). This is normal for:
|
|
- Corrupted downloads
|
|
- Unsupported formats
|
|
- Incomplete files
|
|
|
|
These errors don't stop the cache builder from processing other files.
|
|
|
|
## Integration with Frontend
|
|
|
|
The frontend automatically:
|
|
- Uses cached thumbnails when available
|
|
- Falls back to on-demand generation if cache miss
|
|
- Shows resolution from cache in lightbox (no need to load image first)
|
|
|
|
No frontend changes are required - caching is transparent to users.
|
|
|
|
## Future Enhancements
|
|
|
|
Potential improvements:
|
|
- Progressive thumbnail generation (prioritize recently viewed files)
|
|
- Cleanup of thumbnails for deleted files
|
|
- Configurable thumbnail sizes
|
|
- Batch processing with configurable batch sizes
|
|
- Real-time generation triggered by downloads
|
|
- Cache warming based on user access patterns
|