Todd
378
docs/UNIVERSAL_LOGGING.md
Normal file
378
docs/UNIVERSAL_LOGGING.md
Normal file
@@ -0,0 +1,378 @@
|
||||
# Universal Logging System
|
||||
|
||||
## Overview
|
||||
|
||||
The universal logging system provides consistent, rotated logging across all Media Downloader components with automatic cleanup of old logs.
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Consistent Format**: All components use the same log format
|
||||
- ✅ **Automatic Rotation**: Logs rotate daily at midnight
|
||||
- ✅ **Automatic Cleanup**: Logs older than 7 days are automatically deleted
|
||||
- ✅ **Separate Log Files**: Each component gets its own log file
|
||||
- ✅ **Flexible Levels**: Support for DEBUG, INFO, WARNING, ERROR, CRITICAL, SUCCESS
|
||||
- ✅ **Module Tagging**: Messages tagged with module name for easy filtering
|
||||
|
||||
## Log Format
|
||||
|
||||
```
|
||||
2025-11-13 10:30:00 [MediaDownloader.ComponentName] [Module] [LEVEL] message
|
||||
```
|
||||
|
||||
Example:
|
||||
```
|
||||
2025-11-13 10:30:00 [MediaDownloader.API] [Core] [INFO] Server started on port 8000
|
||||
2025-11-13 10:30:05 [MediaDownloader.Scheduler] [Instagram] [SUCCESS] Downloaded 5 new items
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
# Get logger for your component
|
||||
logger = get_logger('ComponentName')
|
||||
|
||||
# Log messages
|
||||
logger.info("Application started", module="Core")
|
||||
logger.debug("Processing item", module="Instagram")
|
||||
logger.warning("Retry attempt", module="Network")
|
||||
logger.error("Failed to connect", module="API")
|
||||
logger.success("Operation completed", module="Core")
|
||||
```
|
||||
|
||||
### Component Examples
|
||||
|
||||
#### 1. API Server (api.py)
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
# Initialize logger
|
||||
logger = get_logger('API')
|
||||
|
||||
# Log startup
|
||||
logger.info("Starting API server", module="Core")
|
||||
|
||||
# Log requests
|
||||
@app.post("/api/endpoint")
|
||||
async def endpoint():
|
||||
logger.info(f"Processing request", module="Endpoint")
|
||||
try:
|
||||
# ... processing ...
|
||||
logger.success("Request completed", module="Endpoint")
|
||||
return {"success": True}
|
||||
except Exception as e:
|
||||
logger.error(f"Request failed: {e}", module="Endpoint")
|
||||
raise
|
||||
```
|
||||
|
||||
#### 2. Scheduler (scheduler.py)
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
class DownloadScheduler:
|
||||
def __init__(self):
|
||||
# Replace log_callback with universal logger
|
||||
self.logger = get_logger('Scheduler')
|
||||
|
||||
def run(self):
|
||||
self.logger.info("Scheduler started", module="Core")
|
||||
|
||||
for task in self.tasks:
|
||||
self.logger.debug(f"Processing task: {task}", module="Task")
|
||||
# ... task processing ...
|
||||
self.logger.success(f"Task completed: {task}", module="Task")
|
||||
```
|
||||
|
||||
#### 3. Download Modules (instagram_module.py, forum_module.py, etc.)
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
class InstagramModule:
|
||||
def __init__(self):
|
||||
self.logger = get_logger('Instagram')
|
||||
|
||||
def download(self, username):
|
||||
self.logger.info(f"Starting download for {username}", module="Download")
|
||||
|
||||
try:
|
||||
# ... download logic ...
|
||||
self.logger.success(f"Downloaded media for {username}", module="Download")
|
||||
except Exception as e:
|
||||
self.logger.error(f"Download failed: {e}", module="Download")
|
||||
```
|
||||
|
||||
#### 4. Using with Existing log_callback Pattern
|
||||
|
||||
For modules that use `log_callback`, you can get a compatible callback:
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
logger = get_logger('MediaDownloader')
|
||||
|
||||
# Get callback compatible with existing signature
|
||||
log_callback = logger.get_callback()
|
||||
|
||||
# Pass to modules expecting log_callback
|
||||
scheduler = DownloadScheduler(log_callback=log_callback)
|
||||
instagram = InstagramModule(log_callback=log_callback)
|
||||
```
|
||||
|
||||
### Advanced Configuration
|
||||
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
# Custom configuration
|
||||
logger = get_logger(
|
||||
component_name='MyComponent',
|
||||
log_dir='/custom/log/path', # Custom log directory
|
||||
retention_days=14, # Keep logs for 14 days
|
||||
console_level='DEBUG', # Show DEBUG on console
|
||||
file_level='DEBUG' # Save DEBUG to file
|
||||
)
|
||||
```
|
||||
|
||||
### Multi-Module Logging
|
||||
|
||||
Within a single component, you can use different module tags:
|
||||
|
||||
```python
|
||||
logger = get_logger('API')
|
||||
|
||||
# Different modules
|
||||
logger.info("Server started", module="Core")
|
||||
logger.info("User authenticated", module="Auth")
|
||||
logger.info("Database connected", module="Database")
|
||||
logger.info("Request received", module="HTTP")
|
||||
```
|
||||
|
||||
## Log Files
|
||||
|
||||
### Location
|
||||
|
||||
All logs are stored in: `/opt/media-downloader/logs/`
|
||||
|
||||
### File Naming
|
||||
|
||||
- Current log: `{component}.log`
|
||||
- Rotated logs: `{component}.log.{YYYYMMDD}`
|
||||
|
||||
Examples:
|
||||
- `api.log` - Current API logs
|
||||
- `api.log.20251113` - API logs from Nov 13, 2025
|
||||
- `scheduler.log` - Current scheduler logs
|
||||
- `mediadownloader.log` - Main application logs
|
||||
|
||||
### Rotation Schedule
|
||||
|
||||
- **When**: Daily at midnight (00:00)
|
||||
- **Retention**: 7 days
|
||||
- **Automatic Cleanup**: Logs older than 7 days are deleted automatically
|
||||
|
||||
## Component List
|
||||
|
||||
Recommended component names for consistency:
|
||||
|
||||
| Component | Name | Log File |
|
||||
|-----------|------|----------|
|
||||
| API Server | `API` | `api.log` |
|
||||
| Frontend Dev Server | `Frontend` | `frontend.log` |
|
||||
| Scheduler | `Scheduler` | `scheduler.log` |
|
||||
| Main Downloader | `MediaDownloader` | `mediadownloader.log` |
|
||||
| Face Recognition | `FaceRecognition` | `facerecognition.log` |
|
||||
| Cache Builder | `CacheBuilder` | `cachebuilder.log` |
|
||||
| Instagram Module | `Instagram` | `instagram.log` |
|
||||
| TikTok Module | `TikTok` | `tiktok.log` |
|
||||
| Forum Module | `Forum` | `forum.log` |
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### Migrating from Old Logging
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
fh = logging.FileHandler('my.log')
|
||||
fh.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
|
||||
logger.addHandler(fh)
|
||||
|
||||
logger.info("Some message")
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
logger = get_logger('MyComponent')
|
||||
|
||||
logger.info("Some message", module="Core")
|
||||
```
|
||||
|
||||
### Migrating from log_callback Pattern
|
||||
|
||||
**Before:**
|
||||
```python
|
||||
def my_callback(message, level='INFO'):
|
||||
print(f"[{level}] {message}")
|
||||
|
||||
module = SomeModule(log_callback=my_callback)
|
||||
```
|
||||
|
||||
**After:**
|
||||
```python
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
logger = get_logger('MyComponent')
|
||||
module = SomeModule(log_callback=logger.get_callback())
|
||||
```
|
||||
|
||||
## Log Cleanup
|
||||
|
||||
### Automatic Cleanup
|
||||
|
||||
Logs are automatically cleaned up on logger initialization. The system:
|
||||
1. Checks for log files older than `retention_days`
|
||||
2. Deletes old files automatically
|
||||
3. Logs cleanup activity to DEBUG level
|
||||
|
||||
### Manual Cleanup
|
||||
|
||||
To manually clean all logs older than 7 days:
|
||||
|
||||
```bash
|
||||
find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete
|
||||
```
|
||||
|
||||
### Cron Job (Optional)
|
||||
|
||||
Add daily cleanup cron job:
|
||||
|
||||
```bash
|
||||
# Add to root crontab
|
||||
0 0 * * * find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Logs Not Rotating
|
||||
|
||||
**Issue**: Logs grow indefinitely
|
||||
**Solution**: Check that logger is initialized with `get_logger()`, not raw `logging` module
|
||||
|
||||
### Old Logs Not Cleaning Up
|
||||
|
||||
**Issue**: Logs older than 7 days still present
|
||||
**Solution**:
|
||||
1. Check file permissions on log directory
|
||||
2. Restart the component to trigger cleanup
|
||||
3. Run manual cleanup command
|
||||
|
||||
### Missing Log Entries
|
||||
|
||||
**Issue**: Some messages not appearing in logs
|
||||
**Solution**:
|
||||
1. Check console_level and file_level settings
|
||||
2. Ensure module tag is passed: `logger.info("msg", module="Name")`
|
||||
3. Verify log file permissions
|
||||
|
||||
### Multiple Log Entries
|
||||
|
||||
**Issue**: Each log line appears multiple times
|
||||
**Solution**: Logger instantiated multiple times. Use `get_logger()` singleton pattern
|
||||
|
||||
## Performance
|
||||
|
||||
- **Overhead**: Minimal (<1ms per log entry)
|
||||
- **File I/O**: Buffered writes, minimal disk impact
|
||||
- **Rotation**: Happens at midnight, zero runtime impact
|
||||
- **Cleanup**: Only runs on logger initialization
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use Singleton**: Always use `get_logger()` not `UniversalLogger()`
|
||||
2. **Module Tags**: Always specify module parameter
|
||||
3. **Log Levels**:
|
||||
- DEBUG: Verbose debugging information
|
||||
- INFO: General informational messages
|
||||
- WARNING: Warning messages, recoverable issues
|
||||
- ERROR: Error messages, operation failed
|
||||
- CRITICAL: Critical errors, system may fail
|
||||
- SUCCESS: Successful operations (maps to INFO)
|
||||
4. **Message Format**: Be concise but descriptive
|
||||
5. **Sensitive Data**: Never log passwords, tokens, or PII
|
||||
|
||||
## Examples
|
||||
|
||||
### Complete API Integration
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
from fastapi import FastAPI
|
||||
from modules.universal_logger import get_logger
|
||||
|
||||
# Initialize logger
|
||||
logger = get_logger('API')
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup():
|
||||
logger.info("API server starting", module="Core")
|
||||
logger.info("Connecting to database", module="Database")
|
||||
# ... startup tasks ...
|
||||
logger.success("API server ready", module="Core")
|
||||
|
||||
@app.get("/api/data")
|
||||
async def get_data():
|
||||
logger.debug("Processing data request", module="HTTP")
|
||||
try:
|
||||
data = fetch_data()
|
||||
logger.success(f"Returned {len(data)} items", module="HTTP")
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.error(f"Data fetch failed: {e}", module="HTTP")
|
||||
raise
|
||||
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
logger.info("Starting uvicorn", module="Core")
|
||||
uvicorn.run(app, host="0.0.0.0", port=8000)
|
||||
```
|
||||
|
||||
### Complete Scheduler Integration
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
from modules.universal_logger import get_logger
|
||||
from modules.scheduler import DownloadScheduler
|
||||
|
||||
# Initialize logger
|
||||
logger = get_logger('Scheduler')
|
||||
|
||||
# Create scheduler with logger callback
|
||||
scheduler = DownloadScheduler(log_callback=logger.get_callback())
|
||||
|
||||
# Log scheduler activity
|
||||
logger.info("Scheduler initialized", module="Core")
|
||||
|
||||
# Start scheduler
|
||||
scheduler.start()
|
||||
logger.success("Scheduler started successfully", module="Core")
|
||||
```
|
||||
|
||||
## Version
|
||||
|
||||
- **Module**: modules/universal_logger.py
|
||||
- **Added**: Version 6.27.0
|
||||
- **Last Updated**: 2025-11-13
|
||||
Reference in New Issue
Block a user