media-downloader/docs/UNIVERSAL_LOGGING.md

# Universal Logging System

## Overview

The universal logging system provides consistent, rotated logging across all Media Downloader components with automatic cleanup of old logs.

## Features

- ✅ **Consistent Format**: All components use the same log format
- ✅ **Automatic Rotation**: Logs rotate daily at midnight
- ✅ **Automatic Cleanup**: Logs older than 7 days are automatically deleted
- ✅ **Separate Log Files**: Each component gets its own log file
- ✅ **Flexible Levels**: Support for DEBUG, INFO, WARNING, ERROR, CRITICAL, SUCCESS
- ✅ **Module Tagging**: Messages tagged with module name for easy filtering

## Log Format

```
2025-11-13 10:30:00 [MediaDownloader.ComponentName] [Module] [LEVEL] message
```

Example:
```
2025-11-13 10:30:00 [MediaDownloader.API] [Core] [INFO] Server started on port 8000
2025-11-13 10:30:05 [MediaDownloader.Scheduler] [Instagram] [SUCCESS] Downloaded 5 new items
```

## Usage

### Basic Usage

```python
from modules.universal_logger import get_logger

# Get logger for your component
logger = get_logger('ComponentName')

# Log messages
logger.info("Application started", module="Core")
logger.debug("Processing item", module="Instagram")
logger.warning("Retry attempt", module="Network")
logger.error("Failed to connect", module="API")
logger.success("Operation completed", module="Core")
```

### Component Examples

#### 1. API Server (api.py)

```python
from modules.universal_logger import get_logger

# Initialize logger
logger = get_logger('API')

# Log startup
logger.info("Starting API server", module="Core")

# Log requests
@app.post("/api/endpoint")
async def endpoint():
    logger.info(f"Processing request", module="Endpoint")
    try:
        # ... processing ...
        logger.success("Request completed", module="Endpoint")
        return {"success": True}
    except Exception as e:
        logger.error(f"Request failed: {e}", module="Endpoint")
        raise
```

#### 2. Scheduler (scheduler.py)

```python
from modules.universal_logger import get_logger

class DownloadScheduler:
    def __init__(self):
        # Replace log_callback with universal logger
        self.logger = get_logger('Scheduler')

    def run(self):
        self.logger.info("Scheduler started", module="Core")

        for task in self.tasks:
            self.logger.debug(f"Processing task: {task}", module="Task")
            # ... task processing ...
            self.logger.success(f"Task completed: {task}", module="Task")
```

#### 3. Download Modules (instagram_module.py, forum_module.py, etc.)

```python
from modules.universal_logger import get_logger

class InstagramModule:
    def __init__(self):
        self.logger = get_logger('Instagram')

    def download(self, username):
        self.logger.info(f"Starting download for {username}", module="Download")

        try:
            # ... download logic ...
            self.logger.success(f"Downloaded media for {username}", module="Download")
        except Exception as e:
            self.logger.error(f"Download failed: {e}", module="Download")
```

#### 4. Using with Existing log_callback Pattern

For modules that use `log_callback`, you can get a compatible callback:

```python
from modules.universal_logger import get_logger

logger = get_logger('MediaDownloader')

# Get callback compatible with existing signature
log_callback = logger.get_callback()

# Pass to modules expecting log_callback
scheduler = DownloadScheduler(log_callback=log_callback)
instagram = InstagramModule(log_callback=log_callback)
```

### Advanced Configuration

```python
from modules.universal_logger import get_logger

# Custom configuration
logger = get_logger(
    component_name='MyComponent',
    log_dir='/custom/log/path',      # Custom log directory
    retention_days=14,                # Keep logs for 14 days
    console_level='DEBUG',            # Show DEBUG on console
    file_level='DEBUG'                # Save DEBUG to file
)
```

### Multi-Module Logging

Within a single component, you can use different module tags:

```python
logger = get_logger('API')

# Different modules
logger.info("Server started", module="Core")
logger.info("User authenticated", module="Auth")
logger.info("Database connected", module="Database")
logger.info("Request received", module="HTTP")
```

## Log Files

### Location

All logs are stored in: `/opt/media-downloader/logs/`

### File Naming

- Current log: `{component}.log`
- Rotated logs: `{component}.log.{YYYYMMDD}`

Examples:
- `api.log` - Current API logs
- `api.log.20251113` - API logs from Nov 13, 2025
- `scheduler.log` - Current scheduler logs
- `mediadownloader.log` - Main application logs

### Rotation Schedule

- **When**: Daily at midnight (00:00)
- **Retention**: 7 days
- **Automatic Cleanup**: Logs older than 7 days are deleted automatically

## Component List

Recommended component names for consistency:

| Component | Name | Log File |
|-----------|------|----------|
| API Server | `API` | `api.log` |
| Frontend Dev Server | `Frontend` | `frontend.log` |
| Scheduler | `Scheduler` | `scheduler.log` |
| Main Downloader | `MediaDownloader` | `mediadownloader.log` |
| Face Recognition | `FaceRecognition` | `facerecognition.log` |
| Cache Builder | `CacheBuilder` | `cachebuilder.log` |
| Instagram Module | `Instagram` | `instagram.log` |
| TikTok Module | `TikTok` | `tiktok.log` |
| Forum Module | `Forum` | `forum.log` |

## Migration Guide

### Migrating from Old Logging

**Before:**
```python
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

fh = logging.FileHandler('my.log')
fh.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
logger.addHandler(fh)

logger.info("Some message")
```

**After:**
```python
from modules.universal_logger import get_logger

logger = get_logger('MyComponent')

logger.info("Some message", module="Core")
```

### Migrating from log_callback Pattern

**Before:**
```python
def my_callback(message, level='INFO'):
    print(f"[{level}] {message}")

module = SomeModule(log_callback=my_callback)
```

**After:**
```python
from modules.universal_logger import get_logger

logger = get_logger('MyComponent')
module = SomeModule(log_callback=logger.get_callback())
```

## Log Cleanup

### Automatic Cleanup

Logs are automatically cleaned up on logger initialization. The system:
1. Checks for log files older than `retention_days`
2. Deletes old files automatically
3. Logs cleanup activity to DEBUG level

### Manual Cleanup

To manually clean all logs older than 7 days:

```bash
find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete
```

### Cron Job (Optional)

Add daily cleanup cron job:

```bash
# Add to root crontab
0 0 * * * find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete
```

## Troubleshooting

### Logs Not Rotating

**Issue**: Logs grow indefinitely
**Solution**: Check that logger is initialized with `get_logger()`, not raw `logging` module

### Old Logs Not Cleaning Up

**Issue**: Logs older than 7 days still present
**Solution**:
1. Check file permissions on log directory
2. Restart the component to trigger cleanup
3. Run manual cleanup command

### Missing Log Entries

**Issue**: Some messages not appearing in logs
**Solution**:
1. Check console_level and file_level settings
2. Ensure module tag is passed: `logger.info("msg", module="Name")`
3. Verify log file permissions

### Multiple Log Entries

**Issue**: Each log line appears multiple times
**Solution**: Logger instantiated multiple times. Use `get_logger()` singleton pattern

## Performance

- **Overhead**: Minimal (<1ms per log entry)
- **File I/O**: Buffered writes, minimal disk impact
- **Rotation**: Happens at midnight, zero runtime impact
- **Cleanup**: Only runs on logger initialization

## Best Practices

1. **Use Singleton**: Always use `get_logger()` not `UniversalLogger()`
2. **Module Tags**: Always specify module parameter
3. **Log Levels**:
   - DEBUG: Verbose debugging information
   - INFO: General informational messages
   - WARNING: Warning messages, recoverable issues
   - ERROR: Error messages, operation failed
   - CRITICAL: Critical errors, system may fail
   - SUCCESS: Successful operations (maps to INFO)
4. **Message Format**: Be concise but descriptive
5. **Sensitive Data**: Never log passwords, tokens, or PII

## Examples

### Complete API Integration

```python
#!/usr/bin/env python3
from fastapi import FastAPI
from modules.universal_logger import get_logger

# Initialize logger
logger = get_logger('API')

app = FastAPI()

@app.on_event("startup")
async def startup():
    logger.info("API server starting", module="Core")
    logger.info("Connecting to database", module="Database")
    # ... startup tasks ...
    logger.success("API server ready", module="Core")

@app.get("/api/data")
async def get_data():
    logger.debug("Processing data request", module="HTTP")
    try:
        data = fetch_data()
        logger.success(f"Returned {len(data)} items", module="HTTP")
        return data
    except Exception as e:
        logger.error(f"Data fetch failed: {e}", module="HTTP")
        raise

if __name__ == "__main__":
    import uvicorn
    logger.info("Starting uvicorn", module="Core")
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

### Complete Scheduler Integration

```python
#!/usr/bin/env python3
from modules.universal_logger import get_logger
from modules.scheduler import DownloadScheduler

# Initialize logger
logger = get_logger('Scheduler')

# Create scheduler with logger callback
scheduler = DownloadScheduler(log_callback=logger.get_callback())

# Log scheduler activity
logger.info("Scheduler initialized", module="Core")

# Start scheduler
scheduler.start()
logger.success("Scheduler started successfully", module="Core")
```

## Version

- **Module**: modules/universal_logger.py
- **Added**: Version 6.27.0
- **Last Updated**: 2025-11-13