media-downloader/docs/UNIVERSAL_LOGGING.md

Universal Logging System

Overview

The universal logging system provides consistent, rotated logging across all Media Downloader components with automatic cleanup of old logs.

Features

• ✅ Consistent Format: All components use the same log format
• ✅ Automatic Rotation: Logs rotate daily at midnight
• ✅ Automatic Cleanup: Logs older than 7 days are automatically deleted
• ✅ Separate Log Files: Each component gets its own log file
• ✅ Flexible Levels: Support for DEBUG, INFO, WARNING, ERROR, CRITICAL, SUCCESS
• ✅ Module Tagging: Messages tagged with module name for easy filtering

Log Format

2025-11-13 10:30:00 [MediaDownloader.ComponentName] [Module] [LEVEL] message

Example:

2025-11-13 10:30:00 [MediaDownloader.API] [Core] [INFO] Server started on port 8000
2025-11-13 10:30:05 [MediaDownloader.Scheduler] [Instagram] [SUCCESS] Downloaded 5 new items

Usage

Basic Usage

from modules.universal_logger import get_logger

# Get logger for your component
logger = get_logger('ComponentName')

# Log messages
logger.info("Application started", module="Core")
logger.debug("Processing item", module="Instagram")
logger.warning("Retry attempt", module="Network")
logger.error("Failed to connect", module="API")
logger.success("Operation completed", module="Core")

Component Examples

1. API Server (api.py)

from modules.universal_logger import get_logger

# Initialize logger
logger = get_logger('API')

# Log startup
logger.info("Starting API server", module="Core")

# Log requests
@app.post("/api/endpoint")
async def endpoint():
    logger.info(f"Processing request", module="Endpoint")
    try:
        # ... processing ...
        logger.success("Request completed", module="Endpoint")
        return {"success": True}
    except Exception as e:
        logger.error(f"Request failed: {e}", module="Endpoint")
        raise

2. Scheduler (scheduler.py)

from modules.universal_logger import get_logger

class DownloadScheduler:
    def __init__(self):
        # Replace log_callback with universal logger
        self.logger = get_logger('Scheduler')

    def run(self):
        self.logger.info("Scheduler started", module="Core")

        for task in self.tasks:
            self.logger.debug(f"Processing task: {task}", module="Task")
            # ... task processing ...
            self.logger.success(f"Task completed: {task}", module="Task")

3. Download Modules (instagram_module.py, forum_module.py, etc.)

from modules.universal_logger import get_logger

class InstagramModule:
    def __init__(self):
        self.logger = get_logger('Instagram')

    def download(self, username):
        self.logger.info(f"Starting download for {username}", module="Download")

        try:
            # ... download logic ...
            self.logger.success(f"Downloaded media for {username}", module="Download")
        except Exception as e:
            self.logger.error(f"Download failed: {e}", module="Download")

4. Using with Existing log_callback Pattern

For modules that use log_callback, you can get a compatible callback:

from modules.universal_logger import get_logger

logger = get_logger('MediaDownloader')

# Get callback compatible with existing signature
log_callback = logger.get_callback()

# Pass to modules expecting log_callback
scheduler = DownloadScheduler(log_callback=log_callback)
instagram = InstagramModule(log_callback=log_callback)

Advanced Configuration

from modules.universal_logger import get_logger

# Custom configuration
logger = get_logger(
    component_name='MyComponent',
    log_dir='/custom/log/path',      # Custom log directory
    retention_days=14,                # Keep logs for 14 days
    console_level='DEBUG',            # Show DEBUG on console
    file_level='DEBUG'                # Save DEBUG to file
)

Multi-Module Logging

Within a single component, you can use different module tags:

logger = get_logger('API')

# Different modules
logger.info("Server started", module="Core")
logger.info("User authenticated", module="Auth")
logger.info("Database connected", module="Database")
logger.info("Request received", module="HTTP")

Log Files

Location

All logs are stored in: /opt/media-downloader/logs/

File Naming

• Current log: {component}.log
• Rotated logs: {component}.log.{YYYYMMDD}

Examples:

• api.log - Current API logs
• api.log.20251113 - API logs from Nov 13, 2025
• scheduler.log - Current scheduler logs
• mediadownloader.log - Main application logs

Rotation Schedule

• When: Daily at midnight (00:00)
• Retention: 7 days
• Automatic Cleanup: Logs older than 7 days are deleted automatically

Component List

Recommended component names for consistency:

Component	Name	Log File
API Server	API	api.log
Frontend Dev Server	Frontend	frontend.log
Scheduler	Scheduler	scheduler.log
Main Downloader	MediaDownloader	mediadownloader.log
Face Recognition	FaceRecognition	facerecognition.log
Cache Builder	CacheBuilder	cachebuilder.log
Instagram Module	Instagram	instagram.log
TikTok Module	TikTok	tiktok.log
Forum Module	Forum	forum.log

Migration Guide

Migrating from Old Logging

Before:

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

fh = logging.FileHandler('my.log')
fh.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
logger.addHandler(fh)

logger.info("Some message")

After:

from modules.universal_logger import get_logger

logger = get_logger('MyComponent')

logger.info("Some message", module="Core")

Migrating from log_callback Pattern

Before:

def my_callback(message, level='INFO'):
    print(f"[{level}] {message}")

module = SomeModule(log_callback=my_callback)

After:

from modules.universal_logger import get_logger

logger = get_logger('MyComponent')
module = SomeModule(log_callback=logger.get_callback())

Log Cleanup

Automatic Cleanup

Logs are automatically cleaned up on logger initialization. The system:

1. Checks for log files older than retention_days
2. Deletes old files automatically
3. Logs cleanup activity to DEBUG level

Manual Cleanup

To manually clean all logs older than 7 days:

find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete

Cron Job (Optional)

Add daily cleanup cron job:

# Add to root crontab
0 0 * * * find /opt/media-downloader/logs -name "*.log.*" -mtime +7 -delete

Troubleshooting

Logs Not Rotating

Issue: Logs grow indefinitely Solution: Check that logger is initialized with get_logger(), not raw logging module

Old Logs Not Cleaning Up

Issue: Logs older than 7 days still present Solution:

1. Check file permissions on log directory
2. Restart the component to trigger cleanup
3. Run manual cleanup command

Missing Log Entries

Issue: Some messages not appearing in logs Solution:

1. Check console_level and file_level settings
2. Ensure module tag is passed: logger.info("msg", module="Name")
3. Verify log file permissions

Multiple Log Entries

Issue: Each log line appears multiple times Solution: Logger instantiated multiple times. Use get_logger() singleton pattern

Performance

• Overhead: Minimal (<1ms per log entry)
• File I/O: Buffered writes, minimal disk impact
• Rotation: Happens at midnight, zero runtime impact
• Cleanup: Only runs on logger initialization

Best Practices

1. Use Singleton: Always use get_logger() not UniversalLogger()
2. Module Tags: Always specify module parameter
3. Log Levels:
   • DEBUG: Verbose debugging information
   • INFO: General informational messages
   • WARNING: Warning messages, recoverable issues
   • ERROR: Error messages, operation failed
   • CRITICAL: Critical errors, system may fail
   • SUCCESS: Successful operations (maps to INFO)
4. Message Format: Be concise but descriptive
5. Sensitive Data: Never log passwords, tokens, or PII

Examples

Complete API Integration

#!/usr/bin/env python3
from fastapi import FastAPI
from modules.universal_logger import get_logger

# Initialize logger
logger = get_logger('API')

app = FastAPI()

@app.on_event("startup")
async def startup():
    logger.info("API server starting", module="Core")
    logger.info("Connecting to database", module="Database")
    # ... startup tasks ...
    logger.success("API server ready", module="Core")

@app.get("/api/data")
async def get_data():
    logger.debug("Processing data request", module="HTTP")
    try:
        data = fetch_data()
        logger.success(f"Returned {len(data)} items", module="HTTP")
        return data
    except Exception as e:
        logger.error(f"Data fetch failed: {e}", module="HTTP")
        raise

if __name__ == "__main__":
    import uvicorn
    logger.info("Starting uvicorn", module="Core")
    uvicorn.run(app, host="0.0.0.0", port=8000)

Complete Scheduler Integration

#!/usr/bin/env python3
from modules.universal_logger import get_logger
from modules.scheduler import DownloadScheduler

# Initialize logger
logger = get_logger('Scheduler')

# Create scheduler with logger callback
scheduler = DownloadScheduler(log_callback=logger.get_callback())

# Log scheduler activity
logger.info("Scheduler initialized", module="Core")

# Start scheduler
scheduler.start()
logger.success("Scheduler started successfully", module="Core")

Version

• Module: modules/universal_logger.py
• Added: Version 6.27.0
• Last Updated: 2025-11-13