media-downloader/docs/REFACTORING_GUIDE.md

Code Refactoring Guide

Version: 6.52.38 Date: 2025-12-05 Status: In Progress - Gradual Migration

---

Overview

This document describes the code refactoring infrastructure added to address critical technical debt issues identified in the comprehensive code review.

Changes Introduced

1. New Core Infrastructure (web/backend/core/)

core/config.py - Unified Configuration Manager

• Purpose: Single source of truth for all configuration values
• Benefits: Eliminates 4+ different config loading approaches
• Usage:

from web.backend.core.config import settings

# Access configuration
db_path = settings.DB_PATH
timeout = settings.PROCESS_TIMEOUT_MEDIUM
media_base = settings.MEDIA_BASE_PATH

Priority Hierarchy:

1. Environment variables (highest)
2. .env file values
3. Database settings
4. Hardcoded defaults (lowest)

---

core/exceptions.py - Custom Exception Classes

• Purpose: Replace broad except Exception with specific exceptions
• Benefits: Better error handling, debugging, and HTTP status code mapping
• Usage:

from web.backend.core.exceptions import (
    DatabaseError,
    DatabaseQueryError,
    RecordNotFoundError,
    DownloadError,
    NetworkError,
    ValidationError,
    handle_exceptions
)

# Raising specific exceptions
if not record:
    raise RecordNotFoundError("Download not found", {"id": download_id})

# Using decorator for automatic HTTP conversion
@router.get("/api/something")
@handle_exceptions
async def get_something():
    # Exceptions automatically converted to proper HTTP responses
    pass

Exception Mapping:

Exception	HTTP Status
ValidationError	400
AuthError	401
InsufficientPermissionsError	403
RecordNotFoundError	404
DuplicateRecordError	409
RateLimitError	429
DatabaseError	500
NetworkError	502
PlatformUnavailableError	503

---

core/dependencies.py - Shared Dependencies

• Purpose: Centralized FastAPI dependencies for authentication and services
• Benefits: Consistent auth behavior across all routers
• Usage:

from web.backend.core.dependencies import (
    get_current_user,
    get_current_user_optional,
    get_current_user_media,
    require_admin,
    get_database,
    get_settings_manager,
    get_app_state
)

@router.get("/api/protected")
async def protected_endpoint(current_user: Dict = Depends(get_current_user)):
    # User is authenticated
    pass

@router.delete("/api/admin-only")
async def admin_endpoint(current_user: Dict = Depends(require_admin)):
    # User must be admin
    pass

---

core/responses.py - Standardized Response Format

• Purpose: Consistent response structure and date handling
• Benefits: Uniform API contract, ISO 8601 dates everywhere
• Usage:

from web.backend.core.responses import (
    success,
    error,
    paginated,
    to_iso8601,
    from_iso8601,
    now_iso8601
)

# Success response
return success(data={"id": 1}, message="Created successfully")
# Output: {"success": true, "message": "Created successfully", "data": {"id": 1}}

# Paginated response
return paginated(items=results, total=100, page=1, page_size=20)
# Output: {"items": [...], "total": 100, "page": 1, "page_size": 20, "has_more": true}

# Date formatting
timestamp = now_iso8601()  # "2025-12-05T10:30:00Z"
dt = from_iso8601("2025-12-05T10:30:00Z")  # datetime object

---

2. Modular Routers (web/backend/routers/)

Structure

web/backend/routers/
├── __init__.py
├── auth.py          # Authentication endpoints
├── health.py        # Health check endpoints
└── (more to be added)

Creating New Routers

# Example: routers/downloads.py
from fastapi import APIRouter, Depends
from ..core.dependencies import get_current_user
from ..core.exceptions import handle_exceptions

router = APIRouter(prefix="/api/downloads", tags=["Downloads"])

@router.get("/")
@handle_exceptions
async def list_downloads(current_user: Dict = Depends(get_current_user)):
    # Implementation
    pass

---

3. Pydantic Models (web/backend/models/)

models/api_models.py

• Purpose: Centralized request/response models with validation
• Benefits: Type safety, automatic validation, documentation
• Usage:

from web.backend.models.api_models import (
    LoginRequest,
    DownloadResponse,
    BatchDeleteRequest,
    PaginatedResponse
)

@router.post("/batch-delete")
async def batch_delete(request: BatchDeleteRequest):
    # request.file_paths is validated as List[str] with min 1 item
    pass

---

4. Base Instagram Downloader (modules/instagram/)

modules/instagram/base.py

• Purpose: Extract common functionality from FastDL, ImgInn, Toolzu modules
• Benefits: 60-70% code reduction, consistent behavior, easier maintenance

Common Features Extracted:

• Cookie management (database and file-based)
• FlareSolverr/Cloudflare bypass integration
• Rate limiting and batch delays
• Browser management (Playwright)
• Download tracking
• Logging standardization

Usage:

from modules.instagram.base import BaseInstagramDownloader

class MyDownloader(BaseInstagramDownloader):
    SCRAPER_ID = "my_scraper"
    BASE_URL = "https://example.com"

    def _get_content_urls(self, username, content_type):
        # Implementation specific to this scraper
        pass

    def _parse_content(self, html, content_type):
        # Implementation specific to this scraper
        pass

    def _extract_download_url(self, item):
        # Implementation specific to this scraper
        pass

---

Migration Plan

Phase 1: Infrastructure (Complete)

• Create core/config.py - Unified configuration
• Create core/exceptions.py - Custom exceptions
• Create core/dependencies.py - Shared dependencies
• Create core/responses.py - Response standardization
• Create models/api_models.py - Pydantic models
• Create modules/instagram/base.py - Base class

Phase 2: Router Migration (In Progress)

• Create routers/auth.py
• Create routers/health.py
• Create routers/downloads.py
• Create routers/media.py
• Create routers/scheduler.py
• Create routers/face_recognition.py
• Create routers/recycle.py
• Create routers/review.py
• Create routers/video.py
• Create remaining routers

Phase 3: Module Refactoring (Pending)

• Refactor fastdl_module.py to use base class
• Refactor imginn_module.py to use base class
• Refactor toolzu_module.py to use base class
• Update tests

Phase 4: Cleanup (Pending)

• Replace broad exception handlers gradually
• Migrate sync HTTP to async httpx
• Remove deprecated code
• Update documentation

---

Backwards Compatibility

The new infrastructure is designed for gradual migration:

1. api.py remains functional - The monolithic file continues to work
2. New routers can be added incrementally - Include in main app as ready
3. Base classes are optional - Existing modules work unchanged
4. No breaking changes - All existing API contracts preserved

---

Testing

When migrating an endpoint to a router:

1. Create the router file
2. Move endpoint code
3. Update imports to use new core modules
4. Add @handle_exceptions decorator
5. Test endpoint manually
6. Add unit tests
7. Remove from api.py when confident

---

Files Created

File	Purpose	Lines
web/backend/core/__init__.py	Core module init	1
web/backend/core/config.py	Configuration manager	95
web/backend/core/exceptions.py	Custom exceptions	250
web/backend/core/dependencies.py	Shared dependencies	150
web/backend/core/responses.py	Response formatting	140
web/backend/routers/__init__.py	Routers init	1
web/backend/routers/auth.py	Auth endpoints	170
web/backend/routers/health.py	Health endpoints	300
web/backend/models/__init__.py	Models init	1
web/backend/models/api_models.py	Pydantic models	350
web/backend/services/__init__.py	Services init	1
modules/instagram/__init__.py	Instagram module init	2
modules/instagram/base.py	Base downloader class	400

Total new code: ~1,860 lines

---

Next Steps

1. Immediate: Test routers with current api.py
2. Short-term: Migrate remaining routers gradually
3. Medium-term: Refactor Instagram modules to use base class
4. Long-term: Replace all broad exception handlers, add async HTTP

---

Related Documentation

• docs/COMPREHENSIVE_CODE_REVIEW.md - Full code review
• docs/TECHNICAL_DEBT_ANALYSIS.md - Original technical debt analysis
• docs/FEATURE_ROADMAP_2025.md - Feature roadmap