322 lines
8.7 KiB
Markdown
322 lines
8.7 KiB
Markdown
# Code Refactoring Guide
|
|
|
|
**Version:** 6.52.38
|
|
**Date:** 2025-12-05
|
|
**Status:** In Progress - Gradual Migration
|
|
|
|
---
|
|
|
|
## Overview
|
|
|
|
This document describes the code refactoring infrastructure added to address critical technical debt issues identified in the comprehensive code review.
|
|
|
|
## Changes Introduced
|
|
|
|
### 1. New Core Infrastructure (`web/backend/core/`)
|
|
|
|
#### `core/config.py` - Unified Configuration Manager
|
|
- **Purpose:** Single source of truth for all configuration values
|
|
- **Benefits:** Eliminates 4+ different config loading approaches
|
|
- **Usage:**
|
|
```python
|
|
from web.backend.core.config import settings
|
|
|
|
# Access configuration
|
|
db_path = settings.DB_PATH
|
|
timeout = settings.PROCESS_TIMEOUT_MEDIUM
|
|
media_base = settings.MEDIA_BASE_PATH
|
|
```
|
|
|
|
**Priority Hierarchy:**
|
|
1. Environment variables (highest)
|
|
2. .env file values
|
|
3. Database settings
|
|
4. Hardcoded defaults (lowest)
|
|
|
|
---
|
|
|
|
#### `core/exceptions.py` - Custom Exception Classes
|
|
- **Purpose:** Replace broad `except Exception` with specific exceptions
|
|
- **Benefits:** Better error handling, debugging, and HTTP status code mapping
|
|
- **Usage:**
|
|
```python
|
|
from web.backend.core.exceptions import (
|
|
DatabaseError,
|
|
DatabaseQueryError,
|
|
RecordNotFoundError,
|
|
DownloadError,
|
|
NetworkError,
|
|
ValidationError,
|
|
handle_exceptions
|
|
)
|
|
|
|
# Raising specific exceptions
|
|
if not record:
|
|
raise RecordNotFoundError("Download not found", {"id": download_id})
|
|
|
|
# Using decorator for automatic HTTP conversion
|
|
@router.get("/api/something")
|
|
@handle_exceptions
|
|
async def get_something():
|
|
# Exceptions automatically converted to proper HTTP responses
|
|
pass
|
|
```
|
|
|
|
**Exception Mapping:**
|
|
| Exception | HTTP Status |
|
|
|-----------|-------------|
|
|
| ValidationError | 400 |
|
|
| AuthError | 401 |
|
|
| InsufficientPermissionsError | 403 |
|
|
| RecordNotFoundError | 404 |
|
|
| DuplicateRecordError | 409 |
|
|
| RateLimitError | 429 |
|
|
| DatabaseError | 500 |
|
|
| NetworkError | 502 |
|
|
| PlatformUnavailableError | 503 |
|
|
|
|
---
|
|
|
|
#### `core/dependencies.py` - Shared Dependencies
|
|
- **Purpose:** Centralized FastAPI dependencies for authentication and services
|
|
- **Benefits:** Consistent auth behavior across all routers
|
|
- **Usage:**
|
|
```python
|
|
from web.backend.core.dependencies import (
|
|
get_current_user,
|
|
get_current_user_optional,
|
|
get_current_user_media,
|
|
require_admin,
|
|
get_database,
|
|
get_settings_manager,
|
|
get_app_state
|
|
)
|
|
|
|
@router.get("/api/protected")
|
|
async def protected_endpoint(current_user: Dict = Depends(get_current_user)):
|
|
# User is authenticated
|
|
pass
|
|
|
|
@router.delete("/api/admin-only")
|
|
async def admin_endpoint(current_user: Dict = Depends(require_admin)):
|
|
# User must be admin
|
|
pass
|
|
```
|
|
|
|
---
|
|
|
|
#### `core/responses.py` - Standardized Response Format
|
|
- **Purpose:** Consistent response structure and date handling
|
|
- **Benefits:** Uniform API contract, ISO 8601 dates everywhere
|
|
- **Usage:**
|
|
```python
|
|
from web.backend.core.responses import (
|
|
success,
|
|
error,
|
|
paginated,
|
|
to_iso8601,
|
|
from_iso8601,
|
|
now_iso8601
|
|
)
|
|
|
|
# Success response
|
|
return success(data={"id": 1}, message="Created successfully")
|
|
# Output: {"success": true, "message": "Created successfully", "data": {"id": 1}}
|
|
|
|
# Paginated response
|
|
return paginated(items=results, total=100, page=1, page_size=20)
|
|
# Output: {"items": [...], "total": 100, "page": 1, "page_size": 20, "has_more": true}
|
|
|
|
# Date formatting
|
|
timestamp = now_iso8601() # "2025-12-05T10:30:00Z"
|
|
dt = from_iso8601("2025-12-05T10:30:00Z") # datetime object
|
|
```
|
|
|
|
---
|
|
|
|
### 2. Modular Routers (`web/backend/routers/`)
|
|
|
|
#### Structure
|
|
```
|
|
web/backend/routers/
|
|
├── __init__.py
|
|
├── auth.py # Authentication endpoints
|
|
├── health.py # Health check endpoints
|
|
└── (more to be added)
|
|
```
|
|
|
|
#### Creating New Routers
|
|
```python
|
|
# Example: routers/downloads.py
|
|
from fastapi import APIRouter, Depends
|
|
from ..core.dependencies import get_current_user
|
|
from ..core.exceptions import handle_exceptions
|
|
|
|
router = APIRouter(prefix="/api/downloads", tags=["Downloads"])
|
|
|
|
@router.get("/")
|
|
@handle_exceptions
|
|
async def list_downloads(current_user: Dict = Depends(get_current_user)):
|
|
# Implementation
|
|
pass
|
|
```
|
|
|
|
---
|
|
|
|
### 3. Pydantic Models (`web/backend/models/`)
|
|
|
|
#### `models/api_models.py`
|
|
- **Purpose:** Centralized request/response models with validation
|
|
- **Benefits:** Type safety, automatic validation, documentation
|
|
- **Usage:**
|
|
```python
|
|
from web.backend.models.api_models import (
|
|
LoginRequest,
|
|
DownloadResponse,
|
|
BatchDeleteRequest,
|
|
PaginatedResponse
|
|
)
|
|
|
|
@router.post("/batch-delete")
|
|
async def batch_delete(request: BatchDeleteRequest):
|
|
# request.file_paths is validated as List[str] with min 1 item
|
|
pass
|
|
```
|
|
|
|
---
|
|
|
|
### 4. Base Instagram Downloader (`modules/instagram/`)
|
|
|
|
#### `modules/instagram/base.py`
|
|
- **Purpose:** Extract common functionality from FastDL, ImgInn, Toolzu modules
|
|
- **Benefits:** 60-70% code reduction, consistent behavior, easier maintenance
|
|
|
|
#### Common Features Extracted:
|
|
- Cookie management (database and file-based)
|
|
- FlareSolverr/Cloudflare bypass integration
|
|
- Rate limiting and batch delays
|
|
- Browser management (Playwright)
|
|
- Download tracking
|
|
- Logging standardization
|
|
|
|
#### Usage:
|
|
```python
|
|
from modules.instagram.base import BaseInstagramDownloader
|
|
|
|
class MyDownloader(BaseInstagramDownloader):
|
|
SCRAPER_ID = "my_scraper"
|
|
BASE_URL = "https://example.com"
|
|
|
|
def _get_content_urls(self, username, content_type):
|
|
# Implementation specific to this scraper
|
|
pass
|
|
|
|
def _parse_content(self, html, content_type):
|
|
# Implementation specific to this scraper
|
|
pass
|
|
|
|
def _extract_download_url(self, item):
|
|
# Implementation specific to this scraper
|
|
pass
|
|
```
|
|
|
|
---
|
|
|
|
## Migration Plan
|
|
|
|
### Phase 1: Infrastructure (Complete)
|
|
- [x] Create `core/config.py` - Unified configuration
|
|
- [x] Create `core/exceptions.py` - Custom exceptions
|
|
- [x] Create `core/dependencies.py` - Shared dependencies
|
|
- [x] Create `core/responses.py` - Response standardization
|
|
- [x] Create `models/api_models.py` - Pydantic models
|
|
- [x] Create `modules/instagram/base.py` - Base class
|
|
|
|
### Phase 2: Router Migration (In Progress)
|
|
- [x] Create `routers/auth.py`
|
|
- [x] Create `routers/health.py`
|
|
- [ ] Create `routers/downloads.py`
|
|
- [ ] Create `routers/media.py`
|
|
- [ ] Create `routers/scheduler.py`
|
|
- [ ] Create `routers/face_recognition.py`
|
|
- [ ] Create `routers/recycle.py`
|
|
- [ ] Create `routers/review.py`
|
|
- [ ] Create `routers/video.py`
|
|
- [ ] Create remaining routers
|
|
|
|
### Phase 3: Module Refactoring (Pending)
|
|
- [ ] Refactor `fastdl_module.py` to use base class
|
|
- [ ] Refactor `imginn_module.py` to use base class
|
|
- [ ] Refactor `toolzu_module.py` to use base class
|
|
- [ ] Update tests
|
|
|
|
### Phase 4: Cleanup (Pending)
|
|
- [ ] Replace broad exception handlers gradually
|
|
- [ ] Migrate sync HTTP to async httpx
|
|
- [ ] Remove deprecated code
|
|
- [ ] Update documentation
|
|
|
|
---
|
|
|
|
## Backwards Compatibility
|
|
|
|
The new infrastructure is designed for gradual migration:
|
|
|
|
1. **api.py remains functional** - The monolithic file continues to work
|
|
2. **New routers can be added incrementally** - Include in main app as ready
|
|
3. **Base classes are optional** - Existing modules work unchanged
|
|
4. **No breaking changes** - All existing API contracts preserved
|
|
|
|
---
|
|
|
|
## Testing
|
|
|
|
When migrating an endpoint to a router:
|
|
|
|
1. Create the router file
|
|
2. Move endpoint code
|
|
3. Update imports to use new core modules
|
|
4. Add `@handle_exceptions` decorator
|
|
5. Test endpoint manually
|
|
6. Add unit tests
|
|
7. Remove from api.py when confident
|
|
|
|
---
|
|
|
|
## Files Created
|
|
|
|
| File | Purpose | Lines |
|
|
|------|---------|-------|
|
|
| `web/backend/core/__init__.py` | Core module init | 1 |
|
|
| `web/backend/core/config.py` | Configuration manager | 95 |
|
|
| `web/backend/core/exceptions.py` | Custom exceptions | 250 |
|
|
| `web/backend/core/dependencies.py` | Shared dependencies | 150 |
|
|
| `web/backend/core/responses.py` | Response formatting | 140 |
|
|
| `web/backend/routers/__init__.py` | Routers init | 1 |
|
|
| `web/backend/routers/auth.py` | Auth endpoints | 170 |
|
|
| `web/backend/routers/health.py` | Health endpoints | 300 |
|
|
| `web/backend/models/__init__.py` | Models init | 1 |
|
|
| `web/backend/models/api_models.py` | Pydantic models | 350 |
|
|
| `web/backend/services/__init__.py` | Services init | 1 |
|
|
| `modules/instagram/__init__.py` | Instagram module init | 2 |
|
|
| `modules/instagram/base.py` | Base downloader class | 400 |
|
|
|
|
**Total new code:** ~1,860 lines
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Immediate:** Test routers with current api.py
|
|
2. **Short-term:** Migrate remaining routers gradually
|
|
3. **Medium-term:** Refactor Instagram modules to use base class
|
|
4. **Long-term:** Replace all broad exception handlers, add async HTTP
|
|
|
|
---
|
|
|
|
## Related Documentation
|
|
|
|
- `docs/COMPREHENSIVE_CODE_REVIEW.md` - Full code review
|
|
- `docs/TECHNICAL_DEBT_ANALYSIS.md` - Original technical debt analysis
|
|
- `docs/FEATURE_ROADMAP_2025.md` - Feature roadmap
|