Todd
321
docs/REFACTORING_GUIDE.md
Normal file
321
docs/REFACTORING_GUIDE.md
Normal file
@@ -0,0 +1,321 @@
|
||||
# Code Refactoring Guide
|
||||
|
||||
**Version:** 6.52.38
|
||||
**Date:** 2025-12-05
|
||||
**Status:** In Progress - Gradual Migration
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes the code refactoring infrastructure added to address critical technical debt issues identified in the comprehensive code review.
|
||||
|
||||
## Changes Introduced
|
||||
|
||||
### 1. New Core Infrastructure (`web/backend/core/`)
|
||||
|
||||
#### `core/config.py` - Unified Configuration Manager
|
||||
- **Purpose:** Single source of truth for all configuration values
|
||||
- **Benefits:** Eliminates 4+ different config loading approaches
|
||||
- **Usage:**
|
||||
```python
|
||||
from web.backend.core.config import settings
|
||||
|
||||
# Access configuration
|
||||
db_path = settings.DB_PATH
|
||||
timeout = settings.PROCESS_TIMEOUT_MEDIUM
|
||||
media_base = settings.MEDIA_BASE_PATH
|
||||
```
|
||||
|
||||
**Priority Hierarchy:**
|
||||
1. Environment variables (highest)
|
||||
2. .env file values
|
||||
3. Database settings
|
||||
4. Hardcoded defaults (lowest)
|
||||
|
||||
---
|
||||
|
||||
#### `core/exceptions.py` - Custom Exception Classes
|
||||
- **Purpose:** Replace broad `except Exception` with specific exceptions
|
||||
- **Benefits:** Better error handling, debugging, and HTTP status code mapping
|
||||
- **Usage:**
|
||||
```python
|
||||
from web.backend.core.exceptions import (
|
||||
DatabaseError,
|
||||
DatabaseQueryError,
|
||||
RecordNotFoundError,
|
||||
DownloadError,
|
||||
NetworkError,
|
||||
ValidationError,
|
||||
handle_exceptions
|
||||
)
|
||||
|
||||
# Raising specific exceptions
|
||||
if not record:
|
||||
raise RecordNotFoundError("Download not found", {"id": download_id})
|
||||
|
||||
# Using decorator for automatic HTTP conversion
|
||||
@router.get("/api/something")
|
||||
@handle_exceptions
|
||||
async def get_something():
|
||||
# Exceptions automatically converted to proper HTTP responses
|
||||
pass
|
||||
```
|
||||
|
||||
**Exception Mapping:**
|
||||
| Exception | HTTP Status |
|
||||
|-----------|-------------|
|
||||
| ValidationError | 400 |
|
||||
| AuthError | 401 |
|
||||
| InsufficientPermissionsError | 403 |
|
||||
| RecordNotFoundError | 404 |
|
||||
| DuplicateRecordError | 409 |
|
||||
| RateLimitError | 429 |
|
||||
| DatabaseError | 500 |
|
||||
| NetworkError | 502 |
|
||||
| PlatformUnavailableError | 503 |
|
||||
|
||||
---
|
||||
|
||||
#### `core/dependencies.py` - Shared Dependencies
|
||||
- **Purpose:** Centralized FastAPI dependencies for authentication and services
|
||||
- **Benefits:** Consistent auth behavior across all routers
|
||||
- **Usage:**
|
||||
```python
|
||||
from web.backend.core.dependencies import (
|
||||
get_current_user,
|
||||
get_current_user_optional,
|
||||
get_current_user_media,
|
||||
require_admin,
|
||||
get_database,
|
||||
get_settings_manager,
|
||||
get_app_state
|
||||
)
|
||||
|
||||
@router.get("/api/protected")
|
||||
async def protected_endpoint(current_user: Dict = Depends(get_current_user)):
|
||||
# User is authenticated
|
||||
pass
|
||||
|
||||
@router.delete("/api/admin-only")
|
||||
async def admin_endpoint(current_user: Dict = Depends(require_admin)):
|
||||
# User must be admin
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### `core/responses.py` - Standardized Response Format
|
||||
- **Purpose:** Consistent response structure and date handling
|
||||
- **Benefits:** Uniform API contract, ISO 8601 dates everywhere
|
||||
- **Usage:**
|
||||
```python
|
||||
from web.backend.core.responses import (
|
||||
success,
|
||||
error,
|
||||
paginated,
|
||||
to_iso8601,
|
||||
from_iso8601,
|
||||
now_iso8601
|
||||
)
|
||||
|
||||
# Success response
|
||||
return success(data={"id": 1}, message="Created successfully")
|
||||
# Output: {"success": true, "message": "Created successfully", "data": {"id": 1}}
|
||||
|
||||
# Paginated response
|
||||
return paginated(items=results, total=100, page=1, page_size=20)
|
||||
# Output: {"items": [...], "total": 100, "page": 1, "page_size": 20, "has_more": true}
|
||||
|
||||
# Date formatting
|
||||
timestamp = now_iso8601() # "2025-12-05T10:30:00Z"
|
||||
dt = from_iso8601("2025-12-05T10:30:00Z") # datetime object
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Modular Routers (`web/backend/routers/`)
|
||||
|
||||
#### Structure
|
||||
```
|
||||
web/backend/routers/
|
||||
├── __init__.py
|
||||
├── auth.py # Authentication endpoints
|
||||
├── health.py # Health check endpoints
|
||||
└── (more to be added)
|
||||
```
|
||||
|
||||
#### Creating New Routers
|
||||
```python
|
||||
# Example: routers/downloads.py
|
||||
from fastapi import APIRouter, Depends
|
||||
from ..core.dependencies import get_current_user
|
||||
from ..core.exceptions import handle_exceptions
|
||||
|
||||
router = APIRouter(prefix="/api/downloads", tags=["Downloads"])
|
||||
|
||||
@router.get("/")
|
||||
@handle_exceptions
|
||||
async def list_downloads(current_user: Dict = Depends(get_current_user)):
|
||||
# Implementation
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Pydantic Models (`web/backend/models/`)
|
||||
|
||||
#### `models/api_models.py`
|
||||
- **Purpose:** Centralized request/response models with validation
|
||||
- **Benefits:** Type safety, automatic validation, documentation
|
||||
- **Usage:**
|
||||
```python
|
||||
from web.backend.models.api_models import (
|
||||
LoginRequest,
|
||||
DownloadResponse,
|
||||
BatchDeleteRequest,
|
||||
PaginatedResponse
|
||||
)
|
||||
|
||||
@router.post("/batch-delete")
|
||||
async def batch_delete(request: BatchDeleteRequest):
|
||||
# request.file_paths is validated as List[str] with min 1 item
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4. Base Instagram Downloader (`modules/instagram/`)
|
||||
|
||||
#### `modules/instagram/base.py`
|
||||
- **Purpose:** Extract common functionality from FastDL, ImgInn, Toolzu modules
|
||||
- **Benefits:** 60-70% code reduction, consistent behavior, easier maintenance
|
||||
|
||||
#### Common Features Extracted:
|
||||
- Cookie management (database and file-based)
|
||||
- FlareSolverr/Cloudflare bypass integration
|
||||
- Rate limiting and batch delays
|
||||
- Browser management (Playwright)
|
||||
- Download tracking
|
||||
- Logging standardization
|
||||
|
||||
#### Usage:
|
||||
```python
|
||||
from modules.instagram.base import BaseInstagramDownloader
|
||||
|
||||
class MyDownloader(BaseInstagramDownloader):
|
||||
SCRAPER_ID = "my_scraper"
|
||||
BASE_URL = "https://example.com"
|
||||
|
||||
def _get_content_urls(self, username, content_type):
|
||||
# Implementation specific to this scraper
|
||||
pass
|
||||
|
||||
def _parse_content(self, html, content_type):
|
||||
# Implementation specific to this scraper
|
||||
pass
|
||||
|
||||
def _extract_download_url(self, item):
|
||||
# Implementation specific to this scraper
|
||||
pass
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Infrastructure (Complete)
|
||||
- [x] Create `core/config.py` - Unified configuration
|
||||
- [x] Create `core/exceptions.py` - Custom exceptions
|
||||
- [x] Create `core/dependencies.py` - Shared dependencies
|
||||
- [x] Create `core/responses.py` - Response standardization
|
||||
- [x] Create `models/api_models.py` - Pydantic models
|
||||
- [x] Create `modules/instagram/base.py` - Base class
|
||||
|
||||
### Phase 2: Router Migration (In Progress)
|
||||
- [x] Create `routers/auth.py`
|
||||
- [x] Create `routers/health.py`
|
||||
- [ ] Create `routers/downloads.py`
|
||||
- [ ] Create `routers/media.py`
|
||||
- [ ] Create `routers/scheduler.py`
|
||||
- [ ] Create `routers/face_recognition.py`
|
||||
- [ ] Create `routers/recycle.py`
|
||||
- [ ] Create `routers/review.py`
|
||||
- [ ] Create `routers/video.py`
|
||||
- [ ] Create remaining routers
|
||||
|
||||
### Phase 3: Module Refactoring (Pending)
|
||||
- [ ] Refactor `fastdl_module.py` to use base class
|
||||
- [ ] Refactor `imginn_module.py` to use base class
|
||||
- [ ] Refactor `toolzu_module.py` to use base class
|
||||
- [ ] Update tests
|
||||
|
||||
### Phase 4: Cleanup (Pending)
|
||||
- [ ] Replace broad exception handlers gradually
|
||||
- [ ] Migrate sync HTTP to async httpx
|
||||
- [ ] Remove deprecated code
|
||||
- [ ] Update documentation
|
||||
|
||||
---
|
||||
|
||||
## Backwards Compatibility
|
||||
|
||||
The new infrastructure is designed for gradual migration:
|
||||
|
||||
1. **api.py remains functional** - The monolithic file continues to work
|
||||
2. **New routers can be added incrementally** - Include in main app as ready
|
||||
3. **Base classes are optional** - Existing modules work unchanged
|
||||
4. **No breaking changes** - All existing API contracts preserved
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
When migrating an endpoint to a router:
|
||||
|
||||
1. Create the router file
|
||||
2. Move endpoint code
|
||||
3. Update imports to use new core modules
|
||||
4. Add `@handle_exceptions` decorator
|
||||
5. Test endpoint manually
|
||||
6. Add unit tests
|
||||
7. Remove from api.py when confident
|
||||
|
||||
---
|
||||
|
||||
## Files Created
|
||||
|
||||
| File | Purpose | Lines |
|
||||
|------|---------|-------|
|
||||
| `web/backend/core/__init__.py` | Core module init | 1 |
|
||||
| `web/backend/core/config.py` | Configuration manager | 95 |
|
||||
| `web/backend/core/exceptions.py` | Custom exceptions | 250 |
|
||||
| `web/backend/core/dependencies.py` | Shared dependencies | 150 |
|
||||
| `web/backend/core/responses.py` | Response formatting | 140 |
|
||||
| `web/backend/routers/__init__.py` | Routers init | 1 |
|
||||
| `web/backend/routers/auth.py` | Auth endpoints | 170 |
|
||||
| `web/backend/routers/health.py` | Health endpoints | 300 |
|
||||
| `web/backend/models/__init__.py` | Models init | 1 |
|
||||
| `web/backend/models/api_models.py` | Pydantic models | 350 |
|
||||
| `web/backend/services/__init__.py` | Services init | 1 |
|
||||
| `modules/instagram/__init__.py` | Instagram module init | 2 |
|
||||
| `modules/instagram/base.py` | Base downloader class | 400 |
|
||||
|
||||
**Total new code:** ~1,860 lines
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Immediate:** Test routers with current api.py
|
||||
2. **Short-term:** Migrate remaining routers gradually
|
||||
3. **Medium-term:** Refactor Instagram modules to use base class
|
||||
4. **Long-term:** Replace all broad exception handlers, add async HTTP
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- `docs/COMPREHENSIVE_CODE_REVIEW.md` - Full code review
|
||||
- `docs/TECHNICAL_DEBT_ANALYSIS.md` - Original technical debt analysis
|
||||
- `docs/FEATURE_ROADMAP_2025.md` - Feature roadmap
|
||||
Reference in New Issue
Block a user