# SQLite to PostgreSQL Migration Guide

## Overview

This document provides a comprehensive guide for migrating the media-downloader application from SQLite to PostgreSQL.

### Migration Statistics

| Metric | Count |
|--------|-------|
| Total Tables | 53 |
| Files Requiring Changes | 40+ |
| INSERT OR IGNORE/REPLACE | 60+ occurrences |
| datetime() functions | 50+ occurrences |
| PRAGMA statements | 30+ occurrences |
| AUTOINCREMENT columns | 50+ occurrences |
| GROUP_CONCAT functions | 5 occurrences |
| strftime() functions | 10+ occurrences |

---

## Table of Contents

1. [Schema Changes](#1-schema-changes)
2. [Connection Pool Changes](#2-connection-pool-changes)
3. [SQL Syntax Conversions](#3-sql-syntax-conversions)
4. [File-by-File Changes](#4-file-by-file-changes)
5. [Migration Checklist](#5-migration-checklist)
6. [Data Migration Script](#6-data-migration-script)

---

## 1. Schema Changes

### 1.1 PRIMARY KEY AUTOINCREMENT → SERIAL

**SQLite:**
```sql
id INTEGER PRIMARY KEY AUTOINCREMENT
```

**PostgreSQL:**
```sql
id SERIAL PRIMARY KEY
```

Or for larger tables:
```sql
id BIGSERIAL PRIMARY KEY
```

### 1.2 BOOLEAN Columns

**SQLite** stores booleans as integers (0/1). **PostgreSQL** has native BOOLEAN type.

| SQLite | PostgreSQL |
|--------|------------|
| `has_images BOOLEAN DEFAULT 0` | `has_images BOOLEAN DEFAULT false` |
| `enabled INTEGER DEFAULT 1` | `enabled BOOLEAN DEFAULT true` |
| `active BOOLEAN DEFAULT 1` | `active BOOLEAN DEFAULT true` |

### 1.3 BLOB → BYTEA

**SQLite:**
```sql
thumbnail_data BLOB
```

**PostgreSQL:**
```sql
thumbnail_data BYTEA
```

### 1.4 TEXT/JSON Fields

Consider using PostgreSQL's native JSONB for better query performance:

```sql
-- SQLite
metadata TEXT  -- stores JSON as string

-- PostgreSQL (recommended)
metadata JSONB
```

### 1.5 Singleton Tables (CHECK constraint)

These work identically in both databases - no changes needed:
```sql
id INTEGER PRIMARY KEY CHECK (id = 1)
```

---

## 2. Connection Pool Changes

### 2.1 Current SQLite Pool (unified_database.py)

The `DatabasePool` class needs to be rewritten for PostgreSQL.

**Current SQLite:**
```python
import sqlite3

class DatabasePool:
    def __init__(self, db_path: str, pool_size: int = 20):
        for _ in range(pool_size):
            conn = sqlite3.connect(
                db_path,
                check_same_thread=False,
                timeout=30.0,
                isolation_level=None
            )
            conn.row_factory = sqlite3.Row
            conn.execute("PRAGMA journal_mode=WAL")
            # ... other PRAGMA statements
```

**PostgreSQL Replacement:**
```python
import psycopg2
from psycopg2 import pool
from psycopg2.extras import RealDictCursor

class DatabasePool:
    def __init__(self, dsn: str, pool_size: int = 20):
        self.pool = psycopg2.pool.ThreadedConnectionPool(
            minconn=5,
            maxconn=pool_size,
            dsn=dsn,
            cursor_factory=RealDictCursor
        )

    @contextmanager
    def get_connection(self, for_write=False):
        conn = self.pool.getconn()
        try:
            yield conn
            if for_write:
                conn.commit()
        except Exception:
            conn.rollback()
            raise
        finally:
            self.pool.putconn(conn)
```

### 2.2 Remove All PRAGMA Statements

PRAGMA is SQLite-specific. Remove all instances:

| File | Lines | PRAGMA Statement | Action |
|------|-------|------------------|--------|
| unified_database.py | 82-88 | journal_mode, synchronous, cache_size, etc. | Remove |
| unified_database.py | 128 | wal_checkpoint | Remove |
| unified_database.py | 148-151 | journal_mode, synchronous, busy_timeout | Remove |
| unified_database.py | 197-198 | busy_timeout, journal_mode | Remove |
| unified_database.py | 223-224 | journal_mode, busy_timeout | Remove |
| unified_database.py | 233-236 | journal_mode, busy_timeout, synchronous, foreign_keys | Remove |
| unified_database.py | 616-619 | journal_mode, synchronous, cache_size, temp_store | Remove |
| forum_downloader.py | 1361-1362 | journal_mode, synchronous | Remove |
| thumbnail_cache_builder.py | 59, 201, 232, 260, 273 | journal_mode | Remove |
| media.py | 216 | journal_mode | Remove |
| scheduler.py | 111-113 | journal_mode, busy_timeout, synchronous | Remove |
| universal_logger.py | 204 | busy_timeout | Remove |

**Note:** PRAGMA table_info() can be replaced with PostgreSQL's information_schema:
```sql
-- SQLite
PRAGMA table_info(table_name)

-- PostgreSQL
SELECT column_name, data_type, is_nullable
FROM information_schema.columns
WHERE table_name = 'table_name'
```

---

## 3. SQL Syntax Conversions

### 3.1 INSERT OR IGNORE → ON CONFLICT DO NOTHING

**SQLite:**
```sql
INSERT OR IGNORE INTO table (col1, col2) VALUES (?, ?)
```

**PostgreSQL:**
```sql
INSERT INTO table (col1, col2) VALUES ($1, $2)
ON CONFLICT DO NOTHING
```

Or with explicit conflict target:
```sql
INSERT INTO table (col1, col2) VALUES ($1, $2)
ON CONFLICT (col1) DO NOTHING
```

### 3.2 INSERT OR REPLACE → ON CONFLICT DO UPDATE

**SQLite:**
```sql
INSERT OR REPLACE INTO table (id, col1, col2) VALUES (?, ?, ?)
```

**PostgreSQL:**
```sql
INSERT INTO table (id, col1, col2) VALUES ($1, $2, $3)
ON CONFLICT (id) DO UPDATE SET
    col1 = EXCLUDED.col1,
    col2 = EXCLUDED.col2
```

### 3.3 datetime() Functions

| SQLite | PostgreSQL |
|--------|------------|
| `datetime('now')` | `NOW()` or `CURRENT_TIMESTAMP` |
| `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |
| `datetime('now', '-24 hours')` | `NOW() - INTERVAL '24 hours'` |
| `datetime('now', '+30 days')` | `NOW() + INTERVAL '30 days'` |
| `datetime('now', ? \|\| ' days')` | `NOW() + (INTERVAL '1 day' * $1)` |
| `date('now')` | `CURRENT_DATE` |
| `date('now', '-30 days')` | `CURRENT_DATE - INTERVAL '30 days'` |

### 3.4 strftime() → TO_CHAR() / EXTRACT()

| SQLite | PostgreSQL |
|--------|------------|
| `strftime('%Y', col)` | `TO_CHAR(col, 'YYYY')` or `EXTRACT(YEAR FROM col)` |
| `strftime('%m', col)` | `TO_CHAR(col, 'MM')` or `EXTRACT(MONTH FROM col)` |
| `strftime('%d', col)` | `TO_CHAR(col, 'DD')` or `EXTRACT(DAY FROM col)` |
| `strftime('%H', col)` | `TO_CHAR(col, 'HH24')` or `EXTRACT(HOUR FROM col)` |
| `strftime('%Y-%m-%d', col)` | `TO_CHAR(col, 'YYYY-MM-DD')` |
| `strftime('%Y-W%W', col)` | `TO_CHAR(col, 'IYYY-"W"IW')` |

### 3.5 GROUP_CONCAT() → STRING_AGG()

**SQLite:**
```sql
GROUP_CONCAT(column, ', ')
GROUP_CONCAT(DISTINCT column)
```

**PostgreSQL:**
```sql
STRING_AGG(column, ', ')
STRING_AGG(DISTINCT column::text, ',')
```

### 3.6 IFNULL() → COALESCE()

**SQLite:**
```sql
IFNULL(column, 'default')
```

**PostgreSQL:**
```sql
COALESCE(column, 'default')
```

Note: The codebase already uses COALESCE in most places.

### 3.7 Parameter Placeholders

**SQLite (sqlite3):**
```python
cursor.execute("SELECT * FROM table WHERE id = ?", (id,))
```

**PostgreSQL (psycopg2):**
```python
cursor.execute("SELECT * FROM table WHERE id = %s", (id,))
```

### 3.8 Last Insert ID

**SQLite:**
```python
cursor.execute("INSERT INTO table ...")
id = cursor.lastrowid
```

**PostgreSQL:**
```python
cursor.execute("INSERT INTO table ... RETURNING id")
id = cursor.fetchone()[0]
```

### 3.9 LIKE Case Sensitivity

**SQLite:** LIKE is case-insensitive by default
**PostgreSQL:** LIKE is case-sensitive

```sql
-- SQLite (case-insensitive)
WHERE filename LIKE '%pattern%'

-- PostgreSQL (case-insensitive)
WHERE filename ILIKE '%pattern%'
-- OR
WHERE LOWER(filename) LIKE LOWER('%pattern%')
```

---

## 4. File-by-File Changes

### 4.1 Core Database Module

#### `/opt/media-downloader/modules/unified_database.py`

| Line(s) | Current | Change To | Notes |
|---------|---------|-----------|-------|
| 82-88 | PRAGMA statements | Remove | PostgreSQL doesn't use PRAGMA |
| 128 | PRAGMA wal_checkpoint | Remove | |
| 148-151 | PRAGMA statements | Remove | |
| 197-198 | PRAGMA statements | Remove | |
| 223-224 | PRAGMA statements | Remove | |
| 233-236 | PRAGMA statements | Remove | |
| 241 | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY | |
| 326, 347, 367, etc. | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY | ~50 tables |
| 500 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 510 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 616-619 | PRAGMA statements | Remove | |
| 622-665 | Triggers with datetime('now') | Use NOW() | 4 triggers |
| 807 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 877 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 940 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1116 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1119, 1141, 1151, 1239 | PRAGMA table_info | Use information_schema | |
| 1207 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1309 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1374 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1549-1563 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1806 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 1841 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 2293 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |
| 3176 | INSERT OR IGNORE | ON CONFLICT DO NOTHING | |

### 4.2 Paid Content Module

#### `/opt/media-downloader/modules/paid_content/db_adapter.py`

| Line | Current | Change To |
|------|---------|-----------|
| 132 | `INSERT OR IGNORE INTO paid_content_config` | `ON CONFLICT DO NOTHING` |
| 1346 | `INSERT OR REPLACE INTO paid_content_posts` | `ON CONFLICT DO UPDATE` |
| 1436 | `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |
| 1699 | `INSERT OR IGNORE INTO paid_content_post_tags` | `ON CONFLICT DO NOTHING` |
| 1727 | `INSERT OR IGNORE INTO paid_content_post_tags` | `ON CONFLICT DO NOTHING` |

### 4.3 Forum Module

#### `/opt/media-downloader/modules/forum_db_adapter.py`

| Line | Current | Change To |
|------|---------|-----------|
| 88 | `INSERT OR IGNORE INTO forum_threads` | `ON CONFLICT DO NOTHING` |
| 179 | `INSERT OR REPLACE INTO forum_posts` | `ON CONFLICT DO UPDATE` |
| 252 | `INSERT OR REPLACE INTO search_monitors` | `ON CONFLICT DO UPDATE` |
| 454 | `datetime('now', ? \|\| ' days')` | `NOW() + (INTERVAL '1 day' * $1)` |
| 462 | `datetime('now', ? \|\| ' days')` | `NOW() + (INTERVAL '1 day' * $1)` |
| 470 | `datetime('now')` | `NOW()` |

#### `/opt/media-downloader/modules/forum_downloader.py`

| Line | Current | Change To |
|------|---------|-----------|
| 1324 | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY |
| 1361-1362 | PRAGMA statements | Remove |
| 1373 | `datetime('now', '-90 days')` | `NOW() - INTERVAL '90 days'` |
| 1385 | `datetime('now')` | `NOW()` |
| 1397 | `datetime('now', '-180 days')` | `NOW() - INTERVAL '180 days'` |
| 2608 | `INSERT OR IGNORE INTO threads` | `ON CONFLICT DO NOTHING` |
| 2658 | `INSERT OR IGNORE INTO search_results` | `ON CONFLICT DO NOTHING` |
| 2846 | `INSERT OR REPLACE INTO threads` | `ON CONFLICT DO UPDATE` |
| 2912 | `INSERT OR REPLACE INTO posts` | `ON CONFLICT DO UPDATE` |

### 4.4 Backend Routers

#### `/opt/media-downloader/web/backend/routers/media.py`

| Line | Current | Change To |
|------|---------|-----------|
| 216 | `PRAGMA journal_mode=WAL` | Remove |
| 250 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |
| 318 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |
| 1334, 1338, 1391, 1395 | DATE() functions | Compatible, but review |

#### `/opt/media-downloader/web/backend/routers/video_queue.py`

| Line | Current | Change To |
|------|---------|-----------|
| 410 | `datetime('now', '-24 hours')` | `NOW() - INTERVAL '24 hours'` |
| 546 | `INSERT OR REPLACE INTO settings` | `ON CONFLICT DO UPDATE` |
| 553 | `INSERT OR REPLACE INTO settings` | `ON CONFLICT DO UPDATE` |
| 676 | `INSERT OR REPLACE INTO settings` | `ON CONFLICT DO UPDATE` |
| 720 | `cursor.lastrowid` | Use RETURNING clause |
| 1269 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/web/backend/routers/downloads.py`

| Line | Current | Change To |
|------|---------|-----------|
| 353-354 | `datetime('now', '-1 day')` | `NOW() - INTERVAL '1 day'` |
| 1214 | `datetime('now', '-30 days')` | `NOW() - INTERVAL '30 days'` |
| 1285 | `strftime('%H', download_date)` | `EXTRACT(HOUR FROM download_date)` |
| 1287 | `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |
| 1298-1299 | `datetime('now', '-7/-14 days')` | `NOW() - INTERVAL '...'` |
| 1304 | `datetime('now', '-14 days')` | `NOW() - INTERVAL '14 days'` |

#### `/opt/media-downloader/web/backend/routers/recycle.py`

| Line | Current | Change To |
|------|---------|-----------|
| 611 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/web/backend/routers/appearances.py`

| Line | Current | Change To |
|------|---------|-----------|
| 344 | `GROUP_CONCAT(DISTINCT credit_type)` | `STRING_AGG(DISTINCT credit_type, ',')` |
| 348, 366 | `datetime('now')` | `NOW()` |
| 529 | `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |
| 531 | `datetime('now', '-30 days')` | `NOW() - INTERVAL '30 days'` |
| 552 | `GROUP_CONCAT(DISTINCT credit_type)` | `STRING_AGG(DISTINCT credit_type, ',')` |
| 741-742 | `GROUP_CONCAT(DISTINCT ...)` | `STRING_AGG(DISTINCT ..., ',')` |

#### `/opt/media-downloader/web/backend/routers/celebrity.py`

| Line | Current | Change To |
|------|---------|-----------|
| 623 | `cursor.lastrowid` | Use RETURNING clause |
| 907 | `cursor.lastrowid` | Use RETURNING clause |
| 936-946 | `INSERT OR IGNORE` | `ON CONFLICT DO NOTHING` |
| 948-949 | `cursor.lastrowid` | Use RETURNING clause |
| 1166-1189 | `INSERT OR IGNORE` | `ON CONFLICT DO NOTHING` |

#### `/opt/media-downloader/web/backend/routers/video.py`

| Line | Current | Change To |
|------|---------|-----------|
| 877-880 | `INSERT OR REPLACE INTO video_preview_list` | `ON CONFLICT DO UPDATE` |
| 1610-1612 | `INSERT OR REPLACE INTO settings` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/web/backend/routers/config.py`

| Line | Current | Change To |
|------|---------|-----------|
| 554 | `datetime('now', '-1 day')` | `NOW() - INTERVAL '1 day'` |
| 698 | `INSERT OR IGNORE INTO appearance_config` | `ON CONFLICT DO NOTHING` |

#### `/opt/media-downloader/web/backend/routers/discovery.py`

| Line | Current | Change To |
|------|---------|-----------|
| 833 | `datetime('now', '-1 day')` | `NOW() - INTERVAL '1 day'` |
| 840 | `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |
| 846 | `datetime('now', '-1 day')` | `NOW() - INTERVAL '1 day'` |
| 852 | `datetime('now', '-7 days')` | `NOW() - INTERVAL '7 days'` |

#### `/opt/media-downloader/web/backend/routers/stats.py`

| Line | Current | Change To |
|------|---------|-----------|
| 107-115 | `DATE('now', '-30 days')` | `CURRENT_DATE - INTERVAL '30 days'` |
| 167-170 | `DATE('now', '-7 days')` | `CURRENT_DATE - INTERVAL '7 days'` |

#### `/opt/media-downloader/web/backend/routers/face.py`

| Line | Current | Change To |
|------|---------|-----------|
| 513 | `DATE('now', '-30 days')` | `CURRENT_DATE - INTERVAL '30 days'` |

### 4.5 Other Modules

#### `/opt/media-downloader/modules/download_manager.py`

| Line | Current | Change To |
|------|---------|-----------|
| 138 | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY |
| 794 | `INSERT OR REPLACE INTO downloads` | `ON CONFLICT DO UPDATE` |
| 905 | `datetime('now', '-' \|\| ? \|\| ' days')` | `NOW() - (INTERVAL '1 day' * $1)` |

#### `/opt/media-downloader/modules/scheduler.py`

| Line | Current | Change To |
|------|---------|-----------|
| 111-113 | PRAGMA statements | Remove |
| 285 | `INSERT OR REPLACE INTO scheduler_state` | `ON CONFLICT DO UPDATE` |
| 324 | `INSERT OR REPLACE INTO scheduler_state` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/modules/activity_status.py`

| Line | Current | Change To |
|------|---------|-----------|
| 48 | INTEGER PRIMARY KEY CHECK (id = 1) | Keep (compatible) |
| 64 | `INSERT OR IGNORE INTO activity_status` | `ON CONFLICT DO NOTHING` |
| 253 | `INSERT OR REPLACE INTO background_task_status` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/modules/settings_manager.py`

| Line | Current | Change To |
|------|---------|-----------|
| 113 | `INSERT OR REPLACE INTO settings` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/modules/discovery_system.py`

| Line | Current | Change To |
|------|---------|-----------|
| 249 | `INSERT OR IGNORE INTO file_tags` | `ON CONFLICT DO NOTHING` |
| 327 | `INSERT OR IGNORE INTO file_tags` | `ON CONFLICT DO NOTHING` |
| 695 | `INSERT OR IGNORE INTO collection_files` | `ON CONFLICT DO NOTHING` |
| 815, 886, 890, etc. | `strftime()` | `TO_CHAR()` |

#### `/opt/media-downloader/modules/semantic_search.py`

| Line | Current | Change To |
|------|---------|-----------|
| 286 | `INSERT OR REPLACE INTO content_embeddings` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/modules/instagram_repost_detector.py`

| Line | Current | Change To |
|------|---------|-----------|
| 445 | `INSERT OR REPLACE INTO repost_fetch_cache` | `ON CONFLICT DO UPDATE` |
| 708 | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY |

#### `/opt/media-downloader/modules/easynews_monitor.py`

| Line | Current | Change To |
|------|---------|-----------|
| 95 | INTEGER PRIMARY KEY CHECK (id = 1) | Keep (compatible) |
| 116 | PRAGMA table_info | Use information_schema |
| 123 | `INSERT OR IGNORE INTO easynews_config` | `ON CONFLICT DO NOTHING` |
| 130, 349 | INTEGER PRIMARY KEY AUTOINCREMENT | SERIAL PRIMARY KEY |

#### `/opt/media-downloader/modules/youtube_channel_monitor.py`

| Line | Current | Change To |
|------|---------|-----------|
| 970 | `INSERT OR IGNORE INTO youtube_monitor_history` | `ON CONFLICT DO NOTHING` |

#### `/opt/media-downloader/modules/face_recognition_module.py`

| Line | Current | Change To |
|------|---------|-----------|
| 175, 1249, 1419, 1679 | PRAGMA table_info | Use information_schema |
| 1257 | `datetime('now')` | `NOW()` |
| 1333 | `datetime('now')` | `NOW()` |

#### `/opt/media-downloader/modules/thumbnail_cache_builder.py`

| Line | Current | Change To |
|------|---------|-----------|
| 59, 201, 232, 260, 273 | PRAGMA journal_mode=WAL | Remove |
| 203 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |
| 234 | `INSERT OR REPLACE INTO media_metadata` | `ON CONFLICT DO UPDATE` |

#### `/opt/media-downloader/modules/universal_video_downloader.py`

| Line | Current | Change To |
|------|---------|-----------|
| 1058 | `INSERT OR REPLACE INTO downloads` | `ON CONFLICT DO UPDATE` |
| 1344 | `INSERT OR IGNORE INTO downloads` | `ON CONFLICT DO NOTHING` |

#### `/opt/media-downloader/modules/move_module.py`

| Line | Current | Change To |
|------|---------|-----------|
| 276 | `INSERT OR REPLACE INTO thumbnails` | `ON CONFLICT DO UPDATE` |

---

## 5. Migration Checklist

### Phase 1: Preparation
- [ ] Set up PostgreSQL server
- [ ] Create database and user with appropriate permissions
- [ ] Install psycopg2 Python package
- [ ] Back up existing SQLite database

### Phase 2: Schema Migration
- [ ] Convert all `INTEGER PRIMARY KEY AUTOINCREMENT` to `SERIAL PRIMARY KEY`
- [ ] Convert `BOOLEAN DEFAULT 0/1` to `BOOLEAN DEFAULT false/true`
- [ ] Convert `BLOB` columns to `BYTEA`
- [ ] Consider converting `TEXT` JSON columns to `JSONB`
- [ ] Create all indexes (same syntax works)
- [ ] Create all foreign key constraints
- [ ] Convert triggers to use `NOW()` instead of `datetime('now')`

### Phase 3: Connection Layer
- [ ] Replace sqlite3 imports with psycopg2
- [ ] Rewrite DatabasePool class for PostgreSQL
- [ ] Remove all PRAGMA statements
- [ ] Update connection string handling

### Phase 4: Query Migration
- [ ] Replace all `INSERT OR IGNORE` with `ON CONFLICT DO NOTHING`
- [ ] Replace all `INSERT OR REPLACE` with `ON CONFLICT DO UPDATE`
- [ ] Replace all `datetime('now', ...)` with `NOW() - INTERVAL '...'`
- [ ] Replace all `strftime()` with `TO_CHAR()` or `EXTRACT()`
- [ ] Replace all `GROUP_CONCAT()` with `STRING_AGG()`
- [ ] Replace all `IFNULL()` with `COALESCE()` (mostly done)
- [ ] Replace all `?` parameter placeholders with `%s`
- [ ] Replace all `cursor.lastrowid` with `RETURNING` clause
- [ ] Review all `LIKE` operators for case sensitivity

### Phase 5: Data Migration
- [ ] Export data from SQLite
- [ ] Transform data types as needed
- [ ] Import into PostgreSQL
- [ ] Verify row counts match
- [ ] Verify data integrity

### Phase 6: Testing
- [ ] Test all database operations
- [ ] Test date calculations
- [ ] Test upsert operations
- [ ] Test concurrent access
- [ ] Performance testing
- [ ] Integration testing with full application

---

## 6. Data Migration Script

```python
#!/usr/bin/env python3
"""
SQLite to PostgreSQL Data Migration Script
"""

import sqlite3
import psycopg2
from psycopg2.extras import execute_values

# Configuration
SQLITE_PATH = '/opt/media-downloader/database/media_downloader.db'
PG_DSN = 'postgresql://user:password@localhost/media_downloader'

# Tables to migrate (in order due to foreign keys)
TABLES = [
    'downloads',
    'forum_threads',
    'forum_posts',
    'search_monitors',
    'scheduler_state',
    'thread_check_history',
    'download_queue',
    'notifications',
    'recycle_bin',
    'instagram_perceptual_hashes',
    'file_inventory',
    'video_downloads',
    'video_preview_list',
    'tags',
    'file_tags',
    'smart_folders',
    'collections',
    'collection_files',
    'content_embeddings',
    'discovery_scan_queue',
    'user_preferences',
    'scrapers',
    'error_log',
    'error_tracking',
    'celebrity_profiles',
    'celebrity_search_presets',
    'celebrity_discovered_videos',
    'celebrity_appearances',
    'appearance_notifications',
    'appearance_config',
    'video_download_queue',
    'youtube_monitor_settings',
    'youtube_channel_monitors',
    'youtube_monitor_history',
    'easynews_config',
    'easynews_searches',
    'easynews_results',
    'paid_content_services',
    'paid_content_identities',
    'paid_content_creators',
    'paid_content_posts',
    'paid_content_attachments',
    'paid_content_embeds',
    'paid_content_favorites',
    'paid_content_download_history',
    'paid_content_notifications',
    'paid_content_config',
    'paid_content_recycle_bin',
    'paid_content_tags',
    'paid_content_post_tags',
    'key_value_store',
]

def migrate_table(sqlite_conn, pg_conn, table_name):
    """Migrate a single table from SQLite to PostgreSQL"""
    sqlite_cursor = sqlite_conn.cursor()
    pg_cursor = pg_conn.cursor()

    # Get column names
    sqlite_cursor.execute(f"PRAGMA table_info({table_name})")
    columns = [row[1] for row in sqlite_cursor.fetchall()]

    # Fetch all data
    sqlite_cursor.execute(f"SELECT * FROM {table_name}")
    rows = sqlite_cursor.fetchall()

    if not rows:
        print(f"  {table_name}: No data to migrate")
        return

    # Build INSERT statement
    col_names = ', '.join(columns)
    placeholders = ', '.join(['%s'] * len(columns))

    # Use execute_values for batch insert
    insert_sql = f"INSERT INTO {table_name} ({col_names}) VALUES %s ON CONFLICT DO NOTHING"

    try:
        execute_values(pg_cursor, insert_sql, rows)
        pg_conn.commit()
        print(f"  {table_name}: Migrated {len(rows)} rows")
    except Exception as e:
        pg_conn.rollback()
        print(f"  {table_name}: ERROR - {e}")

def main():
    # Connect to both databases
    sqlite_conn = sqlite3.connect(SQLITE_PATH)
    pg_conn = psycopg2.connect(PG_DSN)

    print("Starting migration...")

    for table in TABLES:
        migrate_table(sqlite_conn, pg_conn, table)

    # Reset sequences for SERIAL columns
    pg_cursor = pg_conn.cursor()
    for table in TABLES:
        try:
            pg_cursor.execute(f"""
                SELECT setval(pg_get_serial_sequence('{table}', 'id'),
                              COALESCE(MAX(id), 1))
                FROM {table}
            """)
        except:
            pass  # Table might not have id column
    pg_conn.commit()

    sqlite_conn.close()
    pg_conn.close()

    print("Migration complete!")

if __name__ == '__main__':
    main()
```

---

## Notes and Considerations

### Performance
- PostgreSQL handles concurrent access better than SQLite
- Consider adding appropriate indexes after migration
- Use connection pooling (already implemented)
- Consider using JSONB for metadata fields

### Transaction Isolation
- PostgreSQL has different default isolation levels
- Review transaction handling in critical operations

### Backup Strategy
- Keep SQLite database as backup during transition
- Test rollback procedures

### Monitoring
- Monitor query performance after migration
- Watch for deadlocks with concurrent writes
- Monitor connection pool utilization

---

**Document Version:** 1.0
**Last Updated:** 2026-01-30
**Generated by:** Claude Code Migration Analysis
