OSIT-AE-API-FastAPI/documentation/TODO__Agents.md

# Backend Agent Task List
> Use this file to track steps for complex features or bug fixes.
> **Status:** 🔵 DEPLOYMENT READY - Unified Docker Orchestration Complete.

## 🚀 Recent Infrastructure Wins
- [x] **Self-Contained Build:** `Dockerfile` and `requirements.txt` moved to project root.
- [x] **Dependency Pruning:** Removed 6 redundant/unused Python packages.
- [x] **Unified Orchestration:** API now builds as part of the `aether_container_env` stack.

## 📋 Operational Hardening (Next Steps)
- [x] **Healthcheck:** Implement `/health` route to verify DB/Redis status for Docker orchestration.
- [x] **Config Refactor:** Switch `app/config.py` to `pydantic-settings` to use direct Env Vars (Stop mounting config files).
- [x] **Locking:** Generate a `requirements.lock` for bit-identical builds.

## 🔌 DB Connection Hardening (April 2026 Audit)
> Identified during pre-show review. Issues 1 and 2 likely explain observed random connection lags.

- [x] **[P1] Remove zombie `db_connection.py` import** — `app/routers/api.py` imports `db` from `app/db_connection.py`, creating a parasitic second SQLAlchemy engine at startup that is never updated by `reconnect_db()` after bootstrap. The imported `db` is only used in a commented-out line (`api.py:268`). Fix: remove the import; delete or archive `db_connection.py`.
- [x] **[P1] Fix retry mechanism in `sql_update` / `run_sql_select`** — On `OperationalError`, both call `sql_connect()` → `reconnect_db()` which calls `engine.dispose()`, nuking the entire connection pool mid-flight. Under concurrent requests this kills other in-flight connections. Fix: remove the `sql_connect()` retry call; SQLAlchemy's `pool_pre_ping=True` already handles stale connections — just open a fresh `engine.connect()` for the retry without disposing the pool.
- [ ] **[P2] Add retry logic to `sql_insert` and `sql_select`** — Both are missing the `OperationalError` retry that `sql_update` and `run_sql_select` have. A DB blip during an INSERT (scan record, badge log, etc.) fails silently and returns `False` with no recovery attempt.
- [ ] **[P3] Guard `db = engine.connect()` in `lib_sql_core.py` with try/except** — Line 48 is a bare connect at module load time with no error handling. If MariaDB isn't ready (Docker race), this throws unhandled and crashes the worker. Wrap in try/except like `db_connection.py` already does.
- [x] **[P4] Expose `pool_size` / `max_overflow` as env vars** — `create_ae_engine()` calls `settings.DB.get('pool_size', 10)` but `settings.DB` property doesn't include those keys, so they're always hardcoded 10/20. Add `AE_DB_POOL_SIZE` / `AE_DB_POOL_MAX_OVERFLOW` to `config.py`.

## 📋 Feature Tasks
- [x] **Core Isolation:** Harden `apply_forced_account_filter` to Fail-Closed.
- [x] **IDAA Baseline:** Remove `public_read` from Event, CMS, and Archive objects.
- [x] **Detailed Feedback:** Implement descriptive 403 Forbidden reasons.
- [x] **Polymorphic For_ID Patterns:** Add ID Vision to Address, Contact, and DataStore objects.
- [x] **Event File Hash_SHA256 Fix:** Populate hosted_file_hash_sha256 correctly.
- [ ] **Step 1: ID Vision Parity Audit**
    - [x] Audit Core Event Models (Badge, Session, Presentation).
    - [x] Audit File/Exhibit Models (File, Template, Tracking).
    - [x] Whitelist `account_id` in all Event search definitions.
    - [x] Audit Relational "Low-Priority" Models (Address, Contact, DataStore).
    - [x] **V3 Uniform Lookup System:** Phase 1 & 2 Complete.
    - [ ] Verify SQL Views join in all required `_random` IDs for performance.
- [ ] **Step 2:** Coordination (Verify Frontend uses `x-account-id` instead of token).
- [ ] **Step 3:** Frontend V3 WebSocket integration test — queued after IDAA-specific work. Backend is ready (auth wired, heartbeat presence refresh confirmed, unit tests passing). Frontend guide updated at `GUIDE__AE_API_V3_for_Frontend_websockets.md`.

## 🛡️ Security & Privacy Baseline (IDAA)
- **Status:** **ENFORCED**.
- **Maintenance:** Run `tests/e2e/test_e2e_v3_security_audit.py` after ANY router or registry change.

## 🔑 Credentials / Access Maintenance
- [x] **Bitbucket API Token Migration:** Bitbucket is deprecating app passwords — all existing ones become inactive **2026-06-09**. SSH migration complete; Gitea remote also configured. Ref: https://support.atlassian.com/bitbucket-cloud/docs/api-tokens/

## 🚧 Strategic Goals (V3.5+)
- [ ] **Pydantic V2 / SQLAlchemy 2.0:** Major framework upgrade for performance and type safety.
    - SQLAlchemy 2.0 is likely the easier migration (additive, legacy mode available).
    - Pydantic v2 touches every model definition — do this second.
    - Current pins: `pydantic==1.*`, `SQLAlchemy==1.4.52` — intentional, do not remove until migration is done.
- [x] **Novi-Mailman Bridge:** Cron-based mirror sync between Novi AMS and Mailman 3 — **POC complete 2026-03-17**.
    - Files: `app/methods/e_novi_mailman_methods.py`, `app/routers/api_v3_actions_e_novi_mailman.py`
    - Registered at `/v3/action/e_novi_mailman/`
    - **Confirmed Novi API shape:** No flat member list. Fetch via `/groups/{guid}/members` → UUIDs, then `/customers/{uuid}` for full record. Fields: `Email`, `FirstName`, `LastName`, `Active` (bool), `UnsubscribeFromEmails` (bool). Emails may contain spaces instead of `+` — sanitized with `.replace(' ', '+')`.
    - **Credentials:** All in IDAA site `cfg_json` (`id_random='58_gJESdlUh'`, site id=17). Keys: `novi_api_root_url`, `novi_idaa_api_key`, `mailman_base_url`, `mailman_username`, `mailman_password`, `novi_mailman_sync` (array).
    - **Mailman 3 REST API:** `https://lists.idaa.org/mailman-api` (Nginx proxy → `127.0.0.1:8008` → Docker container). Roster: `/3.1/lists/{list_id_dot}/roster/member`.
    - **Sync logic:** Full mirror — subscribe Novi-only addresses, unsubscribe Mailman-only addresses. Respects `Active=false` and `UnsubscribeFromEmails=true`.
    - **Cron target:** `POST /v3/action/e_novi_mailman/sync` — runs all `novi_mailman_sync` mappings.
    - **Webhook approach abandoned** — cron is simpler; Novi webhook payload format is unknown and Novi hasn't been configured to send webhooks.
    - **Remaining:** Set production group→list mappings in `cfg_json`, configure cron schedule, rotate Mailman `restadmin` password.
- [ ] **Lookup System Batch 2:** Migration of `post_topic`, `user_status`, `file_purpose`.
- [ ] **Zoom Events Integration:** Implement cron synchronization for OAuth2 ticket retrieval.

## 📝 Session Notes (March 11, 2026)
- **Media Methods Hardened:** `clip_video_method` and `convert_file_method` in `app/methods/lib_media.py` updated with improved error logging, PDF validation, and guaranteed temp-file cleanup.
- **V3 Action Migration:** `clip_video` endpoint promoted from legacy `hosted_file` router to V3 action (`/v3/action/hosted_file/{id}/clip_video`). Legacy route now issues a `307` redirect for backward compatibility.
- **Background Scheduling:** `clip_video` V3 action supports `?background=true` (returns `202 Accepted`), enabling async clipping for large files.
- **Robust Deletion:** `delete_file_action` unlink wrapped in `try/except OSError` — filesystem errors are now logged and non-fatal.
- **Unit Tests Added:** `tests/unit/test_unit_media_methods.py` covers `clip_video_method` and `convert_file_method` with full async mocking.
- **Docs Renamed & Updated:** `GUIDE__V3_FRONTEND_API.md` → `GUIDE__AE_API_V3_for_Frontend.md`; `GUIDE__V3_FRONTEND_WEBSOCKETS.md` → `GUIDE__AE_API_V3_for_Frontend_websockets.md`. Frontend guide updated with V3 action paths, background scheduling notes, and correct example URLs.

## 📝 Session Notes (March 10, 2026)
- **Unified Stack:** Merged API orchestration into the master environment.
- **Root Assets:** Docker assets are now part of this git repo again.
- **Pruning:** Successfully reduced dependency bloat in `requirements.txt`.
- **Operational Hardening complete:** Healthcheck, config refactor (pydantic-settings), requirements.lock all done.
- **BuildKit pip cache:** Dockerfile now uses `--mount=type=cache` — rebuild with `docker compose up -d --build ae_api`.
- **Novi-Mailman Bridge scaffolded:** Auth pattern and field names confirmed from existing IDAA Jitsi frontend code.