OSIT-AE-API-FastAPI/documentation/TODO__Agents.md

# Backend Agent Task List
> Use this file to track steps for complex features or bug fixes.
> **Status:** 🔵 DEPLOYMENT READY - Unified Docker Orchestration Complete.

## 🚀 Recent Infrastructure Wins
- [x] **Self-Contained Build:** `Dockerfile` and `requirements.txt` moved to project root.
- [x] **Dependency Pruning:** Removed 6 redundant/unused Python packages.
- [x] **Unified Orchestration:** API now builds as part of the `aether_container_env` stack.

## 📋 Operational Hardening (Next Steps)
- [x] **Healthcheck:** Implement `/health` route to verify DB/Redis status for Docker orchestration.
- [x] **Config Refactor:** Switch `app/config.py` to `pydantic-settings` to use direct Env Vars (Stop mounting config files).
- [x] **Locking:** Generate a `requirements.lock` for bit-identical builds.

## 🔌 DB Connection Hardening (April 2026 Audit)
> Identified during pre-show review. Issues 1 and 2 likely explain observed random connection lags.

- [x] **[P1] Remove zombie `db_connection.py` import** — `app/routers/api.py` imports `db` from `app/db_connection.py`, creating a parasitic second SQLAlchemy engine at startup that is never updated by `reconnect_db()` after bootstrap. The imported `db` is only used in a commented-out line (`api.py:268`). Fix: remove the import; delete or archive `db_connection.py`.
- [x] **[P1] Fix retry mechanism in `sql_update` / `run_sql_select`** — On `OperationalError`, both call `sql_connect()` → `reconnect_db()` which calls `engine.dispose()`, nuking the entire connection pool mid-flight. Under concurrent requests this kills other in-flight connections. Fix: remove the `sql_connect()` retry call; SQLAlchemy's `pool_pre_ping=True` already handles stale connections — just open a fresh `engine.connect()` for the retry without disposing the pool.
- [x] **[P2] Add retry logic to `sql_insert` and `sql_select`** — Added `OperationalError` retry (single fresh connection attempt) to `sql_insert`, `sql_select`, and `sql_insert_or_update`. `IntegrityError` (duplicate key, FK violation) correctly bypasses retry and returns `None` — retrying the same data would fail again.
- [x] **[P3] Guard `db = engine.connect()` in `lib_sql_core.py` with try/except** — Wrapped in try/except; sets `db = None` on failure so Docker startup race no longer crashes the worker.
    - [ ] **[P3 full]** Migrate `lib_schema_v3.py:39` and `lib_api_crud_v3.py:166` off the global `db` to `engine.connect()` context managers, then remove the global `db` entirely.
- [x] **[P4] Expose `pool_size` / `max_overflow` as env vars** — `create_ae_engine()` calls `settings.DB.get('pool_size', 10)` but `settings.DB` property doesn't include those keys, so they're always hardcoded 10/20. Add `AE_DB_POOL_SIZE` / `AE_DB_POOL_MAX_OVERFLOW` to `config.py`.

## 📋 Feature Tasks
- [x] **Core Isolation:** Harden `apply_forced_account_filter` to Fail-Closed.
- [x] **IDAA Baseline:** Remove `public_read` from Event, CMS, and Archive objects.
- [x] **Detailed Feedback:** Implement descriptive 403 Forbidden reasons.
- [x] **Polymorphic For_ID Patterns:** Add ID Vision to Address, Contact, and DataStore objects.
- [x] **Event File Hash_SHA256 Fix:** Populate hosted_file_hash_sha256 correctly.
- [ ] **Step 1: ID Vision Parity Audit**
    - [x] Audit Core Event Models (Badge, Session, Presentation).
    - [x] Audit File/Exhibit Models (File, Template, Tracking).
    - [x] Whitelist `account_id` in all Event search definitions.
    - [x] Audit Relational "Low-Priority" Models (Address, Contact, DataStore).
    - [x] **V3 Uniform Lookup System:** Phase 1 & 2 Complete.
    - [x] **Restore alt-view convenience fields lost in v1→v3 migration (May 2026):** `site_domain` (`account_name`, `account_code`, `account_enable`, `account_enable_from/to`, `site_enable_from/to`, `site_domain_access_key`, `logo_path`, `style_href`, `script_src`, `google_tracking_id`) and `event_session` (`event_presentation_li_qry_str`, `event_presenter_li_qry_str`). Fields added to Pydantic models and `searchable_fields`. Alt-view fields require `?view=alt` for search.
    - [ ] Verify SQL Views join in all required `_random` IDs for performance.
- [ ] **Step 2:** Coordination (Verify Frontend uses `x-account-id` instead of token).
- [ ] **Step 3:** Frontend V3 WebSocket integration test — queued after IDAA-specific work. Backend is ready (auth wired, heartbeat presence refresh confirmed, unit tests passing). Frontend guide updated at `GUIDE__AE_API_V3_for_Frontend_websockets.md`.

## 🔌 IDAA: Server-Side Novi Verification (Mini Project)
> **Status: P1–P4 Complete (May 2026).** Endpoint live at `GET /v3/action/idaa/novi_member/{uuid}`. P5 (frontend migration) is the remaining step.
> Rationale and frontend integration notes: `aether_app_sveltekit/documentation/CLIENT__IDAA_and_customized_mods.md` → "Planned: Server-Side Novi Verification"

**Goal:** Proxy the Novi member-verification call server-to-server (FastAPI → Novi) so members' browser IPs are no longer in the call path.

- [x] **[P1] New router:** `app/routers/api_v3_actions_idaa.py`
  - Route: `GET /v3/action/idaa/novi_member/{uuid}`
  - Required auth: `Depends(get_account_context)` — valid API key + any account context (x-account-id, JWT, or bypass). This is the standard V3 gate.
  - Reads `novi_idaa_api_key` / `novi_api_root_url` from site `cfg_json` via `_load_idaa_cfg()` (same as Mailman bridge)
  - Calls Novi: `GET {novi_api_root_url}/customers/{uuid}` with `Authorization: Basic {api_key}`
  - Normalize email: `.replace(' ', '+')` (Novi quirk — see Novi-Mailman bridge notes)
  - Build display name: `"{FirstName} {LastName[0]}."` format, fall back to `Name` field
  - Returns `{ "verified": true, "full_name": "...", "email": "..." }` on success
  - Returns `404` if Novi 200 with no identity data (empty-member anti-pattern)
  - Returns `429` if Novi rate limits; `503` if Novi unreachable or 5xx
  - Business logic in `app/methods/idaa_novi_verify_methods.py`

- [x] **[P2] Redis cache:**
  - Key: `idaa:novi_member:{uuid}` — TTL 4 hours
    - Note: `account_id` dropped from key — Novi credentials are hardcoded to the IDAA site; same UUID always returns the same data regardless of caller, so per-caller scoping wastes Redis space and halves hit rate.
  - Cache only verified (200) results — do NOT cache 404 (member may have just joined)
  - Uses `redis_client` from `lib_redis_helpers.py` directly

- [x] **[P3] Register in registry:** Added to `routers/registry.py` at `/v3/action/idaa` tag `IDAA Actions (V3)`. Confirmed live — endpoint appears in `/openapi.json`.

- [x] **[P4] Tests:** `tests/unit/test_unit_idaa_novi_verify.py` — 9 tests, all passing.
  - Mock Novi responses (200/empty-200/404/429/503/unreachable)
  - Verify Redis cache is set on 200, hit bypasses Novi call
  - Verify email normalization (space → +)
  - Verify display name format (5 cases)

- [x] **[P5] Coordinate with Frontend Agent** — **Complete (2026-05-19)**
  - Frontend replaced direct `fetch()` to Novi in `+layout.svelte:verify_novi_uuid()`
  - Response codes mapped: 200 → verified, 404 → denied, 429 → `'rate_limited'`, 503 → auto-retried (3s, once) then `'api_error'`
  - 503 auto-retry added same session to match network-error retry path

## 🛡️ Security & Privacy Baseline (IDAA)
- **Status:** **ENFORCED**.
- **Maintenance:** Run `tests/e2e/test_e2e_v3_security_audit.py` after ANY router or registry change.

## 🔑 Credentials / Access Maintenance
- [x] **Bitbucket API Token Migration:** Bitbucket is deprecating app passwords — all existing ones become inactive **2026-06-09**. SSH migration complete; Gitea remote also configured. Ref: https://support.atlassian.com/bitbucket-cloud/docs/api-tokens/

## 🚧 Strategic Goals (V3.5+)
- [ ] **Pydantic V2 / SQLAlchemy 2.0:** Major framework upgrade for performance and type safety.
    - SQLAlchemy 2.0 is likely the easier migration (additive, legacy mode available).
    - Pydantic v2 touches every model definition — do this second.
    - Current pins: `pydantic==1.*`, `SQLAlchemy==1.4.52` — intentional, do not remove until migration is done.
- [x] **Novi-Mailman Bridge:** Cron-based mirror sync between Novi AMS and Mailman 3 — **POC complete 2026-03-17**.
    - Files: `app/methods/e_novi_mailman_methods.py`, `app/routers/api_v3_actions_e_novi_mailman.py`
    - Registered at `/v3/action/e_novi_mailman/`
    - **Confirmed Novi API shape:** No flat member list. Fetch via `/groups/{guid}/members` → UUIDs, then `/customers/{uuid}` for full record. Fields: `Email`, `FirstName`, `LastName`, `Active` (bool), `UnsubscribeFromEmails` (bool). Emails may contain spaces instead of `+` — sanitized with `.replace(' ', '+')`.
    - **Credentials:** All in IDAA site `cfg_json` (`id_random='58_gJESdlUh'`, site id=17). Keys: `novi_api_root_url`, `novi_idaa_api_key`, `mailman_base_url`, `mailman_username`, `mailman_password`, `novi_mailman_sync` (array).
    - **Mailman 3 REST API:** `https://lists.idaa.org/mailman-api` (Nginx proxy → `127.0.0.1:8008` → Docker container). Roster: `/3.1/lists/{list_id_dot}/roster/member`.
    - **Sync logic:** Full mirror — subscribe Novi-only addresses, unsubscribe Mailman-only addresses. Respects `Active=false` and `UnsubscribeFromEmails=true`.
    - **Cron target:** `POST /v3/action/e_novi_mailman/sync` — runs all `novi_mailman_sync` mappings.
    - **Webhook approach abandoned** — cron is simpler; Novi webhook payload format is unknown and Novi hasn't been configured to send webhooks.
    - **Remaining:** Set production group→list mappings in `cfg_json`, configure cron schedule, rotate Mailman `restadmin` password.
- [ ] **Lookup System Batch 2:** Migration of `post_topic`, `user_status`, `file_purpose`.
- [ ] **Post Topic Storage Review:** Revisit whether IDAA BB posts should keep the lookup-based `topic_id`/`topic_name` shape or eventually flatten to saved topic text if the list stays fixed.
- [ ] **Zoom Events Integration:** Implement cron synchronization for OAuth2 ticket retrieval.

## 📝 Session Notes (March 11, 2026)
- **Media Methods Hardened:** `clip_video_method` and `convert_file_method` in `app/methods/lib_media.py` updated with improved error logging, PDF validation, and guaranteed temp-file cleanup.
- **V3 Action Migration:** `clip_video` endpoint promoted from legacy `hosted_file` router to V3 action (`/v3/action/hosted_file/{id}/clip_video`). Legacy route now issues a `307` redirect for backward compatibility.
- **Background Scheduling:** `clip_video` V3 action supports `?background=true` (returns `202 Accepted`), enabling async clipping for large files.
- **Robust Deletion:** `delete_file_action` unlink wrapped in `try/except OSError` — filesystem errors are now logged and non-fatal.
- **Unit Tests Added:** `tests/unit/test_unit_media_methods.py` covers `clip_video_method` and `convert_file_method` with full async mocking.
- **Docs Renamed & Updated:** `GUIDE__V3_FRONTEND_API.md` → `GUIDE__AE_API_V3_for_Frontend.md`; `GUIDE__V3_FRONTEND_WEBSOCKETS.md` → `GUIDE__AE_API_V3_for_Frontend_websockets.md`. Frontend guide updated with V3 action paths, background scheduling notes, and correct example URLs.

## 📝 Session Notes (March 10, 2026)
- **Unified Stack:** Merged API orchestration into the master environment.
- **Root Assets:** Docker assets are now part of this git repo again.
- **Pruning:** Successfully reduced dependency bloat in `requirements.txt`.
- **Operational Hardening complete:** Healthcheck, config refactor (pydantic-settings), requirements.lock all done.
- **BuildKit pip cache:** Dockerfile now uses `--mount=type=cache` — rebuild with `docker compose up -d --build ae_api`.
- **Novi-Mailman Bridge scaffolded:** Auth pattern and field names confirmed from existing IDAA Jitsi frontend code.