OSIT-AE-API-FastAPI/documentation/TODO__Agents.md

Backend Agent Task List

Use this file to track steps for complex features or bug fixes. Status: 🔵 DEPLOYMENT READY - Unified Docker Orchestration Complete.

🚀 Recent Infrastructure Wins

• Self-Contained Build: Dockerfile and requirements.txt moved to project root.
• Dependency Pruning: Removed 6 redundant/unused Python packages.
• Unified Orchestration: API now builds as part of the aether_container_env stack.

📋 Operational Hardening (Next Steps)

• Healthcheck: Implement /health route to verify DB/Redis status for Docker orchestration.
• Config Refactor: Switch app/config.py to pydantic-settings to use direct Env Vars (Stop mounting config files).
• Locking: Generate a requirements.lock for bit-identical builds.

🔌 DB Connection Hardening (April 2026 Audit)

Identified during pre-show review. Issues 1 and 2 likely explain observed random connection lags.

• [P1] Remove zombie db_connection.py import — app/routers/api.py imports db from app/db_connection.py, creating a parasitic second SQLAlchemy engine at startup that is never updated by reconnect_db() after bootstrap. The imported db is only used in a commented-out line (api.py:268). Fix: remove the import; delete or archive db_connection.py.
• [P1] Fix retry mechanism in sql_update / run_sql_select — On OperationalError, both call sql_connect() → reconnect_db() which calls engine.dispose(), nuking the entire connection pool mid-flight. Under concurrent requests this kills other in-flight connections. Fix: remove the sql_connect() retry call; SQLAlchemy's pool_pre_ping=True already handles stale connections — just open a fresh engine.connect() for the retry without disposing the pool.
• [P2] Add retry logic to sql_insert and sql_select — Added OperationalError retry (single fresh connection attempt) to sql_insert, sql_select, and sql_insert_or_update. IntegrityError (duplicate key, FK violation) correctly bypasses retry and returns None — retrying the same data would fail again.
• [P3] Guard db = engine.connect() in lib_sql_core.py with try/except — Wrapped in try/except; sets db = None on failure so Docker startup race no longer crashes the worker.
  • [P3 full] Migrate lib_schema_v3.py:39 and lib_api_crud_v3.py:166 off the global db to engine.connect() context managers, then remove the global db entirely.
• [P4] Expose pool_size / max_overflow as env vars — create_ae_engine() calls settings.DB.get('pool_size', 10) but settings.DB property doesn't include those keys, so they're always hardcoded 10/20. Add AE_DB_POOL_SIZE / AE_DB_POOL_MAX_OVERFLOW to config.py.

📋 Feature Tasks

• Core Isolation: Harden apply_forced_account_filter to Fail-Closed.
• IDAA Baseline: Remove public_read from Event, CMS, and Archive objects.
• Detailed Feedback: Implement descriptive 403 Forbidden reasons.
• Polymorphic For_ID Patterns: Add ID Vision to Address, Contact, and DataStore objects.
• Event File Hash_SHA256 Fix: Populate hosted_file_hash_sha256 correctly.
• Step 1: ID Vision Parity Audit
  • Audit Core Event Models (Badge, Session, Presentation).
  • Audit File/Exhibit Models (File, Template, Tracking).
  • Whitelist account_id in all Event search definitions.
  • Audit Relational "Low-Priority" Models (Address, Contact, DataStore).
  • V3 Uniform Lookup System: Phase 1 & 2 Complete.
  • Restore alt-view convenience fields lost in v1→v3 migration (May 2026): site_domain (account_name, account_code, account_enable, account_enable_from/to, site_enable_from/to, site_domain_access_key, logo_path, style_href, script_src, google_tracking_id) and event_session (event_presentation_li_qry_str, event_presenter_li_qry_str). Fields added to Pydantic models and searchable_fields. Alt-view fields require ?view=alt for search.
  • Verify SQL Views join in all required _random IDs for performance.
• Step 2: Coordination (Verify Frontend uses x-account-id instead of token).
• Step 3: Frontend V3 WebSocket integration test — queued after IDAA-specific work. Backend is ready (auth wired, heartbeat presence refresh confirmed, unit tests passing). Frontend guide updated at GUIDE__AE_API_V3_for_Frontend_websockets.md.

🔌 IDAA: Server-Side Novi Verification (Mini Project)

Status: P1–P4 Complete (May 2026). Endpoint live at GET /v3/action/idaa/novi_member/{uuid}. P5 (frontend migration) is the remaining step. Rationale and frontend integration notes: aether_app_sveltekit/documentation/CLIENT__IDAA_and_customized_mods.md → "Planned: Server-Side Novi Verification"

Goal: Proxy the Novi member-verification call server-to-server (FastAPI → Novi) so members' browser IPs are no longer in the call path.

• [P1] New router: app/routers/api_v3_actions_idaa.py

  • Route: GET /v3/action/idaa/novi_member/{uuid}
  • Required auth: Depends(get_account_context) — valid API key + any account context (x-account-id, JWT, or bypass). This is the standard V3 gate.
  • Reads novi_idaa_api_key / novi_api_root_url from site cfg_json via _load_idaa_cfg() (same as Mailman bridge)
  • Calls Novi: GET {novi_api_root_url}/customers/{uuid} with Authorization: Basic {api_key}
  • Normalize email: .replace(' ', '+') (Novi quirk — see Novi-Mailman bridge notes)
  • Build display name: "{FirstName} {LastName[0]}." format, fall back to Name field
  • Returns { "verified": true, "full_name": "...", "email": "..." } on success
  • Returns 404 if Novi 200 with no identity data (empty-member anti-pattern)
  • Returns 429 if Novi rate limits; 503 if Novi unreachable or 5xx
  • Business logic in app/methods/idaa_novi_verify_methods.py

• [P2] Redis cache:

  • Key: idaa:novi_member:{uuid} — TTL 4 hours
    • Note: account_id dropped from key — Novi credentials are hardcoded to the IDAA site; same UUID always returns the same data regardless of caller, so per-caller scoping wastes Redis space and halves hit rate.
  • Cache only verified (200) results — do NOT cache 404 (member may have just joined)
  • Uses redis_client from lib_redis_helpers.py directly

• [P3] Register in registry: Added to routers/registry.py at /v3/action/idaa tag IDAA Actions (V3). Confirmed live — endpoint appears in /openapi.json.

• [P4] Tests: tests/unit/test_unit_idaa_novi_verify.py — 9 tests, all passing.

  • Mock Novi responses (200/empty-200/404/429/503/unreachable)
  • Verify Redis cache is set on 200, hit bypasses Novi call
  • Verify email normalization (space → +)
  • Verify display name format (5 cases)

• [P5] Coordinate with Frontend Agent — Complete (2026-05-19)

  • Frontend replaced direct fetch() to Novi in +layout.svelte:verify_novi_uuid()
  • Response codes mapped: 200 → verified, 404 → denied, 429 → 'rate_limited', 503 → auto-retried (3s, once) then 'api_error'
  • 503 auto-retry added same session to match network-error retry path

🛡️ Security & Privacy Baseline (IDAA)

• Status: ENFORCED.
• Maintenance: Run tests/e2e/test_e2e_v3_security_audit.py after ANY router or registry change.

🔑 Credentials / Access Maintenance

• Bitbucket API Token Migration: Bitbucket is deprecating app passwords — all existing ones become inactive 2026-06-09. SSH migration complete; Gitea remote also configured. Ref: https://support.atlassian.com/bitbucket-cloud/docs/api-tokens/

🚧 Strategic Goals (V3.5+)

• Pydantic V2 / SQLAlchemy 2.0: Major framework upgrade for performance and type safety.
  • SQLAlchemy 2.0 is likely the easier migration (additive, legacy mode available).
  • Pydantic v2 touches every model definition — do this second.
  • Current pins: pydantic==1.*, SQLAlchemy==1.4.52 — intentional, do not remove until migration is done.
• Novi-Mailman Bridge: Cron-based mirror sync between Novi AMS and Mailman 3 — POC complete 2026-03-17.
  • Files: app/methods/e_novi_mailman_methods.py, app/routers/api_v3_actions_e_novi_mailman.py
  • Registered at /v3/action/e_novi_mailman/
  • Confirmed Novi API shape: No flat member list. Fetch via /groups/{guid}/members → UUIDs, then /customers/{uuid} for full record. Fields: Email, FirstName, LastName, Active (bool), UnsubscribeFromEmails (bool). Emails may contain spaces instead of + — sanitized with .replace(' ', '+').
  • Credentials: All in IDAA site cfg_json (id_random='58_gJESdlUh', site id=17). Keys: novi_api_root_url, novi_idaa_api_key, mailman_base_url, mailman_username, mailman_password, novi_mailman_sync (array).
  • Mailman 3 REST API: https://lists.idaa.org/mailman-api (Nginx proxy → 127.0.0.1:8008 → Docker container). Roster: /3.1/lists/{list_id_dot}/roster/member.
  • Sync logic: Full mirror — subscribe Novi-only addresses, unsubscribe Mailman-only addresses. Respects Active=false and UnsubscribeFromEmails=true.
  • Cron target: POST /v3/action/e_novi_mailman/sync — runs all novi_mailman_sync mappings.
  • Webhook approach abandoned — cron is simpler; Novi webhook payload format is unknown and Novi hasn't been configured to send webhooks.
  • Remaining: Set production group→list mappings in cfg_json, configure cron schedule, rotate Mailman restadmin password.
• Lookup System Batch 2: Migration of post_topic, user_status, file_purpose.
• Post Topic Storage Review: Revisit whether IDAA BB posts should keep the lookup-based topic_id/topic_name shape or eventually flatten to saved topic text if the list stays fixed.
• Zoom Events Integration: Implement cron synchronization for OAuth2 ticket retrieval.

📝 Session Notes (March 11, 2026)

• Media Methods Hardened: clip_video_method and convert_file_method in app/methods/lib_media.py updated with improved error logging, PDF validation, and guaranteed temp-file cleanup.
• V3 Action Migration: clip_video endpoint promoted from legacy hosted_file router to V3 action (/v3/action/hosted_file/{id}/clip_video). Legacy route now issues a 307 redirect for backward compatibility.
• Background Scheduling: clip_video V3 action supports ?background=true (returns 202 Accepted), enabling async clipping for large files.
• Robust Deletion: delete_file_action unlink wrapped in try/except OSError — filesystem errors are now logged and non-fatal.
• Unit Tests Added: tests/unit/test_unit_media_methods.py covers clip_video_method and convert_file_method with full async mocking.
• Docs Renamed & Updated: GUIDE__V3_FRONTEND_API.md → GUIDE__AE_API_V3_for_Frontend.md; GUIDE__V3_FRONTEND_WEBSOCKETS.md → GUIDE__AE_API_V3_for_Frontend_websockets.md. Frontend guide updated with V3 action paths, background scheduling notes, and correct example URLs.

📝 Session Notes (March 10, 2026)

• Unified Stack: Merged API orchestration into the master environment.
• Root Assets: Docker assets are now part of this git repo again.
• Pruning: Successfully reduced dependency bloat in requirements.txt.
• Operational Hardening complete: Healthcheck, config refactor (pydantic-settings), requirements.lock all done.
• BuildKit pip cache: Dockerfile now uses --mount=type=cache — rebuild with docker compose up -d --build ae_api.
• Novi-Mailman Bridge scaffolded: Auth pattern and field names confirmed from existing IDAA Jitsi frontend code.