From 051b2fd7ace6509f3b7545af6dada4ddfe8cf3ea Mon Sep 17 00:00:00 2001 From: Scott Idem Date: Tue, 19 May 2026 18:56:09 -0400 Subject: [PATCH] docs: add agent bootstrap quickstart and consolidate documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add BOOTSTRAP__AI_Agent_Quickstart.md — fast-path entry doc for AI agents covering critical rules, V3 action patterns, Redis/auth/logger_reset gotchas, and a mistakes-agents-have-made section - Expand ARCH__V3_DEVELOPMENT_STANDARDS.md with merged content from ARCH__V3_CRUD_LEARNINGS: dependency injection reference, security/isolation model detail, and FastAPI/Pydantic gotchas - Archive 4 outdated docs: GUIDE__LOCAL_DEVELOPMENT (predates Docker), FRONTEND_API_SAMPLES (belongs in SvelteKit repo), ARCH__UNIFIED_AGENT (replaced by MCP ecosystem), ARCH__V3_CRUD_LEARNINGS (content merged above) Co-Authored-By: Claude Sonnet 4.6 --- .../ARCH__V3_DEVELOPMENT_STANDARDS.md | 42 +- .../BOOTSTRAP__AI_Agent_Quickstart.md | 420 ++++++++++++++++++ .../{ => archive}/ARCH__UNIFIED_AGENT.md | 0 .../{ => archive}/ARCH__V3_CRUD_LEARNINGS.md | 0 .../{ => archive}/FRONTEND_API_SAMPLES.md | 0 .../{ => archive}/GUIDE__LOCAL_DEVELOPMENT.md | 0 6 files changed, 461 insertions(+), 1 deletion(-) create mode 100644 documentation/BOOTSTRAP__AI_Agent_Quickstart.md rename documentation/{ => archive}/ARCH__UNIFIED_AGENT.md (100%) rename documentation/{ => archive}/ARCH__V3_CRUD_LEARNINGS.md (100%) rename documentation/{ => archive}/FRONTEND_API_SAMPLES.md (100%) rename documentation/{ => archive}/GUIDE__LOCAL_DEVELOPMENT.md (100%) diff --git a/documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md b/documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md index 68afaa1..3ce33ad 100644 --- a/documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md +++ b/documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md @@ -61,7 +61,47 @@ Some objects have a richer alternate SQL view (`tbl_alt`) that adds JOINed/compu - The nested search router (`api_crud_v3_nested.py`) already supports `?view=` to switch between registered views. `view=default` uses `tbl_default`; `view=alt` uses `tbl_alt`; additional named views can be added to the object registry as `tbl_` / `mdl_`. - Flat search (`api_crud_v3.py`) does not yet support `?view=` — it always uses `tbl_default`. -## 4. Stability Rules +## 4. V3 Dependency Injection Reference + +All V3 endpoints use granular, composable `Depends()` from `app/lib_general_v3.py`: + +| Dependency | Purpose | +|---|---| +| `get_account_context` → `AccountContext` | Resolves `account_id` with precedence: Header → Query Token → Bypass Header. Raises 403 on guest/missing context. | +| `PaginationParams` | Standardizes `limit` and `offset`. | +| `StatusFilterParams` | Handles `enabled` and `hidden` filtering. | +| `SerializationParams` | Controls Pydantic serialization (`by_alias`, `exclude_unset`). | +| `DelayParams` | Optional latency simulation (`?delay=N`) via `await asyncio.sleep()`. | + +`AccountContext` also carries `administrator`, `manager`, and `super` flags, populated by a deferred DB lookup when a JWT is present. These flags control whether account isolation is bypassed for support tasks. + +## 5. Security and Data Isolation + +### Fail-Closed Strategy +If `account_id` or auth context is missing, the API defaults to a blocking filter (`account_id IS NULL`) — it does NOT fall back to returning all records. Never relax this. + +### Multi-Tenant Isolation +- **Forced account filtering**: `apply_forced_account_filter` injects an `account_id` WHERE clause into every list/search query for non-super users. +- **Post-retrieval verification**: Single-object GET, PATCH, DELETE include a secondary ownership check (`check_account_access`). A mismatch returns 403. +- **Hierarchical verification**: Nested endpoints verify parent ownership before allowing operations on children. +- **Creation guard**: On POST, the user's `account_id` is automatically forced onto the new record. + +### IDAA Privacy Baseline +No IDAA object (Events, Files, Posts, Meetings) is public by default. All routes require `x-account-id` context. The sole exception is `site_domain` (used for site bootstrapping). This is a **Sev-1 class constraint** — violating it has happened before. + +### Bypass / Admin Access +- `x-no-account-id: bypass` → grants super access, resolves to `account_id=1` (One Sky IT Demo). Use only in internal/development utilities; do not expand its use. +- JWT query parameter (`?jwt=...`) is supported for download links and share URLs where custom headers cannot be provided. + +## 6. FastAPI and Pydantic Gotchas + +- **`response: Response` injection**: Use it as a direct type hint in function signature. `Depends(Response)` is not valid and causes router initialization failures. +- **Parameter order**: In function signatures, arguments without defaults must come before `Depends()` arguments. +- **`asyncio.sleep()` not `time.sleep()`**: Blocking the event loop in an async endpoint causes worker timeouts and `502 Bad Gateway` under load. +- **Pydantic V1 only**: Do not use V2-only features (`computed_field`, `model_validator`, etc.). The migration is a separate planned project — see strategic goals in `TODO__Agents.md`. +- **`obj_type_kv_li` in `ae_obj_types_def.py`**: Supports both modern keys (`tbl`, `mdl`) and legacy keys (`table_name`, `base_name`). Legacy V2 endpoints depend on the legacy keys — do not remove them until V1/V2 are fully retired. + +## 7. Stability Rules 1. **Baby Step Testing**: Restart Docker and verify root health after *every* modular change. 2. **Avoid Shadowing**: Never name a module part of the `app.` package the same as a common instance variable (e.g., avoid `app.middleware` package if you use `app = FastAPI()`). diff --git a/documentation/BOOTSTRAP__AI_Agent_Quickstart.md b/documentation/BOOTSTRAP__AI_Agent_Quickstart.md new file mode 100644 index 0000000..3da5164 --- /dev/null +++ b/documentation/BOOTSTRAP__AI_Agent_Quickstart.md @@ -0,0 +1,420 @@ +# Aether API — AI Agent Bootstrap / Quickstart +> **Read this first.** This doc is the fast path to being productive on this project. +> It covers the rules, patterns, and gotchas that matter most. +> Deep dives are in the linked docs at the bottom. + +--- + +## 1. What This Project Is + +**Aether** is an event management platform built by One Sky IT (Scott Idem). +This repo is the backend: **FastAPI + MariaDB**, running inside Docker. + +The frontend (`aether_app_sveltekit/`) talks to this API exclusively via the V3 REST API. +There is **no standalone dev server** — the API runs only in Docker. + +**Key clients:** +- **Conference organizers** — Presentation management, badges, sessions +- **Exhibitors** — Leads capture +- **IDAA** — International Doctors in Alcoholics Anonymous (strictly private medical/recovery community) + +**Stack at a glance:** + +| Layer | Technology | +|---|---| +| Framework | FastAPI + Pydantic V1 (upgrade deferred) | +| Database | MariaDB via SQLAlchemy 1.4 (upgrade deferred) | +| Cache | Redis | +| Auth | Custom headers: `x-aether-api-key` + `x-account-id` | +| Container | Docker / Gunicorn — source is volume-mounted (no rebuild for Python changes, but **restart required**) | + +--- + +## 2. Critical Rules — Read Before Touching Any Code + +### Privacy (Sev-1 class failures if violated) +- **IDAA content is ALWAYS private.** All routes under `/idaa/` require authentication. + A previous agent accidentally exposed IDAA bulletin board data publicly. + This is the single most serious class of mistake on this project. + When in doubt — it's private. Always verify with `Depends(get_account_context)`. +- **Journals** are private personal data. Always authenticated. + +### File Safety +- **Never use `rm`** to delete files. Move to `~/tmp/gemini_trash` instead. +- **Never commit `.env`** files, API keys, or passwords of any kind. +- Third-party credentials (Novi API key, Mailman credentials) live in the **MariaDB `site.cfg_json` column**, not in `.env` or code. + +### Before Every Commit +1. `python3 -m py_compile ` — syntax check +2. Restart Docker and verify the API starts clean: `docker compose restart ae_api` +3. Check logs: `docker compose logs -f ae_api` (look for startup errors or import failures) +4. Run the relevant E2E or unit test suite + +### Before Starting Any Task +- Read `documentation/TODO__Agents.md` — active tasks, known bugs, and what was recently changed and why. +- Check `tests/README.md` — which test suite covers the area you're about to touch. + +### Docker Restart is Mandatory After Python Changes +The API source is volume-mounted, so file edits appear instantly inside the container — but +**Gunicorn does NOT hot-reload**. You MUST run: +```bash +docker compose restart ae_api +``` +after any Python file change before testing. This is the #1 cause of "my change didn't take effect." + +--- + +## 3. Environment & Commands Cheat Sheet + +```bash +# Start full stack +cd ~/OSIT_dev/aether_container_env && docker compose up -d + +# Restart API after any Python change +docker compose restart ae_api + +# Follow API logs +docker compose logs -f ae_api + +# Shell into the container +docker compose exec ae_api bash + +# Run unit tests (from project root) +./environment/bin/python3 -m pytest tests/unit/ -v + +# Run a single E2E test +./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py + +# Check a specific Python file compiles +./environment/bin/python3 -m py_compile app/methods/my_new_methods.py +``` + +**Development API:** `https://dev-api.oneskyit.com` +**Dev API secret key:** `nT0jPeiCfxSifkiDZur9jA` +**Standard test Agent API key:** `PMM4n50teUCaOMMTN8qOJA` + +**Dev DB via phpMyAdmin:** `http://localhost:8081` + +--- + +## 4. V3 Action Router Pattern + +When adding a new action endpoint (not CRUD), follow this structure: + +### File layout +``` +app/methods/my_feature_methods.py ← business logic +app/routers/api_v3_actions_my_feature.py ← route handler (thin) +``` + +Then register in `app/routers/registry.py`: +```python +from app.routers import api_v3_actions_my_feature +# ... +app.include_router(api_v3_actions_my_feature.router, prefix='/v3/action/my_feature', tags=['My Feature (V3 Actions)']) +``` + +### Standard route handler pattern +```python +from fastapi import APIRouter, Depends +from app.lib_general_v3 import AccountContext, get_account_context, DelayParams +from app.models.response_models import Resp_Body_Base, mk_resp +from app.methods.my_feature_methods import my_business_logic + +router = APIRouter() + +@router.get('/my_endpoint/{obj_id}', response_model=Resp_Body_Base) +async def get_my_endpoint( + obj_id: str, + account: AccountContext = Depends(get_account_context), + delay: DelayParams = Depends(), + ): + if delay.sleep_time_s > 0: + await asyncio.sleep(delay.sleep_time_s) + + result = my_business_logic(obj_id) + status = result.get('status', 503) + + if status == 200: + return mk_resp(data=result['data']) + + return mk_resp(data=False, status_code=status, status_message=result.get('reason', 'Error.')) +``` + +### Auth dependency +`Depends(get_account_context)` is the **standard V3 gate**: +- Requires a valid `x-aether-api-key` +- Requires `x-account-id` OR a valid JWT session OR bypass mode +- Raises 403 if `auth_method == 'guest'` or no account context can be resolved +- For IDAA/private routes: this is the minimum gate. Never relax it. + +--- + +## 5. Loading Site-Based Credentials (the `_load_idaa_cfg` Pattern) + +Third-party credentials (Novi API key, Mailman credentials, etc.) are stored in MariaDB +in the `site.cfg_json` column for the relevant site record. Do NOT store them in `.env`. + +The IDAA site record is: +- `id_random = '58_gJESdlUh'` (site id=17) +- Fields: `novi_api_root_url`, `novi_idaa_api_key`, `mailman_base_url`, `mailman_username`, `mailman_password`, `novi_mailman_sync` + +Pattern to load them (use deferred import to avoid circular deps): +```python +IDAA_SITE_ID_RANDOM = '58_gJESdlUh' + +def _load_idaa_cfg() -> dict: + from app.db_sql import load_site_obj + site = load_site_obj(IDAA_SITE_ID_RANDOM) + if not site: + raise RuntimeError('IDAA site record not found') + cfg = site.get('cfg_json') or {} + if isinstance(cfg, str): + import json + cfg = json.loads(cfg) + return cfg +``` + +--- + +## 6. Redis Cache Pattern + +```python +import json +import datetime +from app.lib_redis_helpers import redis_client + +_CACHE_TTL = datetime.timedelta(hours=4) + +def _cache_key(uuid: str) -> str: + return f'idaa:novi_member:{uuid}' + +# Reading +raw = redis_client.get(_cache_key(uuid)) +if raw: + return json.loads(raw) + +# Writing (only on success — never cache error states) +redis_client.setex(_cache_key(uuid), _CACHE_TTL, json.dumps(result)) +``` + +**Key naming convention:** `{module}:{object_type}:{identifier}` — e.g. `idaa:novi_member:{uuid}` + +**Never cache:** +- 404 responses — the member may have just joined; next call should hit the source +- 429 / 503 errors — transient failures should not poison future callers + +**Cache key scoping:** If the underlying data source is the same regardless of caller +(e.g. Novi credentials are hardcoded to the IDAA site — same UUID always returns same data), +drop `account_id` from the key. Per-caller scoping wastes Redis space and halves hit rate. + +--- + +## 7. The `@logger_reset` Decorator — Unit Test Gotcha + +Business logic methods use `@logger_reset` from `app.lib_general`: + +```python +from app.lib_general import logger_reset + +@logger_reset +def my_method(arg): + ... +``` + +**In unit tests, this decorator MUST be mocked as a passthrough.** +If `app.lib_general` is replaced with a plain `MagicMock()`, the decorator becomes a +MagicMock, which when applied to a function replaces it with `MagicMock()()` — the decorated +function is gone and every call returns garbage. + +```python +# WRONG — logger_reset becomes a MagicMock and swallows the function: +sys.modules['app.lib_general'] = MagicMock() + +# CORRECT — make it a passthrough decorator: +mock_lib_general = MagicMock() +mock_lib_general.logger_reset = lambda f: f +sys.modules['app.lib_general'] = mock_lib_general +``` + +This is the #1 cause of unit tests returning `` instead of real dicts. + +--- + +## 8. Field Evolution Checklist + +When a table or view gains, loses, or renames fields — **do all of these in order**: + +1. Update the Pydantic model in `app/models/` +2. Update the SQL view or table projection so GET/SEARCH return the field +3. Update `searchable_fields` in `app/object_definitions/` (only for searchable fields) +4. Add virtual/view-only fields to `fields_to_exclude_from_db` if they must not be persisted +5. Run the relevant schema/search E2E tests +6. **Restart Docker:** `docker compose restart ae_api` + +#### Alt-view fields (in `tbl_alt` only) +Some objects have a richer alternate SQL view triggered by `?view=alt`. These fields +**must still be declared in the Pydantic model and `searchable_fields`** even if they only +appear in the alt view — Pydantic strips undeclared fields silently. + +--- + +## 9. Deferred Imports in Library Modules + +To avoid circular dependency traps, **never import from `app.db_sql` or other app modules +at the top level of a library module**. Use deferred imports inside functions: + +```python +# WRONG — causes circular import at startup: +from app.db_sql import load_site_obj + +def my_func(): + return load_site_obj('abc') + +# CORRECT: +def my_func(): + from app.db_sql import load_site_obj + return load_site_obj('abc') +``` + +--- + +## 10. Pydantic / SQLAlchemy Version Pins — Do Not Remove + +Current intentional pins: +- `pydantic==1.*` +- `SQLAlchemy==1.4.52` + +A Pydantic V2 and SQLAlchemy 2.0 migration is planned but not started. Until then, +**do not upgrade these packages** — V2 touches every model definition and the migration +is a dedicated project. + +--- + +## 11. Mistakes Agents Have Made on This Project + +These are real incidents — know them before you start. + +1. **IDAA data exposed publicly** — an agent removed an auth guard from the bulletin board + router. Consequence: private IDAA recovery community data was publicly accessible. + Always verify `Depends(get_account_context)` is present on every IDAA route. + +2. **"My code change has no effect"** — Python file was edited but Docker was not restarted. + The API runs Gunicorn inside Docker with no hot-reload. `docker compose restart ae_api` + is required after every Python change. + +3. **`@logger_reset` mock swallows functions in unit tests** — see Section 7 above. + Symptom: `assert result['status'] == 200` fails with `TypeError: 'MagicMock' is not subscriptable`. + +4. **`pytest` / `pytest-asyncio` not installed after venv rebuild** — these are dev-only + dependencies not in `requirements.txt`. After any OS Python update (e.g., Arch Linux + upgrading to a new Python minor), rebuild the venv and reinstall: + ```bash + ./environment/bin/pip install pytest pytest-asyncio + ``` + +5. **Global `db` connection used instead of context manager** — `lib_sql_core.py` has a + global `db = engine.connect()` that is a fragile single connection, not a pool. + For new methods, prefer `engine.connect()` as a context manager. See `TODO__Agents.md` + → "[P3 full]" task for the planned migration. + +6. **`bypass` mode hardcodes `account_id=1`** — `x-no-account-id: bypass` resolves to + `account_id=1` (One Sky IT Demo). Lookup overrides from the Demo account can leak into + bypass sessions. Do not expand bypass usage without documenting the allowlist case. + +7. **Caching error states in Redis** — caching 404 or 5xx responses poisons the cache. + A member who just joined Novi would be denied for 4 hours if their 404 was cached. + Only cache verified success (200) results. + +8. **Not running `docker compose restart ae_api` between model changes and E2E tests** — + the E2E suite hits the live API, which is still running the old code until restarted. + Tests will pass or fail against stale behavior and the results are meaningless. + +--- + +## 12. Test Patterns + +### Unit tests (fast, no DB/network) +```bash +./environment/bin/python3 -m pytest tests/unit/ -v +``` +- Mock all DB/Redis/HTTP at the top of the file before importing the module under test +- `@logger_reset` must be mocked as a passthrough (see Section 7) +- Always run from the **project root** — scripts use `sys.path.append(os.getcwd())` + +### E2E tests (live API at dev-api.oneskyit.com) +```bash +./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py +``` +- Require the Docker stack to be running with a working DB connection +- Use standard output format: `[✅ PASS]` / `[❌ FAIL]` +- Verify IDs in responses are **strings** (not integers) — ID Vision compliance + +### Which tests to run +| Change type | Required suites | +|---|---| +| Model / ID Vision changes | `test_e2e_v3_demo_parity.py`, vision parity tests | +| Search / filter changes | `test_e2e_v3_search_engine.py` | +| Auth / account context changes | `test_e2e_v3_security_audit.py`, `test_e2e_v3_auth_security.py` | +| Any router or registry change | `test_e2e_v3_security_audit.py` | +| Novi-Mailman bridge changes | `test_e2e_v3_action_novi_mailman.py` | +| IDAA Novi verify changes | `tests/unit/test_unit_idaa_novi_verify.py` | + +--- + +## 13. Source Layout (Quick Reference) + +``` +app/ + main.py — FastAPI app, lifespan, CORS + config.py — Pydantic Settings (from .env via Docker) + routers/ + registry.py — All router registrations live here + api_crud_v3.py — Generic V3 CRUD (flat) + api_crud_v3_nested.py — Generic V3 CRUD (nested/parent-owned) + api_v3_actions_*.py — Action endpoints (one file per domain) + dependencies_v3.py — Shared FastAPI Depends() helpers + models/ — Pydantic V1 models (one file per domain) + object_definitions/ — Per-object searchable_fields, field metadata + ae_obj_types_def.py — Object type registry (all Aether types defined here) + methods/ — Business logic (one file per feature/domain) + lib_api_crud_v3.py — Generic CRUD handler (all object types share this) + lib_schema_v3.py — Dynamic schema/field resolution per object type + lib_general_v3.py — AccountContext, get_account_context, DelayParams + lib_general.py — logger_reset and general utilities + db_sql.py — Import facade (always import from here) + lib_sql_core.py — SQLAlchemy engine, global db connection (Source of Truth) + lib_sql_crud.py — sql_insert, sql_select, sql_update, etc. + lib_redis_helpers.py — redis_client global instance + +tests/ + unit/ — Isolated logic tests (mock everything, no DB) + integration/ — Requires local MariaDB/Redis + e2e/ — Network tests against live dev API + tools/ — Admin utilities (stress test, registry generator) + mock_config_helper.py — Mock app.config.settings — use in all unit tests + README.md — Which tests cover what; when to run them + +documentation/ + TODO__Agents.md — Active tasks + session notes ← always read first + ARCH__V3_DEVELOPMENT_STANDARDS.md — Master V3 standards doc + ARCH__V3_CORE.md — Module architecture (lifespan, DB layers, logging) + GUIDE__AE_API_V3_for_Frontend.md — Frontend integration guide (keep current) + GUIDE__DEVELOPMENT.md — Commit SOP, verification checklist +``` + +--- + +## 14. Reading Order for Deeper Dives + +| What you need | Read | +|---|---| +| Active tasks + known bugs | `documentation/TODO__Agents.md` ← always first | +| V3 standards and strategy | `documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md` | +| Module architecture | `documentation/ARCH__V3_CORE.md` | +| Frontend API integration guide | `documentation/GUIDE__AE_API_V3_for_Frontend.md` | +| WebSocket integration | `documentation/GUIDE__AE_API_V3_for_Frontend_websockets.md` | +| Commit / verification SOP | `documentation/GUIDE__DEVELOPMENT.md` | +| Which tests to run | `tests/README.md` | +| Object type registry | `app/ae_obj_types_def.py` | +| Shared agent docs | `~/agents_sync/aether/docs/UNIFIED_AGENT_ARCH.md` | diff --git a/documentation/ARCH__UNIFIED_AGENT.md b/documentation/archive/ARCH__UNIFIED_AGENT.md similarity index 100% rename from documentation/ARCH__UNIFIED_AGENT.md rename to documentation/archive/ARCH__UNIFIED_AGENT.md diff --git a/documentation/ARCH__V3_CRUD_LEARNINGS.md b/documentation/archive/ARCH__V3_CRUD_LEARNINGS.md similarity index 100% rename from documentation/ARCH__V3_CRUD_LEARNINGS.md rename to documentation/archive/ARCH__V3_CRUD_LEARNINGS.md diff --git a/documentation/FRONTEND_API_SAMPLES.md b/documentation/archive/FRONTEND_API_SAMPLES.md similarity index 100% rename from documentation/FRONTEND_API_SAMPLES.md rename to documentation/archive/FRONTEND_API_SAMPLES.md diff --git a/documentation/GUIDE__LOCAL_DEVELOPMENT.md b/documentation/archive/GUIDE__LOCAL_DEVELOPMENT.md similarity index 100% rename from documentation/GUIDE__LOCAL_DEVELOPMENT.md rename to documentation/archive/GUIDE__LOCAL_DEVELOPMENT.md