# Aether API — AI Agent Bootstrap / Quickstart
> **Read this first.** This doc is the fast path to being productive on this project.
> It covers the rules, patterns, and gotchas that matter most.
> Deep dives are in the linked docs at the bottom.

---

## 1. What This Project Is

**Aether** is an event management platform built by One Sky IT (Scott Idem).
This repo is the backend: **FastAPI + MariaDB**, running inside Docker.

The frontend (`aether_app_sveltekit/`) talks to this API exclusively via the V3 REST API.
There is **no standalone dev server** — the API runs only in Docker.

**Key clients:**
- **Conference organizers** — Presentation management, badges, sessions
- **Exhibitors** — Leads capture
- **IDAA** — International Doctors in Alcoholics Anonymous (strictly private medical/recovery community)

**Stack at a glance:**

| Layer | Technology |
|---|---|
| Framework | FastAPI + Pydantic V1 (upgrade deferred) |
| Database | MariaDB via SQLAlchemy 1.4 (upgrade deferred) |
| Cache | Redis |
| Auth | Custom headers: `x-aether-api-key` + `x-account-id` |
| Container | Docker / Gunicorn — source is volume-mounted (no rebuild for Python changes, but **restart required**) |

---

## 2. Critical Rules — Read Before Touching Any Code

### Privacy (Sev-1 class failures if violated)
- **IDAA content is ALWAYS private.** All routes under `/idaa/` require authentication.
  A previous agent accidentally exposed IDAA bulletin board data publicly.
  This is the single most serious class of mistake on this project.
  When in doubt — it's private. Always verify with `Depends(get_account_context)`.
- **Journals** are private personal data. Always authenticated.

### File Safety
- **Never use `rm`** to delete files. Move to `~/tmp/gemini_trash` instead.
- **Never commit `.env`** files, API keys, or passwords of any kind.
- Third-party credentials (Novi API key, Mailman credentials) live in the **MariaDB `site.cfg_json` column**, not in `.env` or code.

### Before Every Commit
1. `python3 -m py_compile <changed_file>` — syntax check
2. Restart Docker and verify the API starts clean: `docker compose restart ae_api`
3. Check logs: `docker compose logs -f ae_api` (look for startup errors or import failures)
4. Run the relevant E2E or unit test suite

### Before Starting Any Task
- Read `documentation/TODO__Agents.md` — active tasks, known bugs, and what was recently changed and why.
- Check `tests/README.md` — which test suite covers the area you're about to touch.

### Docker Restart is Mandatory After Python Changes
The API source is volume-mounted, so file edits appear instantly inside the container — but
**Gunicorn does NOT hot-reload**. You MUST run:
```bash
docker compose restart ae_api
```
after any Python file change before testing. This is the #1 cause of "my change didn't take effect."

---

## 3. Environment & Commands Cheat Sheet

```bash
# Start full stack
cd ~/OSIT_dev/aether_container_env && docker compose up -d

# Restart API after any Python change
docker compose restart ae_api

# Follow API logs
docker compose logs -f ae_api

# Shell into the container
docker compose exec ae_api bash

# Run unit tests (from project root)
./environment/bin/python3 -m pytest tests/unit/ -v

# Run a single E2E test
./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py

# Check a specific Python file compiles
./environment/bin/python3 -m py_compile app/methods/my_new_methods.py
```

**Development API:** `https://dev-api.oneskyit.com`
**Dev API secret key:** `nT0jPeiCfxSifkiDZur9jA`
**Standard test Agent API key:** `PMM4n50teUCaOMMTN8qOJA`

**Dev DB via phpMyAdmin:** `http://localhost:8081`

---

## 4. V3 Action Router Pattern

When adding a new action endpoint (not CRUD), follow this structure:

### File layout
```
app/methods/my_feature_methods.py   ← business logic
app/routers/api_v3_actions_my_feature.py  ← route handler (thin)
```

Then register in `app/routers/registry.py`:
```python
from app.routers import api_v3_actions_my_feature
# ...
app.include_router(api_v3_actions_my_feature.router, prefix='/v3/action/my_feature', tags=['My Feature (V3 Actions)'])
```

### Standard route handler pattern
```python
from fastapi import APIRouter, Depends
from app.lib_general_v3 import AccountContext, get_account_context, DelayParams
from app.models.response_models import Resp_Body_Base, mk_resp
from app.methods.my_feature_methods import my_business_logic

router = APIRouter()

@router.get('/my_endpoint/{obj_id}', response_model=Resp_Body_Base)
async def get_my_endpoint(
        obj_id: str,
        account: AccountContext = Depends(get_account_context),
        delay: DelayParams = Depends(),
        ):
    if delay.sleep_time_s > 0:
        await asyncio.sleep(delay.sleep_time_s)

    result = my_business_logic(obj_id)
    status = result.get('status', 503)

    if status == 200:
        return mk_resp(data=result['data'])

    return mk_resp(data=False, status_code=status, status_message=result.get('reason', 'Error.'))
```

### Auth dependency
`Depends(get_account_context)` is the **standard V3 gate**:
- Requires a valid `x-aether-api-key`
- Requires `x-account-id` OR a valid JWT session OR bypass mode
- Raises 403 if `auth_method == 'guest'` or no account context can be resolved
- For IDAA/private routes: this is the minimum gate. Never relax it.

---

## 5. Loading Site-Based Credentials (the `_load_idaa_cfg` Pattern)

Third-party credentials (Novi API key, Mailman credentials, etc.) are stored in MariaDB
in the `site.cfg_json` column for the relevant site record. Do NOT store them in `.env`.

The IDAA site record is:
- `id_random = '58_gJESdlUh'` (site id=17)
- Fields: `novi_api_root_url`, `novi_idaa_api_key`, `mailman_base_url`, `mailman_username`, `mailman_password`, `novi_mailman_sync`

Pattern to load them (use deferred import to avoid circular deps):
```python
IDAA_SITE_ID_RANDOM = '58_gJESdlUh'

def _load_idaa_cfg() -> dict:
    from app.db_sql import load_site_obj
    site = load_site_obj(IDAA_SITE_ID_RANDOM)
    if not site:
        raise RuntimeError('IDAA site record not found')
    cfg = site.get('cfg_json') or {}
    if isinstance(cfg, str):
        import json
        cfg = json.loads(cfg)
    return cfg
```

---

## 6. Redis Cache Pattern

```python
import json
import datetime
from app.lib_redis_helpers import redis_client

_CACHE_TTL = datetime.timedelta(hours=4)

def _cache_key(uuid: str) -> str:
    return f'idaa:novi_member:{uuid}'

# Reading
raw = redis_client.get(_cache_key(uuid))
if raw:
    return json.loads(raw)

# Writing (only on success — never cache error states)
redis_client.setex(_cache_key(uuid), _CACHE_TTL, json.dumps(result))
```

**Key naming convention:** `{module}:{object_type}:{identifier}` — e.g. `idaa:novi_member:{uuid}`

**Never cache:**
- 404 responses — the member may have just joined; next call should hit the source
- 429 / 503 errors — transient failures should not poison future callers

**Cache key scoping:** If the underlying data source is the same regardless of caller
(e.g. Novi credentials are hardcoded to the IDAA site — same UUID always returns same data),
drop `account_id` from the key. Per-caller scoping wastes Redis space and halves hit rate.

---

## 7. The `@logger_reset` Decorator — Unit Test Gotcha

Business logic methods use `@logger_reset` from `app.lib_general`:

```python
from app.lib_general import logger_reset

@logger_reset
def my_method(arg):
    ...
```

**In unit tests, this decorator MUST be mocked as a passthrough.**
If `app.lib_general` is replaced with a plain `MagicMock()`, the decorator becomes a
MagicMock, which when applied to a function replaces it with `MagicMock()()` — the decorated
function is gone and every call returns garbage.

```python
# WRONG — logger_reset becomes a MagicMock and swallows the function:
sys.modules['app.lib_general'] = MagicMock()

# CORRECT — make it a passthrough decorator:
mock_lib_general = MagicMock()
mock_lib_general.logger_reset = lambda f: f
sys.modules['app.lib_general'] = mock_lib_general
```

This is the #1 cause of unit tests returning `<MagicMock name='...'>` instead of real dicts.

---

## 8. Field Evolution Checklist

When a table or view gains, loses, or renames fields — **do all of these in order**:

1. Update the Pydantic model in `app/models/`
2. Update the SQL view or table projection so GET/SEARCH return the field
3. Update `searchable_fields` in `app/object_definitions/` (only for searchable fields)
4. Add virtual/view-only fields to `fields_to_exclude_from_db` if they must not be persisted
5. Run the relevant schema/search E2E tests
6. **Restart Docker:** `docker compose restart ae_api`

#### Alt-view fields (in `tbl_alt` only)
Some objects have a richer alternate SQL view triggered by `?view=alt`. These fields
**must still be declared in the Pydantic model and `searchable_fields`** even if they only
appear in the alt view — Pydantic strips undeclared fields silently.

---

## 9. Deferred Imports in Library Modules

To avoid circular dependency traps, **never import from `app.db_sql` or other app modules
at the top level of a library module**. Use deferred imports inside functions:

```python
# WRONG — causes circular import at startup:
from app.db_sql import load_site_obj

def my_func():
    return load_site_obj('abc')

# CORRECT:
def my_func():
    from app.db_sql import load_site_obj
    return load_site_obj('abc')
```

---

## 10. Pydantic / SQLAlchemy Version Pins — Do Not Remove

Current intentional pins:
- `pydantic==1.*`
- `SQLAlchemy==1.4.52`

A Pydantic V2 and SQLAlchemy 2.0 migration is planned but not started. Until then,
**do not upgrade these packages** — V2 touches every model definition and the migration
is a dedicated project.

---

## 11. Mistakes Agents Have Made on This Project

These are real incidents — know them before you start.

1. **IDAA data exposed publicly** — an agent removed an auth guard from the bulletin board
   router. Consequence: private IDAA recovery community data was publicly accessible.
   Always verify `Depends(get_account_context)` is present on every IDAA route.

2. **"My code change has no effect"** — Python file was edited but Docker was not restarted.
   The API runs Gunicorn inside Docker with no hot-reload. `docker compose restart ae_api`
   is required after every Python change.

3. **`@logger_reset` mock swallows functions in unit tests** — see Section 7 above.
   Symptom: `assert result['status'] == 200` fails with `TypeError: 'MagicMock' is not subscriptable`.

4. **`pytest` / `pytest-asyncio` not installed after venv rebuild** — these are dev-only
   dependencies not in `requirements.txt`. After any OS Python update (e.g., Arch Linux
   upgrading to a new Python minor), rebuild the venv and reinstall:
   ```bash
   ./environment/bin/pip install pytest pytest-asyncio
   ```

5. **Global `db` connection used instead of context manager** — `lib_sql_core.py` has a
   global `db = engine.connect()` that is a fragile single connection, not a pool.
   For new methods, prefer `engine.connect()` as a context manager. See `TODO__Agents.md`
   → "[P3 full]" task for the planned migration.

6. **`bypass` mode hardcodes `account_id=1`** — `x-no-account-id: bypass` resolves to
   `account_id=1` (One Sky IT Demo). Lookup overrides from the Demo account can leak into
   bypass sessions. Do not expand bypass usage without documenting the allowlist case.

7. **Caching error states in Redis** — caching 404 or 5xx responses poisons the cache.
   A member who just joined Novi would be denied for 4 hours if their 404 was cached.
   Only cache verified success (200) results.

8. **Not running `docker compose restart ae_api` between model changes and E2E tests** —
   the E2E suite hits the live API, which is still running the old code until restarted.
   Tests will pass or fail against stale behavior and the results are meaningless.

---

## 12. Test Patterns

### Unit tests (fast, no DB/network)
```bash
./environment/bin/python3 -m pytest tests/unit/ -v
```
- Mock all DB/Redis/HTTP at the top of the file before importing the module under test
- `@logger_reset` must be mocked as a passthrough (see Section 7)
- Always run from the **project root** — scripts use `sys.path.append(os.getcwd())`

### E2E tests (live API at dev-api.oneskyit.com)
```bash
./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py
```
- Require the Docker stack to be running with a working DB connection
- Use standard output format: `[✅ PASS]` / `[❌ FAIL]`
- Verify IDs in responses are **strings** (not integers) — ID Vision compliance

### Which tests to run
| Change type | Required suites |
|---|---|
| Model / ID Vision changes | `test_e2e_v3_demo_parity.py`, vision parity tests |
| Search / filter changes | `test_e2e_v3_search_engine.py` |
| Auth / account context changes | `test_e2e_v3_security_audit.py`, `test_e2e_v3_auth_security.py` |
| Any router or registry change | `test_e2e_v3_security_audit.py` |
| Novi-Mailman bridge changes | `test_e2e_v3_action_novi_mailman.py` |
| IDAA Novi verify changes | `tests/unit/test_unit_idaa_novi_verify.py` |

---

## 13. Source Layout (Quick Reference)

```
app/
  main.py                    — FastAPI app, lifespan, CORS
  config.py                  — Pydantic Settings (from .env via Docker)
  routers/
    registry.py              — All router registrations live here
    api_crud_v3.py           — Generic V3 CRUD (flat)
    api_crud_v3_nested.py    — Generic V3 CRUD (nested/parent-owned)
    api_v3_actions_*.py      — Action endpoints (one file per domain)
    dependencies_v3.py       — Shared FastAPI Depends() helpers
  models/                    — Pydantic V1 models (one file per domain)
  object_definitions/        — Per-object searchable_fields, field metadata
  ae_obj_types_def.py        — Object type registry (all Aether types defined here)
  methods/                   — Business logic (one file per feature/domain)
  lib_api_crud_v3.py         — Generic CRUD handler (all object types share this)
  lib_schema_v3.py           — Dynamic schema/field resolution per object type
  lib_general_v3.py          — AccountContext, get_account_context, DelayParams
  lib_general.py             — logger_reset and general utilities
  db_sql.py                  — Import facade (always import from here)
  lib_sql_core.py            — SQLAlchemy engine, global db connection (Source of Truth)
  lib_sql_crud.py            — sql_insert, sql_select, sql_update, etc.
  lib_redis_helpers.py       — redis_client global instance

tests/
  unit/                      — Isolated logic tests (mock everything, no DB)
  integration/               — Requires local MariaDB/Redis
  e2e/                       — Network tests against live dev API
  tools/                     — Admin utilities (stress test, registry generator)
  mock_config_helper.py      — Mock app.config.settings — use in all unit tests
  README.md                  — Which tests cover what; when to run them

documentation/
  TODO__Agents.md            — Active tasks + session notes ← always read first
  ARCH__V3_DEVELOPMENT_STANDARDS.md — Master V3 standards doc
  ARCH__V3_CORE.md           — Module architecture (lifespan, DB layers, logging)
  GUIDE__AE_API_V3_for_Frontend.md  — Frontend integration guide (keep current)
  GUIDE__DEVELOPMENT.md      — Commit SOP, verification checklist
```

---

## 14. Reading Order for Deeper Dives

| What you need | Read |
|---|---|
| Active tasks + known bugs | `documentation/TODO__Agents.md` ← always first |
| V3 standards and strategy | `documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md` |
| Module architecture | `documentation/ARCH__V3_CORE.md` |
| Frontend API integration guide | `documentation/GUIDE__AE_API_V3_for_Frontend.md` |
| WebSocket integration | `documentation/GUIDE__AE_API_V3_for_Frontend_websockets.md` |
| Commit / verification SOP | `documentation/GUIDE__DEVELOPMENT.md` |
| Which tests to run | `tests/README.md` |
| Object type registry | `app/ae_obj_types_def.py` |
| Shared agent docs | `~/agents_sync/aether/docs/UNIFIED_AGENT_ARCH.md` |