docs: add agent bootstrap quickstart and consolidate documentation
- Add BOOTSTRAP__AI_Agent_Quickstart.md — fast-path entry doc for AI agents covering critical rules, V3 action patterns, Redis/auth/logger_reset gotchas, and a mistakes-agents-have-made section - Expand ARCH__V3_DEVELOPMENT_STANDARDS.md with merged content from ARCH__V3_CRUD_LEARNINGS: dependency injection reference, security/isolation model detail, and FastAPI/Pydantic gotchas - Archive 4 outdated docs: GUIDE__LOCAL_DEVELOPMENT (predates Docker), FRONTEND_API_SAMPLES (belongs in SvelteKit repo), ARCH__UNIFIED_AGENT (replaced by MCP ecosystem), ARCH__V3_CRUD_LEARNINGS (content merged above) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Scott Idem
420
documentation/BOOTSTRAP__AI_Agent_Quickstart.md
Normal file
420
documentation/BOOTSTRAP__AI_Agent_Quickstart.md
Normal file
@@ -0,0 +1,420 @@
|
||||
# Aether API — AI Agent Bootstrap / Quickstart
|
||||
> **Read this first.** This doc is the fast path to being productive on this project.
|
||||
> It covers the rules, patterns, and gotchas that matter most.
|
||||
> Deep dives are in the linked docs at the bottom.
|
||||
|
||||
---
|
||||
|
||||
## 1. What This Project Is
|
||||
|
||||
**Aether** is an event management platform built by One Sky IT (Scott Idem).
|
||||
This repo is the backend: **FastAPI + MariaDB**, running inside Docker.
|
||||
|
||||
The frontend (`aether_app_sveltekit/`) talks to this API exclusively via the V3 REST API.
|
||||
There is **no standalone dev server** — the API runs only in Docker.
|
||||
|
||||
**Key clients:**
|
||||
- **Conference organizers** — Presentation management, badges, sessions
|
||||
- **Exhibitors** — Leads capture
|
||||
- **IDAA** — International Doctors in Alcoholics Anonymous (strictly private medical/recovery community)
|
||||
|
||||
**Stack at a glance:**
|
||||
|
||||
| Layer | Technology |
|
||||
|---|---|
|
||||
| Framework | FastAPI + Pydantic V1 (upgrade deferred) |
|
||||
| Database | MariaDB via SQLAlchemy 1.4 (upgrade deferred) |
|
||||
| Cache | Redis |
|
||||
| Auth | Custom headers: `x-aether-api-key` + `x-account-id` |
|
||||
| Container | Docker / Gunicorn — source is volume-mounted (no rebuild for Python changes, but **restart required**) |
|
||||
|
||||
---
|
||||
|
||||
## 2. Critical Rules — Read Before Touching Any Code
|
||||
|
||||
### Privacy (Sev-1 class failures if violated)
|
||||
- **IDAA content is ALWAYS private.** All routes under `/idaa/` require authentication.
|
||||
A previous agent accidentally exposed IDAA bulletin board data publicly.
|
||||
This is the single most serious class of mistake on this project.
|
||||
When in doubt — it's private. Always verify with `Depends(get_account_context)`.
|
||||
- **Journals** are private personal data. Always authenticated.
|
||||
|
||||
### File Safety
|
||||
- **Never use `rm`** to delete files. Move to `~/tmp/gemini_trash` instead.
|
||||
- **Never commit `.env`** files, API keys, or passwords of any kind.
|
||||
- Third-party credentials (Novi API key, Mailman credentials) live in the **MariaDB `site.cfg_json` column**, not in `.env` or code.
|
||||
|
||||
### Before Every Commit
|
||||
1. `python3 -m py_compile <changed_file>` — syntax check
|
||||
2. Restart Docker and verify the API starts clean: `docker compose restart ae_api`
|
||||
3. Check logs: `docker compose logs -f ae_api` (look for startup errors or import failures)
|
||||
4. Run the relevant E2E or unit test suite
|
||||
|
||||
### Before Starting Any Task
|
||||
- Read `documentation/TODO__Agents.md` — active tasks, known bugs, and what was recently changed and why.
|
||||
- Check `tests/README.md` — which test suite covers the area you're about to touch.
|
||||
|
||||
### Docker Restart is Mandatory After Python Changes
|
||||
The API source is volume-mounted, so file edits appear instantly inside the container — but
|
||||
**Gunicorn does NOT hot-reload**. You MUST run:
|
||||
```bash
|
||||
docker compose restart ae_api
|
||||
```
|
||||
after any Python file change before testing. This is the #1 cause of "my change didn't take effect."
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment & Commands Cheat Sheet
|
||||
|
||||
```bash
|
||||
# Start full stack
|
||||
cd ~/OSIT_dev/aether_container_env && docker compose up -d
|
||||
|
||||
# Restart API after any Python change
|
||||
docker compose restart ae_api
|
||||
|
||||
# Follow API logs
|
||||
docker compose logs -f ae_api
|
||||
|
||||
# Shell into the container
|
||||
docker compose exec ae_api bash
|
||||
|
||||
# Run unit tests (from project root)
|
||||
./environment/bin/python3 -m pytest tests/unit/ -v
|
||||
|
||||
# Run a single E2E test
|
||||
./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py
|
||||
|
||||
# Check a specific Python file compiles
|
||||
./environment/bin/python3 -m py_compile app/methods/my_new_methods.py
|
||||
```
|
||||
|
||||
**Development API:** `https://dev-api.oneskyit.com`
|
||||
**Dev API secret key:** `nT0jPeiCfxSifkiDZur9jA`
|
||||
**Standard test Agent API key:** `PMM4n50teUCaOMMTN8qOJA`
|
||||
|
||||
**Dev DB via phpMyAdmin:** `http://localhost:8081`
|
||||
|
||||
---
|
||||
|
||||
## 4. V3 Action Router Pattern
|
||||
|
||||
When adding a new action endpoint (not CRUD), follow this structure:
|
||||
|
||||
### File layout
|
||||
```
|
||||
app/methods/my_feature_methods.py ← business logic
|
||||
app/routers/api_v3_actions_my_feature.py ← route handler (thin)
|
||||
```
|
||||
|
||||
Then register in `app/routers/registry.py`:
|
||||
```python
|
||||
from app.routers import api_v3_actions_my_feature
|
||||
# ...
|
||||
app.include_router(api_v3_actions_my_feature.router, prefix='/v3/action/my_feature', tags=['My Feature (V3 Actions)'])
|
||||
```
|
||||
|
||||
### Standard route handler pattern
|
||||
```python
|
||||
from fastapi import APIRouter, Depends
|
||||
from app.lib_general_v3 import AccountContext, get_account_context, DelayParams
|
||||
from app.models.response_models import Resp_Body_Base, mk_resp
|
||||
from app.methods.my_feature_methods import my_business_logic
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.get('/my_endpoint/{obj_id}', response_model=Resp_Body_Base)
|
||||
async def get_my_endpoint(
|
||||
obj_id: str,
|
||||
account: AccountContext = Depends(get_account_context),
|
||||
delay: DelayParams = Depends(),
|
||||
):
|
||||
if delay.sleep_time_s > 0:
|
||||
await asyncio.sleep(delay.sleep_time_s)
|
||||
|
||||
result = my_business_logic(obj_id)
|
||||
status = result.get('status', 503)
|
||||
|
||||
if status == 200:
|
||||
return mk_resp(data=result['data'])
|
||||
|
||||
return mk_resp(data=False, status_code=status, status_message=result.get('reason', 'Error.'))
|
||||
```
|
||||
|
||||
### Auth dependency
|
||||
`Depends(get_account_context)` is the **standard V3 gate**:
|
||||
- Requires a valid `x-aether-api-key`
|
||||
- Requires `x-account-id` OR a valid JWT session OR bypass mode
|
||||
- Raises 403 if `auth_method == 'guest'` or no account context can be resolved
|
||||
- For IDAA/private routes: this is the minimum gate. Never relax it.
|
||||
|
||||
---
|
||||
|
||||
## 5. Loading Site-Based Credentials (the `_load_idaa_cfg` Pattern)
|
||||
|
||||
Third-party credentials (Novi API key, Mailman credentials, etc.) are stored in MariaDB
|
||||
in the `site.cfg_json` column for the relevant site record. Do NOT store them in `.env`.
|
||||
|
||||
The IDAA site record is:
|
||||
- `id_random = '58_gJESdlUh'` (site id=17)
|
||||
- Fields: `novi_api_root_url`, `novi_idaa_api_key`, `mailman_base_url`, `mailman_username`, `mailman_password`, `novi_mailman_sync`
|
||||
|
||||
Pattern to load them (use deferred import to avoid circular deps):
|
||||
```python
|
||||
IDAA_SITE_ID_RANDOM = '58_gJESdlUh'
|
||||
|
||||
def _load_idaa_cfg() -> dict:
|
||||
from app.db_sql import load_site_obj
|
||||
site = load_site_obj(IDAA_SITE_ID_RANDOM)
|
||||
if not site:
|
||||
raise RuntimeError('IDAA site record not found')
|
||||
cfg = site.get('cfg_json') or {}
|
||||
if isinstance(cfg, str):
|
||||
import json
|
||||
cfg = json.loads(cfg)
|
||||
return cfg
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Redis Cache Pattern
|
||||
|
||||
```python
|
||||
import json
|
||||
import datetime
|
||||
from app.lib_redis_helpers import redis_client
|
||||
|
||||
_CACHE_TTL = datetime.timedelta(hours=4)
|
||||
|
||||
def _cache_key(uuid: str) -> str:
|
||||
return f'idaa:novi_member:{uuid}'
|
||||
|
||||
# Reading
|
||||
raw = redis_client.get(_cache_key(uuid))
|
||||
if raw:
|
||||
return json.loads(raw)
|
||||
|
||||
# Writing (only on success — never cache error states)
|
||||
redis_client.setex(_cache_key(uuid), _CACHE_TTL, json.dumps(result))
|
||||
```
|
||||
|
||||
**Key naming convention:** `{module}:{object_type}:{identifier}` — e.g. `idaa:novi_member:{uuid}`
|
||||
|
||||
**Never cache:**
|
||||
- 404 responses — the member may have just joined; next call should hit the source
|
||||
- 429 / 503 errors — transient failures should not poison future callers
|
||||
|
||||
**Cache key scoping:** If the underlying data source is the same regardless of caller
|
||||
(e.g. Novi credentials are hardcoded to the IDAA site — same UUID always returns same data),
|
||||
drop `account_id` from the key. Per-caller scoping wastes Redis space and halves hit rate.
|
||||
|
||||
---
|
||||
|
||||
## 7. The `@logger_reset` Decorator — Unit Test Gotcha
|
||||
|
||||
Business logic methods use `@logger_reset` from `app.lib_general`:
|
||||
|
||||
```python
|
||||
from app.lib_general import logger_reset
|
||||
|
||||
@logger_reset
|
||||
def my_method(arg):
|
||||
...
|
||||
```
|
||||
|
||||
**In unit tests, this decorator MUST be mocked as a passthrough.**
|
||||
If `app.lib_general` is replaced with a plain `MagicMock()`, the decorator becomes a
|
||||
MagicMock, which when applied to a function replaces it with `MagicMock()()` — the decorated
|
||||
function is gone and every call returns garbage.
|
||||
|
||||
```python
|
||||
# WRONG — logger_reset becomes a MagicMock and swallows the function:
|
||||
sys.modules['app.lib_general'] = MagicMock()
|
||||
|
||||
# CORRECT — make it a passthrough decorator:
|
||||
mock_lib_general = MagicMock()
|
||||
mock_lib_general.logger_reset = lambda f: f
|
||||
sys.modules['app.lib_general'] = mock_lib_general
|
||||
```
|
||||
|
||||
This is the #1 cause of unit tests returning `<MagicMock name='...'>` instead of real dicts.
|
||||
|
||||
---
|
||||
|
||||
## 8. Field Evolution Checklist
|
||||
|
||||
When a table or view gains, loses, or renames fields — **do all of these in order**:
|
||||
|
||||
1. Update the Pydantic model in `app/models/`
|
||||
2. Update the SQL view or table projection so GET/SEARCH return the field
|
||||
3. Update `searchable_fields` in `app/object_definitions/` (only for searchable fields)
|
||||
4. Add virtual/view-only fields to `fields_to_exclude_from_db` if they must not be persisted
|
||||
5. Run the relevant schema/search E2E tests
|
||||
6. **Restart Docker:** `docker compose restart ae_api`
|
||||
|
||||
#### Alt-view fields (in `tbl_alt` only)
|
||||
Some objects have a richer alternate SQL view triggered by `?view=alt`. These fields
|
||||
**must still be declared in the Pydantic model and `searchable_fields`** even if they only
|
||||
appear in the alt view — Pydantic strips undeclared fields silently.
|
||||
|
||||
---
|
||||
|
||||
## 9. Deferred Imports in Library Modules
|
||||
|
||||
To avoid circular dependency traps, **never import from `app.db_sql` or other app modules
|
||||
at the top level of a library module**. Use deferred imports inside functions:
|
||||
|
||||
```python
|
||||
# WRONG — causes circular import at startup:
|
||||
from app.db_sql import load_site_obj
|
||||
|
||||
def my_func():
|
||||
return load_site_obj('abc')
|
||||
|
||||
# CORRECT:
|
||||
def my_func():
|
||||
from app.db_sql import load_site_obj
|
||||
return load_site_obj('abc')
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Pydantic / SQLAlchemy Version Pins — Do Not Remove
|
||||
|
||||
Current intentional pins:
|
||||
- `pydantic==1.*`
|
||||
- `SQLAlchemy==1.4.52`
|
||||
|
||||
A Pydantic V2 and SQLAlchemy 2.0 migration is planned but not started. Until then,
|
||||
**do not upgrade these packages** — V2 touches every model definition and the migration
|
||||
is a dedicated project.
|
||||
|
||||
---
|
||||
|
||||
## 11. Mistakes Agents Have Made on This Project
|
||||
|
||||
These are real incidents — know them before you start.
|
||||
|
||||
1. **IDAA data exposed publicly** — an agent removed an auth guard from the bulletin board
|
||||
router. Consequence: private IDAA recovery community data was publicly accessible.
|
||||
Always verify `Depends(get_account_context)` is present on every IDAA route.
|
||||
|
||||
2. **"My code change has no effect"** — Python file was edited but Docker was not restarted.
|
||||
The API runs Gunicorn inside Docker with no hot-reload. `docker compose restart ae_api`
|
||||
is required after every Python change.
|
||||
|
||||
3. **`@logger_reset` mock swallows functions in unit tests** — see Section 7 above.
|
||||
Symptom: `assert result['status'] == 200` fails with `TypeError: 'MagicMock' is not subscriptable`.
|
||||
|
||||
4. **`pytest` / `pytest-asyncio` not installed after venv rebuild** — these are dev-only
|
||||
dependencies not in `requirements.txt`. After any OS Python update (e.g., Arch Linux
|
||||
upgrading to a new Python minor), rebuild the venv and reinstall:
|
||||
```bash
|
||||
./environment/bin/pip install pytest pytest-asyncio
|
||||
```
|
||||
|
||||
5. **Global `db` connection used instead of context manager** — `lib_sql_core.py` has a
|
||||
global `db = engine.connect()` that is a fragile single connection, not a pool.
|
||||
For new methods, prefer `engine.connect()` as a context manager. See `TODO__Agents.md`
|
||||
→ "[P3 full]" task for the planned migration.
|
||||
|
||||
6. **`bypass` mode hardcodes `account_id=1`** — `x-no-account-id: bypass` resolves to
|
||||
`account_id=1` (One Sky IT Demo). Lookup overrides from the Demo account can leak into
|
||||
bypass sessions. Do not expand bypass usage without documenting the allowlist case.
|
||||
|
||||
7. **Caching error states in Redis** — caching 404 or 5xx responses poisons the cache.
|
||||
A member who just joined Novi would be denied for 4 hours if their 404 was cached.
|
||||
Only cache verified success (200) results.
|
||||
|
||||
8. **Not running `docker compose restart ae_api` between model changes and E2E tests** —
|
||||
the E2E suite hits the live API, which is still running the old code until restarted.
|
||||
Tests will pass or fail against stale behavior and the results are meaningless.
|
||||
|
||||
---
|
||||
|
||||
## 12. Test Patterns
|
||||
|
||||
### Unit tests (fast, no DB/network)
|
||||
```bash
|
||||
./environment/bin/python3 -m pytest tests/unit/ -v
|
||||
```
|
||||
- Mock all DB/Redis/HTTP at the top of the file before importing the module under test
|
||||
- `@logger_reset` must be mocked as a passthrough (see Section 7)
|
||||
- Always run from the **project root** — scripts use `sys.path.append(os.getcwd())`
|
||||
|
||||
### E2E tests (live API at dev-api.oneskyit.com)
|
||||
```bash
|
||||
./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py
|
||||
```
|
||||
- Require the Docker stack to be running with a working DB connection
|
||||
- Use standard output format: `[✅ PASS]` / `[❌ FAIL]`
|
||||
- Verify IDs in responses are **strings** (not integers) — ID Vision compliance
|
||||
|
||||
### Which tests to run
|
||||
| Change type | Required suites |
|
||||
|---|---|
|
||||
| Model / ID Vision changes | `test_e2e_v3_demo_parity.py`, vision parity tests |
|
||||
| Search / filter changes | `test_e2e_v3_search_engine.py` |
|
||||
| Auth / account context changes | `test_e2e_v3_security_audit.py`, `test_e2e_v3_auth_security.py` |
|
||||
| Any router or registry change | `test_e2e_v3_security_audit.py` |
|
||||
| Novi-Mailman bridge changes | `test_e2e_v3_action_novi_mailman.py` |
|
||||
| IDAA Novi verify changes | `tests/unit/test_unit_idaa_novi_verify.py` |
|
||||
|
||||
---
|
||||
|
||||
## 13. Source Layout (Quick Reference)
|
||||
|
||||
```
|
||||
app/
|
||||
main.py — FastAPI app, lifespan, CORS
|
||||
config.py — Pydantic Settings (from .env via Docker)
|
||||
routers/
|
||||
registry.py — All router registrations live here
|
||||
api_crud_v3.py — Generic V3 CRUD (flat)
|
||||
api_crud_v3_nested.py — Generic V3 CRUD (nested/parent-owned)
|
||||
api_v3_actions_*.py — Action endpoints (one file per domain)
|
||||
dependencies_v3.py — Shared FastAPI Depends() helpers
|
||||
models/ — Pydantic V1 models (one file per domain)
|
||||
object_definitions/ — Per-object searchable_fields, field metadata
|
||||
ae_obj_types_def.py — Object type registry (all Aether types defined here)
|
||||
methods/ — Business logic (one file per feature/domain)
|
||||
lib_api_crud_v3.py — Generic CRUD handler (all object types share this)
|
||||
lib_schema_v3.py — Dynamic schema/field resolution per object type
|
||||
lib_general_v3.py — AccountContext, get_account_context, DelayParams
|
||||
lib_general.py — logger_reset and general utilities
|
||||
db_sql.py — Import facade (always import from here)
|
||||
lib_sql_core.py — SQLAlchemy engine, global db connection (Source of Truth)
|
||||
lib_sql_crud.py — sql_insert, sql_select, sql_update, etc.
|
||||
lib_redis_helpers.py — redis_client global instance
|
||||
|
||||
tests/
|
||||
unit/ — Isolated logic tests (mock everything, no DB)
|
||||
integration/ — Requires local MariaDB/Redis
|
||||
e2e/ — Network tests against live dev API
|
||||
tools/ — Admin utilities (stress test, registry generator)
|
||||
mock_config_helper.py — Mock app.config.settings — use in all unit tests
|
||||
README.md — Which tests cover what; when to run them
|
||||
|
||||
documentation/
|
||||
TODO__Agents.md — Active tasks + session notes ← always read first
|
||||
ARCH__V3_DEVELOPMENT_STANDARDS.md — Master V3 standards doc
|
||||
ARCH__V3_CORE.md — Module architecture (lifespan, DB layers, logging)
|
||||
GUIDE__AE_API_V3_for_Frontend.md — Frontend integration guide (keep current)
|
||||
GUIDE__DEVELOPMENT.md — Commit SOP, verification checklist
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 14. Reading Order for Deeper Dives
|
||||
|
||||
| What you need | Read |
|
||||
|---|---|
|
||||
| Active tasks + known bugs | `documentation/TODO__Agents.md` ← always first |
|
||||
| V3 standards and strategy | `documentation/ARCH__V3_DEVELOPMENT_STANDARDS.md` |
|
||||
| Module architecture | `documentation/ARCH__V3_CORE.md` |
|
||||
| Frontend API integration guide | `documentation/GUIDE__AE_API_V3_for_Frontend.md` |
|
||||
| WebSocket integration | `documentation/GUIDE__AE_API_V3_for_Frontend_websockets.md` |
|
||||
| Commit / verification SOP | `documentation/GUIDE__DEVELOPMENT.md` |
|
||||
| Which tests to run | `tests/README.md` |
|
||||
| Object type registry | `app/ae_obj_types_def.py` |
|
||||
| Shared agent docs | `~/agents_sync/aether/docs/UNIFIED_AGENT_ARCH.md` |
|
||||
Reference in New Issue
Block a user