docs: add agent bootstrap quickstart and consolidate documentation

- Add BOOTSTRAP__AI_Agent_Quickstart.md — fast-path entry doc for AI agents
  covering critical rules, V3 action patterns, Redis/auth/logger_reset gotchas,
  and a mistakes-agents-have-made section
- Expand ARCH__V3_DEVELOPMENT_STANDARDS.md with merged content from
  ARCH__V3_CRUD_LEARNINGS: dependency injection reference, security/isolation
  model detail, and FastAPI/Pydantic gotchas
- Archive 4 outdated docs: GUIDE__LOCAL_DEVELOPMENT (predates Docker),
  FRONTEND_API_SAMPLES (belongs in SvelteKit repo), ARCH__UNIFIED_AGENT
  (replaced by MCP ecosystem), ARCH__V3_CRUD_LEARNINGS (content merged above)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Scott Idem
2026-05-19 18:56:09 -04:00
parent 221854df90
commit 051b2fd7ac
6 changed files with 461 additions and 1 deletions

View File

@@ -0,0 +1,102 @@
# Specification: Unified Aether AI Agent (UE-AE-01)
## 1. Vision & Purpose
The **Unified Aether AI Agent** is a single, cohesive AI entity designed to eliminate the friction of multi-agent coordination. It possesses "Total System Awareness," allowing it to understand how a change in the database schema on a remote server impacts the FastAPI backend, the Nginx proxy, and the SvelteKit frontend simultaneously.
---
## 2. System Architecture & Operational Domains
### A. Data Layer (MariaDB)
* **Location:** Separate Virtual Server (Remote VM).
* **Role:** Master data storage.
* **Agent Access Requirements:**
* Remote SQL execution capabilities.
* SSH access for database maintenance and schema inspection.
* Knowledge of cross-server connection strings and security groups.
### B. Caching & Messaging Layer (Redis)
* **Location:** Docker Container (Main Workstation).
* **Role:** Session management, ID resolution (Random ID mapping), and real-time messaging.
* **Agent Access Requirements:**
* Ability to execute `redis-cli` commands via Docker.
* Direct inspection of key-value pairs for troubleshooting.
### C. API Backend Layer (FastAPI / Python)
* **Location:** Docker Container (Main Workstation).
* **Role:** Business logic, CRUD V3 implementation, JWT authentication, and multi-tenant isolation.
* **Agent Access Requirements:**
* Full filesystem access to `osit-api-fastapi/`.
* Ability to manage Python environments and dependencies.
* Docker container management (logs, restarts, shell execution).
### D. Frontend Layer (SvelteKit / TypeScript)
* **Location:** Local Filesystem (Main Workstation).
* **Role:** User Interface, API consumption, and client-side state management.
* **Agent Access Requirements:**
* Full filesystem access to SvelteKit project directories.
* Ability to execute build tools (npm, vite) and linting (eslint, prettier).
* Browser automation for E2E testing (Playwright).
### E. Routing Layer (Nginx)
* **Location:** Host System or Docker.
* **Role:** SSL termination and reverse proxying for the API and Frontend.
* **Agent Access Requirements:**
* Ability to modify and reload Nginx configuration files.
* Diagnostic access to Nginx access/error logs.
### F. Storage Layer (Syncthing / Hosted Files)
* **Location:** `/home/scott/OSIT/hosted_files/` (Main Workstation) and synchronized Remote Servers.
* **Role:** Extremely important persistent storage for files served via the API (e.g., `hosted_file`, `event_file`).
* **API-Driven Access:** Files are frequently accessed/downloaded directly via API endpoints. The agent must understand the routing and security logic (JWT validation) governing these downloads.
* **Synchronization:** Managed via **Syncthing** (similar to the `agents_sync` directory), ensuring real-time mirroring across the Aether ecosystem.
* **Agent Access Requirements:**
* Full filesystem access to the local hosted files directory.
* Ability to verify synchronization status and resolve conflicts.
* Understanding of the relationship between file metadata in MariaDB and physical assets in this directory.
### G. Workstation Development Environment
* **Base Path:** `/home/scott/OSIT_dev/`
* **Project Repositories:**
* `aether_container_env/`: Docker Compose and environment configuration.
* `aether_api_fastapi/`: The Python/FastAPI backend source.
* `ae_app_svelte_tailwind_skeleton/`: The SvelteKit/TypeScript frontend source.
* **Network & Proxy Path:**
* Docker containers on the workstation are proxied via **Nginx on a separate Home Server** (Proxmox VM hosting Home Assistant, Jellyfin, Jitsi, etc.).
* **External Access URL:** `https://dev-api.oneskyit.com`
* **Agent Access Requirements:**
* Total awareness of the inter-connected paths between these three main directories.
* Knowledge of the home server's proxy logic to debug external connectivity vs. internal container health.
---
## 3. Communication & Context Strategy
### A. Integrated Global Memory
The Unified Agent will move away from separate `GEMINI.md` files in favor of a **Global System Context**. This context tracks:
1. **Service Map:** Mapping of ports, paths (e.g., `/v3/crud/`), and container IDs.
2. **Dependency Graph:** Visualizing how modules across different repositories interact.
3. **Boot Order Logic:** Understanding the fragile initialization requirements of the stack.
### B. Agent Sync Orchestration
The agent acts as the primary orchestrator for the `~/agents_sync/` directory:
- **Log Aggregation:** Pulling logs from MariaDB, FastAPI, and Nginx into a central diagnostic stream.
- **Inbound Messaging:** Processing user instructions from the `inbox/` and updating "System Health" status files.
---
## 4. Key Capabilities
1. **Cross-Stack Debugging:** Tracing a "500 Internal Server Error" from a Svelte fetch call, through the Nginx proxy, into the FastAPI logic, and finally identifying the missing column in the remote MariaDB table.
2. **Automated Schema Synchronization:** Reading Pydantic models and MariaDB table schemas to automatically generate and update TypeScript interfaces and `.editable_fields.ts` definitions in the Svelte project (`src/lib/ae_core/`).
3. **Log Stream Aggregation:** Simultaneous monitoring of Svelte console output, Nginx access/error logs, and FastAPI container logs to provide instant root-cause identification for cross-stack failures.
4. **Automated Lifecycle Management:** Orchestrating the "Change-Restart-Verify" loop. The agent should automatically trigger targeted Docker container restarts whenever backend code is modified to ensure the frontend is always interacting with the latest logic.
5. **Environment-Aware Refactoring:** Safely breaking up monolithic files (like `lib_general.py`) while knowing exactly which services are impacted and verifying them across the full stack.
6. **Automated Full-Stack Verification:** Writing a backend migration and a frontend UI component in a single turn, then verifying the integration with an automated test suite.
---
## 5. Security & Safety
- **Credential Isolation:** Secrets and API keys remain in `.env` files; the agent only manages the logic to use them.
- **Incremental Deployment:** Changes are applied service-by-service with health checks at every stage.
- **Sandboxing Awareness:** The agent operates with the knowledge that it is running directly on the user's workstation and remote infrastructure.

View File

@@ -0,0 +1,78 @@
# Aether API V3 CRUD: Architecture and Learnings
This document summarizes the development of the V3 CRUD API, the architectural choices made, and the lessons learned during the process.
## 1. V3 CRUD Architecture
The V3 CRUD API (`/v3/crud/`) is designed to run in parallel with legacy V1 and V2 endpoints. It introduces a hierarchical, nested URL structure and leverages modern FastAPI features for better maintainability and performance.
### Key Features:
- **Nested URL Structure**: Enforces parent-child relationships (e.g., `/v3/crud/site/{site_id}/site_domain/`).
- **Granular Dependencies**: Instead of a monolithic common parameters object, V3 uses specialized, reusable dependencies from `app/lib_general_v3.py`:
- `AccountContext`: Resolves account ID with clear precedence (Header > Query Token > Bypass Header).
- `PaginationParams`: Standardizes `limit` and `offset`.
- `StatusFilterParams`: Handles `enabled` and `hidden` status filtering.
- `SerializationParams`: Controls Pydantic serialization options (`by_alias`, `exclude_unset`).
- `DelayParams`: Facilitates optional latency simulation for testing.
- **Non-blocking Delay**: Uses `await asyncio.sleep()` to simulate network latency without blocking the Gunicorn worker's event loop.
- **Data-Driven Configuration**: Uses the modern format in `app/ae_obj_types_def.py` to map objects to tables and models.
- **Advanced Search (POST)**: Supports complex, nested filtering via `POST /v3/crud/{obj_type}/search`.
- Recursive AND/OR logic.
- Full operator support: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `is_null`, `is_not_null`, `like`, `contains`, `startswith`, `endswith`.
- Safe parameterization using unique generated names (e.g., `:sp_1`) to prevent collisions.
## 2. Backward Compatibility Strategy
To ensure that the introduction of V3 doesn't break legacy V1 and V2 endpoints:
- **Parallel Routes**: All V3 logic is isolated in `app/routers/api_crud_v3.py`.
- **Hybrid Configuration**: The `obj_type_kv_li` dictionary in `app/ae_obj_types_def.py` has been updated to include both modern keys (e.g., `tbl`, `mdl`) and legacy keys (e.g., `table_name`, `base_name`). This allows V2 endpoints to continue functioning normally while V3 endpoints use the refined structure.
- **Stable Core Imports**: Core modules like `app/lib_general.py` and `app/models/response_models.py` have been refactored to maintain their existing exports (`log`, `logging`, `mk_resp`) while internally adopting more robust module-level logging.
## 3. Learnings and Best Practices
### Logging and Startup Stability
- **Isolate Loggers**: Modules should instantiate their own module-level loggers (`logging.getLogger(__name__)`) rather than importing a global instance. This breaks circular dependencies and improves traceability.
- **Robust Configuration**: Logging configuration (`dictConfig`) should be wrapped in `try...except` to prevent application crashes due to environment issues (e.g., missing log directories in Docker).
- **Explicit Imports**: Always use `import logging.config` before calling `logging.config.dictConfig` to ensure the module is fully initialized.
### FastAPI and Pydantic
- **Dependency Injection**: Use `response: Response` as a type hint for standard injection. Avoid `Depends(Response)` as it is not a valid dependency provider and can cause router initialization failures.
- **Python Parameter Order**: In function signatures, non-default arguments (like `response: Response`) must precede arguments with default values or `Depends()`.
- **Async Concurrency**: Use `asyncio.sleep()` instead of `time.sleep()` in async endpoints. Blocking the event loop in a high-concurrency environment leads to worker timeouts and `502 Bad Gateway` errors.
- **Pydantic Compatibility**: Ensure all new models and utility functions remain compatible with Pydantic v1.10 until a full project-wide migration to v2 is planned. Avoid V2-only features like `computed_field` or `model_validator`.
## 4. Current Migration Status
The following objects have been migrated to the modern V3 configuration and are supported by the V3 API:
- `journal`, `journal_entry`
- `site`, `site_domain`
- `account`, `account_cfg`
- `address`, `contact`
- `order`, `order_line`
- `organization`, `page`
- `data_store`, `activity_log`
- `archive`, `archive_content`
- `hosted_file`, `post`, `post_comment`
- `person`, `user`
- `lu_country`, `lu_country_subdivision`, `lu_time_zone`
## 5. Security and Data Isolation
Implemented in Jan 2026, the V3 architecture enforces strict data isolation and role-aware authorization.
### Role-Aware Authorization (`AuthContext`)
- **Deferred User Lookup**: When a JWT is provided, the API performs a deferred database lookup via `load_user_obj` to populate `administrator`, `manager`, and `super` flags.
- **Role Scoping**: These flags are used to bypass account isolation for support and system maintenance tasks.
### Multi-Tenant Isolation
- **Fail-Closed Strategy (Feb 2026)**: The API enforces a strict "Fail-Closed" mindset. If an authentication context or account ID is missing, the system defaults to a blocking filter (e.g., `account_id IS NULL`). It no longer allows "Fail-Open" behavior where missing context returns all records.
- **IDAA Privacy Baseline**: Based on the International Drunk Doctors Anonymous requirement, the system defaults to **Maximum Isolation**. No object (Events, Files, Posts, Meetings) is public by default. Every object requires an `x-account-id` context, with the sole exception of `site_domain` used for bootstrapping.
- **Forced Account Filtering**: For all non-super users, the API automatically injects an `account_id` filter into every list and search query. This is enforced at the SQL level via `apply_forced_account_filter`.
- **Detailed 403 Feedback**: Security failures now return descriptive error messages (e.g., "API Key expired", "Account context required") to assist developers in debugging header/context issues while maintaining strict isolation.
- **Post-Retrieval Verification**: Single object retrievals (`GET`), updates (`PATCH`), and deletions (`DELETE`) include a secondary ownership check (`check_account_access`). If a user attempts to access an ID belonging to another account, they receive a `403 Forbidden` response.
- **Hierarchical Verification**: Child/Nested endpoints verify the ownership of the parent object before allowing operations on children, preventing cross-tenant "sideways" traversal.
- **Creation Guard**: When creating records (`POST`), the user's `account_id` is automatically forced onto the new record, preventing a user from creating data for another account.
### Bypass and Utility Access
- **X-No-Account-ID**: A specialized header used by development utilities and administrative scripts to bypass standard account isolation. This header grants `super` permissions and should only be used in trusted internal environments.
- **JWT Query Parameter**: Supported for direct file downloads and sharing links where custom headers cannot be provided.

View File

@@ -0,0 +1,347 @@
```ts
export async function get_ae_obj_id_crud({
api_cfg,
no_account_id = false,
obj_type,
obj_id,
use_alt_table = false,
use_alt_base = false,
inc = {},
enabled = 'enabled',
hidden = 'not_hidden',
limit = 999999,
offset = 0,
data = {},
// key,
// jwt = null,
headers = {},
params = {},
timeout = 25000,
return_meta = false,
log_lvl = 0
}: {
api_cfg: any;
no_account_id?: boolean;
obj_type: string;
obj_id: string;
use_alt_table?: boolean;
use_alt_base?: boolean;
inc?: any;
enabled?: 'enabled' | 'all' | 'not_enabled' | undefined;
hidden?: 'hidden' | 'all' | 'not_hidden' | undefined;
limit?: number;
offset?: number;
data?: any;
// key: string,
// jwt?: string,
headers?: any;
params?: key_val;
timeout?: number;
return_meta?: boolean;
log_lvl?: number;
}) {
```
```ts
// Updated 2024-05-23
export const patch_object = async function patch_object({
api_cfg = null,
endpoint = '',
params = {},
data = {},
return_meta = false,
log_lvl = 0,
retry_count = 5 // Number of retry attempts
}: {
api_cfg: any;
endpoint: string;
params?: any;
data?: any;
return_meta?: boolean;
log_lvl?: number;
retry_count?: number;
}) {
```
```ts
// Updated 2024-05-23
export const post_object = async function post_object({
api_cfg = null,
endpoint = '',
params = {},
data = {},
form_data = null,
return_meta = false,
return_blob = false,
filename = '',
auto_download = false,
// The task_id value should be a random string that is unique to the task. This is used to identify the task in the message event.
task_id = crypto.randomUUID(),
log_lvl = 0,
retry_count = 5
}: {
api_cfg: any;
endpoint: string;
params?: any;
data?: any;
form_data?: any;
return_meta?: boolean;
return_blob?: boolean;
filename?: string;
auto_download?: boolean;
task_id?: string;
log_lvl?: number;
retry_count?: number;
}) {
```
```ts
// Updated 2024-05-23
export const delete_object = async function delete_object({
api_cfg = null,
endpoint = '',
params = {},
data = {},
return_meta = false,
log_lvl = 0,
retry_count = 5 // Number of retry attempts
}: {
api_cfg: any;
endpoint: string;
params?: any;
data?: any;
return_meta?: boolean;
log_lvl?: number;
retry_count?: number;
}) {
```
```ts
import type { key_val } from '$lib/stores/ae_stores';
import { get_object } from './api_get_object';
// Refactored 2025-11-13 to use a lookup map for endpoints.
const objTypeToEndpointMap: Record<string, string> = {
account: '/crud/account/list',
address: '/crud/address/list',
archive: '/crud/archive/list',
archive_content: '/crud/archive/content/list',
activity_log: '/crud/activity_log/list',
contact: '/crud/contact/list',
data_store: '/crud/data_store/list',
event: '/crud/event/list',
event_abstract: '/crud/event/abstract/list',
event_badge: '/crud/event/badge/list',
event_badge_template: '/crud/event/badge/template/list',
event_device: '/crud/event/device/list',
event_exhibit: '/crud/event/exhibit/list',
event_exhibit_tracking: '/crud/event/exhibit/tracking/list',
event_file: '/crud/event/file/list',
event_location: '/crud/event/location/list',
event_person: '/crud/event/person/list',
event_presentation: '/crud/event/presentation/list',
event_presenter: '/crud/event/presenter/list',
event_session: '/crud/event/session/list',
event_track: '/crud/event/track/list',
grant: '/crud/grant/list',
hosted_file: '/crud/hosted_file/list',
journal: '/crud/journal/list',
journal_entry: '/crud/journal/entry/list',
order: '/crud/order/list',
order_line: '/crud/order/line/list',
page: '/crud/page/list',
person: '/crud/person/list',
post: '/crud/post/list',
post_comment: '/crud/post/comment/list',
site: '/crud/site/list',
sponsorship_cfg: '/crud/sponsorship/cfg/list',
sponsorship: '/crud/sponsorship/list',
// user: '/crud/user/list',
'lu-country_subdivision': '/crud/lu/country_subdivision/list',
'lu-country': '/crud/lu/country/list',
'lu-time_zone': '/crud/lu/time_zone/list'
};
function getEndpointForObjType(obj_type: string, for_obj_type?: string): string {
if (obj_type === 'lu' && for_obj_type) {
const key = `lu-${for_obj_type}`;
const endpoint = objTypeToEndpointMap[key];
if (endpoint) return endpoint;
}
const endpoint = objTypeToEndpointMap[obj_type];
if (endpoint) return endpoint;
throw new Error(`Unknown object type: ${obj_type}`);
}
type OrderBy = { [key: string]: 'ASC' | 'DESC' };
interface GetAeObjLiForObjIdCrudV2Params {
api_cfg: any; // Consider defining a specific type for api_cfg
obj_type: string;
for_obj_type: string;
for_obj_id?: string;
use_alt_tbl?: boolean | string;
use_alt_mdl?: boolean | string;
use_alt_exp?: boolean | string;
inc?: key_val;
enabled?: 'all' | 'enabled' | 'not_enabled';
hidden?: 'all' | 'hidden' | 'not_hidden';
order_by_li?: OrderBy[] | null;
limit?: number;
offset?: number;
headers?: Record<string, string>;
params_json?: any;
params?: key_val;
log_lvl?: number;
}
export async function get_ae_obj_li_for_obj_id_crud_v2({
api_cfg,
obj_type,
for_obj_type,
for_obj_id,
use_alt_tbl = false,
use_alt_mdl = false,
use_alt_exp = false,
enabled = 'enabled',
hidden = 'not_hidden',
order_by_li = null,
limit = 999999,
offset = 0,
headers = {},
params_json = null,
params = {},
log_lvl = 0
}: GetAeObjLiForObjIdCrudV2Params) {
if (log_lvl) {
console.log('*** get_ae_obj_li_for_obj_id_crud_v2() ***');
}
try {
const endpoint = `/v2${getEndpointForObjType(obj_type, for_obj_type)}`;
if (log_lvl) {
console.log('Endpoint:', endpoint);
}
// We need to remove a few parameters from the params object that are not allowed.
delete params['qry__enabled'];
delete params['qry__hidden'];
delete params['qry__limit'];
delete params['qry__offset'];
if (for_obj_type) params['for_obj_type'] = for_obj_type;
if (for_obj_id) params['for_obj_id'] = for_obj_id;
if (use_alt_tbl === true) params['tbl_alt'] = 'alt';
if (use_alt_mdl === true) params['mdl_alt'] = 'alt';
if (use_alt_exp === true) params['exp_alt'] = 'alt';
const allowed_enabled_list = ['all', 'enabled', 'not_enabled'];
if (allowed_enabled_list.includes(enabled)) {
params['enabled'] = enabled;
}
const allowed_hidden_list = ['all', 'hidden', 'not_hidden'];
if (allowed_hidden_list.includes(hidden)) {
params['hidden'] = hidden;
}
// NOTE: The order_by_li variable is in the "headers" because URL GET params do not handle complex objects very well.
if (order_by_li) {
headers['order_by_li'] = JSON.stringify(order_by_li);
}
if (limit > 0) params['limit'] = limit;
if (offset > 0) params['offset'] = offset;
if (params_json) {
// NOTE: "jp" stands for "JSON Params". This is a JSON object that needs to be safely converted to a string for the params.
// Max characters for a GET request is ~2000. This is a limitation of the browser.
const json_params_str = encodeURIComponent(JSON.stringify(params_json));
if (json_params_str.length > 2083) {
// Using console.error instead of throwing an error to avoid crashing the app for a known limitation.
console.error(
`The JSON object is too large to be used as a GET parameter. Max length is 2083 characters. Length = ${json_params_str.length}`
);
return false;
}
params['jp'] = json_params_str;
}
if (log_lvl) {
console.log('Params:', params);
}
const object_li_get_promise = await get_object({
api_cfg: api_cfg,
endpoint: endpoint,
headers: headers,
params: params,
log_lvl: log_lvl
});
if (log_lvl > 1) {
console.log(object_li_get_promise);
}
return object_li_get_promise;
} catch (error) {
console.error('Error in get_ae_obj_li_for_obj_id_crud_v2:', error);
return false; // Or handle the error as appropriate
}
}
```
```ts
/**
* Aether V3 Search (POST)
* Standardized search with recursive logical grouping and wildcard support.
*/
export async function search_ae_obj_v3({
api_cfg,
obj_type,
search_query, // Example: { q: "%", and: [{ field: "enable", op: "eq", value: true }] }
view = 'default',
limit = 100,
offset = 0,
log_lvl = 0
}) {
const endpoint = `/v3/crud/${obj_type}/search`;
const params = { view, limit, offset };
return await post_object({
api_cfg,
endpoint,
params,
data: search_query,
log_lvl
});
}
/**
* Aether V3 Schema Discovery (GET)
* Returns database and Pydantic model metadata for an object.
*/
export async function get_obj_schema_v3({ api_cfg, obj_type, view = 'default' }) {
const endpoint = `/v3/crud/${obj_type}/schema`;
const params = { view };
return await get_object({ api_cfg, endpoint, params });
}
/**
* Initial Site/Domain Resolution (Legacy V1/V2 Public Route)
* Used to bootstrap the app context (resolve account_id) from the current FQDN.
*/
export async function resolve_site_context({ api_cfg, fqdn }) {
// This specific path bypasses X-Account-ID requirement
const endpoint = `/crud/site/domain/${fqdn}`;
const params = { use_alt_table: true, use_alt_base: true };
return await get_object({ api_cfg, endpoint, params });
}
```

View File

@@ -0,0 +1,78 @@
# Local Development Guide
This guide provides instructions for setting up and running the Aether API locally for development.
## 1. Prerequisites
- Python 3.9+ (or as specified in `requirements.txt`)
- `virtualenv` or `venv`
## 2. Setup Procedure
### Create and Activate Environment
```bash
# Go to root of application
cd ~/path/to/directory/aether_api_fastapi/
# Create new application environment
virtualenv environment
# Activate application environment
source environment/bin/activate
```
### Install Requirements
```bash
# Install application requirements
pip install -r admin/requirements.txt
# Troubleshooting/Force Reinstall if needed:
# pip install --upgrade --force-reinstall -r admin/requirements.txt
# pip install --ignore-installed -r admin/requirements.txt
```
## 3. Running the Application
### Start with Uvicorn (Reload enabled)
```bash
uvicorn app.main:app --host 0.0.0.0 --port 5005 --reload
```
### Accessing the API
The application will be available at:
- API Root: [http://localhost:5005](http://localhost:5005)
- Swagger Docs: [http://localhost:5005/docs](http://localhost:5005/docs)
## 4. Git Workflow Basics
### Initialization
```bash
git init
git remote add origin https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api.git
```
### Basic Commands
```bash
# Adding and committing
git add .
git commit -m 'Your commit message'
# Pushing to branches
git push -u origin master
git push -u origin development
```
### Branch Management
```bash
# List branches
git branch -a
# Create and switch to new branch
git branch new-branch-name
git switch new-branch-name
```
## 5. Deployment Basics (Manual)
If you are cloning for the first time on a server:
```bash
git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api-fastapi.git /srv/http/destination_path
```