OSIT-AE-API-FastAPI/documentation/ARCH__V3_CRUD_LEARNINGS.md

# Aether API V3 CRUD: Architecture and Learnings

This document summarizes the development of the V3 CRUD API, the architectural choices made, and the lessons learned during the process.

## 1. V3 CRUD Architecture

The V3 CRUD API (`/v3/crud/`) is designed to run in parallel with legacy V1 and V2 endpoints. It introduces a hierarchical, nested URL structure and leverages modern FastAPI features for better maintainability and performance.

### Key Features:
- **Nested URL Structure**: Enforces parent-child relationships (e.g., `/v3/crud/site/{site_id}/site_domain/`).
- **Granular Dependencies**: Instead of a monolithic common parameters object, V3 uses specialized, reusable dependencies from `app/lib_general_v3.py`:
    - `AccountContext`: Resolves account ID with clear precedence (Header > Query Token > Bypass Header).
    - `PaginationParams`: Standardizes `limit` and `offset`.
    - `StatusFilterParams`: Handles `enabled` and `hidden` status filtering.
    - `SerializationParams`: Controls Pydantic serialization options (`by_alias`, `exclude_unset`).
    - `DelayParams`: Facilitates optional latency simulation for testing.
- **Non-blocking Delay**: Uses `await asyncio.sleep()` to simulate network latency without blocking the Gunicorn worker's event loop.
- **Data-Driven Configuration**: Uses the modern format in `app/ae_obj_types_def.py` to map objects to tables and models.
- **Advanced Search (POST)**: Supports complex, nested filtering via `POST /v3/crud/{obj_type}/search`.
    - Recursive AND/OR logic.
    - Full operator support: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `is_null`, `is_not_null`, `like`, `contains`, `startswith`, `endswith`.
    - Safe parameterization using unique generated names (e.g., `:sp_1`) to prevent collisions.

## 2. Backward Compatibility Strategy

To ensure that the introduction of V3 doesn't break legacy V1 and V2 endpoints:
- **Parallel Routes**: All V3 logic is isolated in `app/routers/api_crud_v3.py`.
- **Hybrid Configuration**: The `obj_type_kv_li` dictionary in `app/ae_obj_types_def.py` has been updated to include both modern keys (e.g., `tbl`, `mdl`) and legacy keys (e.g., `table_name`, `base_name`). This allows V2 endpoints to continue functioning normally while V3 endpoints use the refined structure.
- **Stable Core Imports**: Core modules like `app/lib_general.py` and `app/models/response_models.py` have been refactored to maintain their existing exports (`log`, `logging`, `mk_resp`) while internally adopting more robust module-level logging.

## 3. Learnings and Best Practices

### Logging and Startup Stability
- **Isolate Loggers**: Modules should instantiate their own module-level loggers (`logging.getLogger(__name__)`) rather than importing a global instance. This breaks circular dependencies and improves traceability.
- **Robust Configuration**: Logging configuration (`dictConfig`) should be wrapped in `try...except` to prevent application crashes due to environment issues (e.g., missing log directories in Docker).
- **Explicit Imports**: Always use `import logging.config` before calling `logging.config.dictConfig` to ensure the module is fully initialized.

### FastAPI and Pydantic
- **Dependency Injection**: Use `response: Response` as a type hint for standard injection. Avoid `Depends(Response)` as it is not a valid dependency provider and can cause router initialization failures.
- **Python Parameter Order**: In function signatures, non-default arguments (like `response: Response`) must precede arguments with default values or `Depends()`.
- **Async Concurrency**: Use `asyncio.sleep()` instead of `time.sleep()` in async endpoints. Blocking the event loop in a high-concurrency environment leads to worker timeouts and `502 Bad Gateway` errors.
- **Pydantic Compatibility**: Ensure all new models and utility functions remain compatible with Pydantic v1.10 until a full project-wide migration to v2 is planned. Avoid V2-only features like `computed_field` or `model_validator`.

## 4. Current Migration Status

The following objects have been migrated to the modern V3 configuration and are supported by the V3 API:
- `journal`, `journal_entry`
- `site`, `site_domain`
- `account`, `account_cfg`
- `address`, `contact`
- `order`, `order_line`
- `organization`, `page`
- `data_store`, `activity_log`
- `archive`, `archive_content`
- `hosted_file`, `post`, `post_comment`
- `person`, `user`
- `lu_country`, `lu_country_subdivision`, `lu_time_zone`

## 5. Security and Data Isolation

Implemented in Jan 2026, the V3 architecture enforces strict data isolation and role-aware authorization.

### Role-Aware Authorization (`AuthContext`)
- **Deferred User Lookup**: When a JWT is provided, the API performs a deferred database lookup via `load_user_obj` to populate `administrator`, `manager`, and `super` flags.
- **Role Scoping**: These flags are used to bypass account isolation for support and system maintenance tasks.

### Multi-Tenant Isolation
- **Fail-Closed Strategy (Feb 2026)**: The API enforces a strict "Fail-Closed" mindset. If an authentication context or account ID is missing, the system defaults to a blocking filter (e.g., `account_id IS NULL`). It no longer allows "Fail-Open" behavior where missing context returns all records.
- **IDAA Privacy Baseline**: Based on the International Drunk Doctors Anonymous requirement, the system defaults to **Maximum Isolation**. No object (Events, Files, Posts, Meetings) is public by default. Every object requires an `x-account-id` context, with the sole exception of `site_domain` used for bootstrapping.
- **Forced Account Filtering**: For all non-super users, the API automatically injects an `account_id` filter into every list and search query. This is enforced at the SQL level via `apply_forced_account_filter`.
- **Detailed 403 Feedback**: Security failures now return descriptive error messages (e.g., "API Key expired", "Account context required") to assist developers in debugging header/context issues while maintaining strict isolation.
- **Post-Retrieval Verification**: Single object retrievals (`GET`), updates (`PATCH`), and deletions (`DELETE`) include a secondary ownership check (`check_account_access`). If a user attempts to access an ID belonging to another account, they receive a `403 Forbidden` response.
- **Hierarchical Verification**: Child/Nested endpoints verify the ownership of the parent object before allowing operations on children, preventing cross-tenant "sideways" traversal.
- **Creation Guard**: When creating records (`POST`), the user's `account_id` is automatically forced onto the new record, preventing a user from creating data for another account.

### Bypass and Utility Access
- **X-No-Account-ID**: A specialized header used by development utilities and administrative scripts to bypass standard account isolation. This header grants `super` permissions and should only be used in trusted internal environments.
- **JWT Query Parameter**: Supported for direct file downloads and sharing links where custom headers cannot be provided.