Scott.Idem/OSIT-AE-API-FastAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Saving current notes. About to reorganize the documentation directory.

Browse Source
This commit is contained in:
Scott Idem
2026-01-15 17:48:44 -05:00
parent acd770962b
commit 28d5843d52
2 changed files with 60 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
GEMINI.md
View File

@@ -22,30 +22,22 @@ I am the **primary orchestrator and main helper** for the development of the **U
- **Environment Sync Clarification**: Critical understanding that the `/home/scott/OSIT_dev/` directory is **NOT** a synchronized directory across different machines (e.g., Workstation vs. Laptop). This means virtual environments and installed dependencies are environment-specific and do not propagate via Syncthing. This is crucial for debugging cross-machine environment issues. - **Environment Sync Clarification**: Critical understanding that the `/home/scott/OSIT_dev/` directory is **NOT** a synchronized directory across different machines (e.g., Workstation vs. Laptop). This means virtual environments and installed dependencies are environment-specific and do not propagate via Syncthing. This is crucial for debugging cross-machine environment issues.
- **Circular Import Sensitivity**: Importing `JSONResponse` from `fastapi` in `app/routers/api.py` triggers a circular dependency crash. It is safer to rely on FastAPI's default dict-to-JSON serialization or use `mk_resp` without explicit `JSONResponse` typing in that specific file. - **Circular Import Sensitivity**: Importing `JSONResponse` from `fastapi` in `app/routers/api.py` triggers a circular dependency crash. It is safer to rely on FastAPI's default dict-to-JSON serialization or use `mk_resp` without explicit `JSONResponse` typing in that specific file.
## Session Summary & Progress (Jan 13, 2026) ## Session Summary & Progress (Jan 15, 2026)
This session focused on resolving startup errors in the `ae_api` container and improving database connectivity testing. This session focused on a comprehensive modernization of the API's core infrastructure to resolve boot-time circular dependencies and improve maintainability.
* **Bug Fixes & Stability:** * **FastAPI Entry Point Modularization:**
* **Fixed NameError:** Resolved missing `mk_resp` import in `app/main.py`. * **Registry Pattern:** Created `app/routers/registry.py` to centralize 50+ router registrations, slimming down `main.py`.
* **Fixed AttributeError:** Corrected `response_class=PlainTextResponse` to `JSONResponse` (or default) for dict-returning endpoints. * **Lifespan Context:** Integrated the `lifespan` context manager in `app/main.py` to manage startup/shutdown tasks cleanly.
* **Resolved Circular Dependency:** Identified and fixed a circular import caused by `JSONResponse` usage in `app/routers/api.py`. * **Naming Collision Fix:** Resolved a critical conflict between the `app` instance and the `app.middleware` package by using explicit aliases.
* **Infrastructure & Testing:** * **Database Logic Decoupling:**
* **Safe SQL Test Endpoint:** Relocated and refactored the `sql_test` endpoint from `app/main.py` to `app/routers/api.py` (accessible at `/api/sql_test`). * **Layered Architecture:** Split `app/db_sql.py` into `lib_sql_core` (engine/connection) and `lib_sql_crud` (high-level helpers).
* **Safety Improvement:** Updated the test query to use `SELECT NOW(), VERSION()` instead of querying the `account` table, ensuring a safe connectivity check without exposing data. * **Dynamic Engine Refresh:** Implemented `reconnect_db()` to allow the SQLAlchemy engine to update credentials after the initial boot.
* **V3 CRUD Modularization (Phase 1-3):** * **Robust Configuration & Logging:**
* Successfully refactored `app/routers/api_crud_v3.py` into a modular architecture. * **Logging Sandbox:** Moved `dictConfig` to `app/lib_log_v3.py` and deferred its execution to avoid boot-time traps.
* **Logic Extraction:** Created `app/lib_api_crud_v3.py` for shared security and filtering logic. * **Safe Bootstrap:** Created `app/lib_config_v3.py` with a non-fatal fallback mechanism for DB configuration sync.
* **Nested Routes:** Created `app/routers/api_crud_v3_nested.py` to isolate relational parent-child endpoints. * **Architecture Documentation:**
* **Schema Extraction:** Created `app/lib_schema_v3.py` for database/model introspection. * Created `documentation/V3_CORE_ARCHITECTURE.md` to document the new file structure and bootstrap sequence.
* **DX & Error Transparency:**
* **Error Bubbling:** Updated `app/db_sql.py` to capture SQL exceptions and implemented `format_db_error` to provide human-readable details in API responses.
* **Validation Endpoint:** Added `POST /v3/crud/{obj_type}/validate` for dry-run payload verification.
* **Automatic Sanitization:** Backend now automatically strips virtual lookup fields (`*_id_random`) and view-only fields (via `fields_to_exclude_from_db`) from write payloads.
* **Object Model Alignment:**
* **Address & Contact:** Updated models and JSON mappings to include `hide`, `priority`, `sort`, `group`, and `notes`. Fixed missing imports and unsafe validator access.
* **Journal:** Added `fields_to_exclude_from_db` to filter out view-only fields (e.g., `person_full_name`) during updates.
* **Post:** Added `external_person_id` to searchable fields.
## Current To-Do List ## Current To-Do List
@@ -56,12 +48,14 @@ This session focused on resolving startup errors in the `ae_api` container and i
### 2. Infrastructure & Technical Debt ### 2. Infrastructure & Technical Debt
- [ ] **Psutil Container Update**: Rebuilding the API image with C-build tools (GCC) to support native system monitoring. (Deferred / Paused) - [ ] **Psutil Container Update**: Rebuilding the API image with C-build tools (GCC) to support native system monitoring. (Deferred / Paused)
- [ ] **Fix ComfyUI Container**: Resolve the restart loop in the Ollama/WebUI environment. - [ ] **Fix ComfyUI Container**: Resolve the restart loop in the Ollama/WebUI environment.
- [ ] **Pydantic V2 Migration**: Begin impact analysis for the 400+ validators and .dict() calls.
### 3. Other Agent Tasks ### 3. Other Agent Tasks
- [ ] **Review and process new messages from various agents.** (ID: 134908245) - In Progress (Internal gemini_cli task). - [ ] **Review and process new messages from various agents.** (ID: 134908245) - In Progress (Internal gemini_cli task).
- [x] **[scott_wks] Perform Initial System Health Audit** (ID: 131939030) - Done. - [x] **[scott_wks] Perform Initial System Health Audit** (ID: 131939030) - Done.
- [x] **[API-V3] Implement Error Bubbling and Validation Endpoint** (ID: 163752448) - Done. - [x] **[API-V3] Implement Error Bubbling and Validation Endpoint** (ID: 163752448) - Done.
- [x] **[API] Fix Startup Errors and Refactor SQL Test Endpoint** - Done. - [x] **[API] Modularize main.py and extract Core Registries** - Done Jan 15.
- [x] **[API] Stabilize Logging and Configuration Bootstrap** - Done Jan 15.
### Workflow & Collaboration ### Workflow & Collaboration

43
documentation/V3_CORE_ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,43 @@
# Aether API V3 Core Architecture (Refactor 2026-01-15)
This document describes the modular architecture of the Aether API backend, finalized in January 2026. The goal of this refactor was to resolve boot-time circular dependencies, eliminate naming collisions, and isolate global state from business logic.
## 1. Application Entry Point (`app/main.py`)
The `app/main.py` file is now a lightweight orchestrator. Its primary roles are:
- Defining the FastAPI application instance.
- Managing the **lifespan** context manager for startup/shutdown tasks.
- Calling the centralized registries for routers and middleware.
### The Lifespan Sequence:
1. **Logging Setup**: `setup_logging(settings)` is called first but safely.
2. **Configuration Bootstrap**: `bootstrap_db_config(settings)` queries the `cfg` table.
3. **Database Refresh**: `reconnect_db()` refreshes the SQLAlchemy engine with production credentials.
4. **Validation**: `validate_critical_config(settings)` ensures the environment is production-ready.
## 2. Centralized Registries
To keep `main.py` clean, registrations are moved to dedicated modules:
- **`app/routers/registry.py`**: Exports `setup_routers(app)`. Handles the import and inclusion of 50+ application routers.
- **`app/middleware.py`**: A dedicated home for utility middlewares (e.g., `add_process_time_header`).
## 3. Modular Database Logic
The monolithic `db_sql.py` has been split into three distinct layers:
- **`app/lib_sql_core.py` (Foundational)**: Manages the global SQLAlchemy `engine`, the `db` connection, and the thread-local error state. It is the "Source of Truth" for the connection.
- **`app/lib_sql_crud.py` (Logic)**: Contains high-level CRUD helpers (`sql_insert`, `sql_select`, etc.) that import the connection from the core.
- **`app/db_sql.py` (Facade)**: Maintains backward compatibility by importing and re-exporting all functions. Existing code should still import from `app.db_sql`.
## 4. Modernized Logging (`app/lib_log_v3.py`)
Logging configuration is no longer executed at the module level.
- **`setup_logging(settings)`**: Must be explicitly called during application startup.
- **`logger_reset`**: A decorator used throughout the app to ensure consistent log state.
- **`app/log.py`**: Acts as a facade for backward compatibility.
## 5. Configuration Sync (`app/lib_config_v3.py`)
Handles the multi-layered configuration strategy:
1. **Initial**: Values are loaded from Docker `.env` via `app/config.py`.
2. **Bootstrap**: `bootstrap_db_config` pulls production values from the MariaDB `cfg` table.
3. **Safety**: Implements a non-fatal fallback; if the DB sync fails, the app continues using `.env` settings.