OSIT-AE-API-FastAPI/GEMINI.md

# Gemini Added Memories

## My Role and Operating Principles

I am an interactive CLI agent assisting with software engineering tasks for One Sky IT, LLC, primarily on the Aether API project. My core mandates include:
-   Adhering to project conventions and existing code style.
-   **Never assuming library/framework availability; always verifying project usage.**
-   Implementing changes idiomatically and with minimal, high-value comments.
-   Being proactive, including adding tests for new features/fixes.
-   **Confirming ambiguity or actions beyond clear scope with the user.**
-   Prioritizing user control and project conventions.
-   **Strictly adhering to instructions and utilizing available tools effectively.**
-   **Awaiting explicit user instructions for significant architectural changes or critical decisions.**

## Project Context - Aether API (FastAPI)

-   **Owner/Developer:** Scott Idem (user).
-   **System Name:** Aether (AE).
-   **Purpose:** Events Presentation Management, Events Badge Printing, Leads, Attendee Tracking, Presentation Launcher, Journals, Archives, Posts.
-   **Started:** Mid-2018.
-   **Frontend History:** Python Flask -> Svelte (current: SvelteKit).
-   **Current API Version (FastAPI):** v4.9.0.
-   **V3 Implementation:** Modern parallel CRUD and Search endpoints under `/v3/crud`.

### API Versioning & Strategy

-   `/crud` (v1): Legacy, still used by older frontend parts.
-   `/v2/crud` (v2.5): Modern, preferred, and mostly functional endpoint.
-   `/v3/crud`: The goal of this project phase. A new, parallel implementation with a refined structure and advanced search. **Runs alongside v1 and v2.**

### Technical Learnings
- **Startup Errors & Logging:** The "worker failed to boot" error is often an import-time error or a logging configuration failure.
    - **Root Cause:** If `logging.config.dictConfig` fails (e.g., due to missing `/logs` directories in Docker), the entire application crashes.
    - **Prevention:** Always wrap logging config in `try/except` and use `import logging.config` explicitly.
    - **Circular Dependencies:** These are frequently masked as logging errors because `app.log` is imported very early in most files. Breaking these loops by moving imports inside functions (deferred imports) is a primary fix.
- **Circular Dependencies during Refactoring:** Attempting to move base CRUD logic and engine initialization into separate modules can trigger "Worker failed to boot" if not done carefully.
    - **Issue:** Moving `db` and `engine` to a separate file like `db_connection.py` often creates circular loops with `db_sql.py` or `log.py` because they are imported by almost every other file at the module level.
    - **Resolution:** A "Facade Pattern" was used for `db_sql.py`, where helper functions (Search builders, Redis lookups) are moved to `lib_sql_search.py` and `lib_redis_helpers.py`, but the core connection and CRUD stay in the original file to maintain boot order stability.
- **V3 API Dependencies:** Standardized `Response` injection should use plain type hints (e.g., `response: Response`) to avoid router initialization failures.
- **Pydantic Compatibility:** The current environment uses Pydantic v1.10. Avoid v2 features like `computed_field` or `model_validator` to prevent startup crashes.

### V3 Architectural Progress (Jan 2026)

-   **Modular Object Definitions:** Monolithic `ae_obj_types_def.py` refactored into domain-specific files in `app/object_definitions/` (core, events, journals, orders, cms, lookups, membership, other).
-   **Granular Dependencies:** Monolithic `Common_Route_Params` replaced with specialized dependencies in `app/lib_general_v3.py` (AccountContext, Pagination, StatusFilter, Serialization, Delay).
-   **Advanced Search (POST):** Implemented `POST /v3/crud/{obj}/search` supporting recursive AND/OR grouping and standardized full-text search via the `q` property.
-   **Security Hardening:** Implemented a 5-level recursion depth limit and a field allowlist (`searchable_fields`) for the Search API.
-   **Non-blocking Concurrency:** Standardized on `asyncio.sleep()` for delay simulation to prevent Gunicorn worker hangs.

## Session Learnings & Progress (Jan 2-7, 2026)

### Agent Bridge & Docker Integration (Jan 7, 2026)
- **Agent Bridge Implementation**: Developed `app/routers/agent_bridge.py` to provide administrative insight into the Docker runtime environment.
    - Endpoints include `/status`, `/system/usage`, `/logs`, `/logs/list`, `/processes`, and `/container/metadata`.
    - Implemented `is_admin` helper for consistent security across administrative tools.
    - Added support for listing and tailing various log files, with path sanitization.
- **MCP Docker Explorer**: Created `mcp_docker_explorer.py` as a standalone client to test the Model Context Protocol (MCP) Docker server integration. This script uses `npx` to run `@modelcontextprotocol/server-docker`.
- **Authentication Exceptions**: 
    - Acknowledged that `x_no_account_id` is used intentionally in some contexts, though a more robust solution will be needed later.
    - **Direct Download Links**: Noted that Hosted Files and Event Files often require direct download links that cannot include custom headers, as they are often shared directly with users.

### V3 CRUD Infrastructure & Search
- **Modular Object Definitions**: Refactored `ae_obj_types_def.py` into modular domain files in `app/object_definitions/`.
- **Advanced Search Fixes**: 
    - Resolved account listing and search issues by implementing `get_supported_filters` in `api_crud_v3.py`. This helper automatically adjusts `status_filter` (enabled/hidden) based on model field support (e.g., handles tables missing the `hide` column).
    - Improved standardized full-text search (`q` parameter) in `sql_search_qry_part` with a dry-run check for the `default_qry_str` column. It now falls back to a `LIKE` search across all searchable fields if the column is missing.
    - Added support for the `%` wildcard in `q`, allowing it to bypass filtering and return all records.
- **Data Integrity & Aliasing**: Fixed a critical issue where aliased fields (like `account_id_random`) were returned as `null` by adding `allow_population_by_field_name = True` to Pydantic models (updated `Account_Base`, `Site_Domain_Base`, etc.).
- **Consolidated V3 Router**: Systematically cleaned up `api_crud_v3.py`, removing duplicate endpoint definitions and standardizing logic across all CRUD and Search routes.
- **Robust Error Handling**: Updated V3 routers to return 500 status codes on database failures instead of masking errors with empty result sets.

### Database Schema Insights
- Verified schema for core tables (`account`, `event`, `person`, `user`, `data_store`, `site`, `site_domain`).
- Noted that `site_domain` lacks a `hide` column, and `hosted_file_link` lacks both `hide` and `enable`.
- Most major tables now have a standardized `hide` column (added to `account` on Jan 6, 2026).

## Current To-Do List

1. **Docker MCP Integration (Priority: High)**: Proceed with integrating the Docker MCP server into the Gemini CLI environment for direct container management.
2. **Security - Field Allowlists (Priority: Done)**: Finished populating `searchable_fields` for all object definitions (Core, CMS, Events, Membership, Orders, Other).
3. **Refactoring - Modularize `db_sql.py` (Priority: Done/Low)**: Successfully implemented a facade pattern, moving search builders and Redis helpers to modular files. This reduced `db_sql.py` by nearly 500 lines while preserving stability. Further modularization of core CRUD should only be attempted if stability risks are mitigated.
4. **Specialized Endpoints (Priority: Medium)**: Plan modernization of custom logic (importing, websockets) to match V3 patterns.
5. **Security - Authentication (Priority: High)**: Continue refining and enforcing JWT-based authentication across all V3 endpoints, while respecting the need for header-less direct download links for certain file types.
6. **Account ID Handling (Priority: Low)**: Address the `x_no_account_id` usage with a more permanent and secure architecture.

### Workflow & Collaboration
- **`GEMINI.md` Strategy:** The user is creating `GEMINI.md` files in key project directories. Their understanding is that context flows from the current directory up the tree, with `~/.gemini/GEMINI.md` serving as a global catch-all for general memories.
- **Agents Sync (rsync):** Shared documentation, notes, and architectural updates are pushed to the `agents_sync` directory using `rsync`. This allows real-time coordination between different specialized agents (e.g., FastAPI backend and Svelte frontend agents).
- **Home Server:** The user self-hosts a Proxmox server for services like Nextcloud.