# Aether (AE) SvelteKit Application

This project is a Svelte and SvelteKit based application, part of the Aether (AE) system. It uses Tailwind CSS and Skeleton for styling and some elements. This is the frontend UI/UX. The backend API uses Python FastAPI.

Core Aether modules

- accounts - client account, not user account
- hosted_files
- people
- users
- sites and site_domains

Additional Aether modules

- events
    - presentation management - import the program data (events, session, presentations, presenters, event files, locations/rooms, devices)
    - launcher - Technically this is used with presentation management. It is part of the native app that uses Electron. One of the libraries is for functions that only work when the site is opened in an Electron app. For example the regular browser can not move files around on the local computer or run local commands.
    - badge printing
    - lead retrieval - attendee tracking; QR codes
- journals - journal, documentation, notes, diary, blog, etc
- idaa - One of my clients

## Documentation

- TODO.md
- Svelte - Introducing runes - https://svelte.dev/blog/runes
- Svelte - Breaking changes in runes mode - https://svelte.dev/docs/svelte/v5-migration-guide#Breaking-changes-in-runes-mode
- Dexie.js - Getting Started - https://dexie.org/docs/Tutorial/Getting-started
- Dexie.js - API Quick Reference - https://dexie.org/docs/API-Reference#quick-reference

## Ignored Directories

The following directories are ignored for various operations (e.g., search, file listing) to focus on relevant source code:

- `build`
- `node_modules`
- `tests`

## Development Guidelines

### Svelte v5 and SvelteKit

This project uses Svelte v5 with runes enabled. This introduces significant differences from Svelte v4. It is critical to adhere to v5 conventions to avoid bugs.

- **Reactivity:** State is managed with `$state` and `$derived`. Props can be made two-way bindable with `$bindable`. Avoid direct mutation of props.
- **Event Handling (Updated 2025-11-20):**
    - **DOM Events:** Use the lowercase `onevent` attribute (e.g., `onclick`, `oninput`). Event modifiers like `|preventDefault` may not work as expected; handle prevention logic inside the function (e.g., `event.preventDefault()`).
    - **Component Events:** Continue to use the `on:eventname` directive for events dispatched from child components.
- **Stores and `liveQuery`:**
    - To access the value of a store in Svelte v5, you must use the `$store_name` syntax (e.g., `$ae_api`).
    - Dexie `liveQuery` returns an observable. To use it in a component, you must subscribe to it within `onMount` to avoid SSR errors. The value from the subscription should then be assigned to a `$state` variable.
- **Migration Guide:** For a comprehensive overview of the changes, refer to the official [Svelte 5 Migration Guide](https://svelte.dev/docs/svelte/v5-migration-guide).

### ID Convention: `id` vs. `id_random`

- **Always use `id_random`:** The API returns both a numeric `id` and a string-based `[obj_type]_id_random`. For all frontend operations (routing, data fetching, local storage), you **must** use the `_id_random` string.
- **Local Storage:** When saving an object to the local IndexedDB (Dexie), the `id_random` value should be aliased to both `id` (as the primary key) and `[obj_type]_id`. This ensures consistency with Dexie's expectations and the rest of the application's data access patterns.

---

## Refactoring Notes

### UI Library Updates (2025-12-08)

**Context:** All project packages, including `@skeletonlabs/skeleton` (v4.7.4) and `flowbite-svelte` (v4.0.1), have been successfully updated to their current versions. The `npm run dev` and `build` commands are now working as expected.

**Key Insight:** The previous issues and blocking build errors related to UI library compatibility were largely resolved by addressing specific conflicts, particularly those associated with the `ae_comp__badge_template_form.svelte` file. This indicates that direct integration and careful handling of component-specific dependencies were crucial to achieving library compatibility.

**Current State:**
*   The project successfully builds and runs with the latest versions of SkeletonLabs and Flowbite-Svelte.
*   The codebase can now move forward with development, leveraging the updated UI libraries.

### Data Fetching & Processing Pattern (2025-11-20)

A standard pattern for fetching, processing, and caching data has been established to ensure consistency and separation of concerns.

1.  **API Function (`load_*`, `search_*`, etc.):** This function is responsible for interacting with the API. It takes parameters needed for the API call (e.g., `event_id`, search strings).
2.  **Data Processor (`process_ae_obj__*_props`):** The API function's results are immediately passed to this dedicated processor function for the specific object type.
    - The API function MUST pass any necessary contextual data (like a parent `event_id`) to the processor.
    - The processor is responsible for all data shaping and enrichment before the data is cached.
3.  **Handling API Inconsistencies:** The processor is the designated location for handling any inconsistencies in the data returned by the API.
    - **Example (`event_badge`):** The `search__event_badge` API endpoint does not include the `event_id` in its results. The `process_ae_obj__event_badge_props` function now accepts the `event_id` as a parameter and injects it into each badge object before it's saved. This centralizes the fix and keeps the API-calling function clean.
4.  **Database Caching (`db_save_ae_obj_li__ae_obj`):** After processing, the clean and consistent data is passed to the generic `db_save` function to be cached in IndexedDB.

This pattern isolates API logic from data shaping logic, making the code more modular, predictable, and easier to debug.

### Session Learnings (2025-12-12)

**Context:** Prepared for the upcoming IDAA meeting by working on the new Jitsi video conferencing page and its corresponding Novi iframe template.

**Key Learnings & Actions:**
*   **Suppressing Svelte Warnings:** Added an `onwarn` handler to `svelte.config.js` to suppress `state_referenced_locally`, `non_reactive_update`, and `css_unused_selector` warnings for the time being. This helps to focus on critical errors during development.
*   **Configuration Management:**
    - Initial approach of using `.env` files for Novi/Jitsi configuration was incorrect for this project.
    - **Correct Approach:** Configuration values (`jitsi_token_endpoint`, `novi_idaa_api_key`, `novi_idaa_group_guid_li`) are stored in the database and accessed via the `$ae_loc.site_cfg_json` Svelte store.
    - The `.../video_conferences/+page.svelte` component was refactored to use the `$ae_loc` store instead of `import.meta.env`.
    - The logic for fetching moderators was updated to iterate through a list of group GUIDs (`novi_idaa_group_guid_li`) from the store.
*   **Static Iframe Bridges:**
    - The `static/idaa_novi_iframe_jitsi_meeting.html` file acts as a bridge, taking Novi-templated user variables (`<%=Novi.User.*%>`) and passing them as URL parameters to the Svelte application (`/idaa/video_conferences`).
    - **User Feedback:** An attempt to simplify this bridge by removing a client-side `fetch` call for user details was reverted by the user. This indicates that the original, more complex logic (which constructs `full_name` from `FirstName` and `LastName`) is necessary. This is an important lesson in not over-simplifying third-party integration code without full context.
    - The bridge was updated to include an interactive UI for setting Jitsi parameters (e.g., sound settings, moderator settings) and dynamically rebuilding the iframe `src`.

---
## Jitsi "God Mode" Live Stats Architecture (2025-12-15)

### Goal
For the IDAA client, create an administrative dashboard ("god mode") to monitor live Jitsi meetings—including meeting duration, participant lists, and moderator roles—without needing to join the meetings directly. This is primarily for ensuring meetings are running correctly and for data collection.

### Architectural Decision
A purely client-side approach using the Jitsi External API is not feasible, as it requires being a participant in the meeting. A server-side solution is necessary. After considering a log-parsing approach, the following hybrid architecture was chosen as the most robust and scalable solution.

1.  **Server-Side XMPP Bot (Stats Collector):**
    *   A new, lightweight backend service will be created (likely in Python or Node.js).
    *   This service will act as a bot, using special credentials to connect to the Jitsi instance's main XMPP server (Prosody).
    *   The bot will automatically and invisibly join the Multi-User Chat (MUC) for each active conference. It will not process any audio or video, only presence and metadata.
    *   By listening to real-time XMPP events, it will track participant joins, leaves, and role changes.

2.  **Database Integration:**
    *   As the XMPP bot receives events, it will write structured data directly into the Aether system's database (e.g., a `jitsi_events` table).
    *   This provides two key benefits: a source for the live dashboard and a clean, persistent historical record for future analysis, avoiding the fragility of parsing raw text logs.

3.  **API Endpoint:**
    *   A new, secure endpoint (e.g., `/api/jitsi/live-stats`) will be added to the FastAPI backend.
    *   This endpoint will query the database to get the current state of all active meetings.

4.  **Frontend Admin Dashboard:**
    *   A new, access-controlled page (e.g., `/admin/jitsi-stats`) will be built in the Svelte application.
    *   This page will periodically fetch data from the new API endpoint to display a live overview of all ongoing meetings.

### Rationale
This approach was chosen over log-parsing because the XMPP protocol is a stable, official API for Jitsi, making the solution less likely to break on future Jitsi updates. It provides true real-time data for the live dashboard while simultaneously creating a valuable, structured dataset for historical reporting.

---
## Jitsi Stats Reporting Debugging (End of Day 2025-12-15)

### Objective
As a pragmatic first step towards the "God Mode" view, implement a "moderator-pushed" stats reporting system. A moderator's browser will periodically send the meeting stats to an existing Aether activity log endpoint.

### Progress
1.  **Client-Side Logic:** The Svelte component now collects live meeting stats (duration, participants, roles) via the Jitsi External API.
2.  **Reporting Function:** A `report_meeting_stats` function has been created to bundle this data and call the `create_ae_obj__activity_log` API function.
3.  **Trigger:** A 30-second timer (`setInterval`) is implemented to call the reporting function automatically for users where `is_moderator` is true.

### Current Problem
The feature is not working. The browser's console logs show that the `report_meeting_stats` function is being called and the data payload is being correctly prepared. However, the API call fails silently:
*   No success or error messages are logged in the console from the `try...catch` block around the API call.
*   The API call never reaches the FastAPI server (no entry in the server's access logs).

### Investigation & Last Actions
*   My initial hypothesis was a hanging network request due to a missing timeout.
*   **Action Taken:** I modified the underlying `post_object` function in `src/lib/ae_api/api_post_object.ts` to add a 20-second timeout to the `fetch` call.
*   **Result:** This had no effect. The request still disappears without triggering the success, error, or timeout conditions. This suggests the issue is preventing the `fetch` call from executing correctly at a fundamental level.

### Next Step
The crucial next step is to use the **Network** tab in the browser's developer tools. We need to observe what happens, if anything, at the exact moment the "Stats payload being sent" message appears in the console. This will tell us if a request is being blocked by the browser itself (e.g., CORS pre-flight failure) or if it's failing to even be dispatched.

---
## Jitsi Stats Reporting Debugging (Final Resolution - 2025-12-16)

**Context:** Continued debugging the silent failure of the `create_ae_obj__activity_log` API call within the Jitsi video conferencing page.

**Root Cause Identification:**
1.  **Hidden Console Errors:** The primary reason for the prolonged debugging was that Firefox Developer Tools had console errors hidden. This meant that crucial `TypeError` messages, which would have immediately pointed to the issue, were not visible.
2.  **Incorrect Parameter Passing:** The `api_cfg` parameter, expected by `create_ae_obj__activity_log`, was not correctly resolved in the calling context. Specifically, the `$ae_api` Svelte store, which provides the necessary `api_cfg` object, was not being imported into the component. Accessing properties on an `undefined` `$ae_api` (or a malformed `api_cfg` extracted from it) led to an uncaught `TypeError` that was being suppressed by the console settings.

**Solution:**
*   Ensured console errors were visible in Firefox Developer Tools.
*   Added `import { ae_api } from '$lib/stores/ae_stores';` to the relevant Svelte component (`src/routes/idaa/(idaa)/video_conferences/+page.svelte`).
*   Ensured `api_cfg` was correctly passed as `$ae_api` to the `create_ae_obj__activity_log` function.

**Key Learnings:**
*   **Always Verify Console Settings:** Before deep-diving into complex debugging, ensure that browser developer tools are configured to show all error types. A simple UI misclick can hide critical information.
*   **Parameter Resolution is Key:** Even with `async/await` and `try...catch` blocks, errors related to `undefined` or improperly resolved parameters passed to functions can cause silent failures if the error occurs before the `try` block can effectively catch it, or if console reporting is suppressed.
*   **Initial Error Messages are Paramount:** The first error message is often the most accurate indicator of the root cause, even if subsequent attempts to fix it seem to fail. Had the `TypeError` been visible, the debugging process would have been significantly shorter.
*   **The "Silent Failure" often has a hidden error message:** A function that appears to "disappear" or fail silently often has an underlying error being suppressed or unhandled in a way that prevents it from being logged.

**Outcome:**
The activity logging functionality is now working as expected. While the original hypothesis of a circular dependency was a plausible architectural issue, the immediate problem was a more fundamental runtime error exacerbated by hidden console output. The temporary isolation of the activity log function (`src/lib/ae_idaa/idaa_activity_log.ts`) is no longer needed.

---
## Aether Ops Extension (Gemini CLI)

The project utilizes a custom Gemini CLI extension (`aether-ops`) to orchestrate the stack via the Model Context Protocol (MCP).

### Key Tools
- **`verify_api_integrity`**: Executes the `test_ae_api_v3.py` suite. Verifies that the FastAPI backend is reachable and that the "Bootstrap Paradox" unauthenticated bypass is functioning correctly.
- **`schema_sync`**: Synchronizes Pydantic models with TypeScript interfaces.
- **`sql_query`**: Direct read-only access to the MariaDB for state verification.
- **`docker_restart` / `docker_logs`**: Controls the backend and database containers.
- **`send_message` / `add_task`**: Coordination tools for multi-agent workflows.

### Workflow Integration
Always run `verify_api_integrity` after making changes to:
1.  Core API helpers in `src/lib/ae_api/`.
2.  Backend CRUD routes in the FastAPI project.
3.  Site domain or account lookup logic.

### Vision
The project is moving towards a single, unified agent (UE-AE-01) with "Total System Awareness" across the entire Aether stack: MariaDB (Remote), FastAPI (Docker), SvelteKit (Local), Nginx (Proxy), and Syncthing (Storage).

### Frontend Feedback & Pain Points
The `frontend_svelte` agent provided critical feedback to `backend_fastapi` for the UE-AE-01 proposal:
- **Automated Schema Sync:** UE-AE-01 can automatically generate TypeScript interfaces and `.editable_fields.ts` whitelists from Pydantic models and MariaDB schemas.
- **Cross-Stack Debugging:** Unified access to Nginx, FastAPI, and Svelte logs will allow for instant root-cause analysis of 500 errors.
- **Bootstrap Paradox Resolution:** Identified a circular dependency where `site_domain` lookup required a JWT, but the JWT required the `account_id` from the lookup. Resolved by allowing unauthenticated V3 search for `site_domain`.
- **Environment Orchestration:** The agent should automatically manage Docker restarts and environment syncs after backend modifications.

### Status
- **V3 Search Verification:** Successfully implemented `lookup_site_domain_v3` in `ae_core__site.ts` and added a test button in `/testing`. 
- **Migration Plan:** Root layout `+layout.ts` is slated for migration from `lookup_site_domain` (legacy) to `lookup_site_domain_v3` to fully leverage the Bootstrap Paradox fix.

---
## Session Learnings (2025-12-16)

### Client-Side Reporting Pattern

**Context:** Implemented a new reports page for Jitsi meetings (`/idaa/jitsi_reports`) without requiring a new, dedicated API endpoint.

**Strategy:**
1.  **New Reports Module:** Created a dedicated file for client-side reporting logic at `src/lib/ae_reports/reports_functions.ts`.
2.  **Fetch Flat Data:** The `load_jitsi_report` function uses the generic `api.get_ae_obj_li_for_obj_id_crud_v2` function to fetch a flat list of all relevant records (e.g., all `activity_log` entries related to Jitsi meetings).
3.  **Process on Client:** The flat list is then processed in the browser. A JavaScript `Map` is used to group the individual log entries by `external_client_id` (the unique meeting ID), transforming the flat data into a hierarchical structure suitable for a report (one parent object per meeting, with an array of discrete events inside).
4.  **Render:** The SvelteKit `load` function in `+page.ts` calls the processing function, and the `+page.svelte` component receives the structured data and renders it.

**Key Insight:** This pattern is effective for creating new views and reports quickly when the underlying data is accessible via existing generic API functions, avoiding the need for immediate backend development. However, it can be less performant for very large datasets, as the filtering and aggregation happen on the client.

### API Helper Completeness & SvelteKit URL Handling

**Context:** Encountered two subsequent errors while building the reports page.

**Learnings:**
1.  **`Unknown object type` Error:** The generic `get_ae_obj_li_for_obj_id_crud_v2` function failed with this error for `activity_log`.
    *   **Root Cause:** The function relies on an internal lookup map (`objTypeToEndpointMap`) to resolve object types to API endpoints. This map was missing an entry for `activity_log`.
    *   **Solution:** The fix was to edit `src/lib/ae_api/api_get__crud_obj_li_v2.ts` and add `'activity_log': '/crud/activity_log/list'` to the map. This is a reminder that generic helpers need to be explicitly updated to support new object types.
2.  **`data.url is undefined` TypeError:** This error occurred inside an asynchronous Jitsi event handler when trying to access `data.url.origin`.
    *   **Root Cause:** The `data` prop, passed from SvelteKit's `load` function, is not reliably scoped or available inside asynchronous callbacks fired by external, third-party libraries.
    *   **Solution:** The robust solution is to use SvelteKit's dedicated `$page` store. By importing `import { page } from '$app/stores';` and using `$page.url`, we can reliably access the current URL information from anywhere in the component, including async callbacks.

### Session Learnings (2026-01-02)

**Context:** Migrated the Journals and Events - Badges modules to the new Aether API V3 CRUD endpoints and resolved several Svelte 5 / Dexie integration issues.

**Key Accomplishments:**
- **Aether API V3 Frontend Implementation:** Created robust TypeScript wrappers for V3: `get_ae_obj_v3`, `get_ae_obj_li_v3`, `get_nested_obj_li_v3`, and `search_ae_obj_v3`. These support the new nested URL structure, hybrid filtering (query params + POST body), and global keyword search (`q` property).
- **Journals Module Migration:** Partially migrated `ae_journals` to V3 (List and Search).
    - **Bug Fix (Missing Parent ID):** Discovered that V3 nested routes often omit the parent ID in response objects. Implemented automatic parent ID injection in the data processor (`process_ae_obj__journal_entry_props`) to maintain local Dexie relationships.
    - **Bug Fix (New Entry Button):** Fixed a runtime crash on the "New Entry" button in `+layout.svelte` by removing illegal `$effect` calls from the `onclick` handler and fixing incorrect `$derived` variable access.
- **Events - Badges Module Migration:** Partially migrated `ae_events__event_badge` to V3 (List and Search).
    - **Search Optimization:** Refined the `search_query` structure to utilize the backend's new `q` property for global text search and standard `and` filters for specific fields like `badge_type_code` and `print_count`.
    - **Filter Fixes:** Resolved issues with "print status" and "affiliations" filters by reverting to the standard `like` operator (as `ilike` and `contains` were unsupported by the backend) and ensuring `print_count` is included in the persistent schema.
- **Documentation & Planning:** Updated `TODO.md` with a comprehensive V3 integration roadmap and documented actionable feedback for the FastAPI backend.

**Key Learnings:**
- **Svelte 5 & Dexie LiveQuery:** Remember that while Svelte 5 runes (`$state`, `$derived`) don't use the `$` prefix, **Dexie `liveQuery` results are observables** and still require the `$` prefix (e.g., `$lq__obj`) to access their reactive value in the template and script.
- **API V3 Implicit Context:** When using nested API routes (e.g., `/v3/crud/parent/{id}/child/`), the child objects returned may not contain the `parent_id`. The frontend must proactively inject this ID during processing if it's required for local database indexing or filtering.
- **Search Logic Construction:** When building complex V3 `search_query` objects, avoid including empty `and` or `or` arrays, as some backend parsers may strictly validate their presence or content. Only attach these properties if they contain at least one filter.
- **Backend Operator Support:** Always verify supported operators (`like`, `eq`, `gt`, etc.) in the backend FastAPI implementation. Using unsupported operators like `ilike` or `contains` will cause immediate backend `ValueError` crashes.

### Session Learnings (2026-01-07) - Session 2

**Context:** Established formal agent identity, verified synchronization crons, and tested the expanded Agent Sync toolset on the workstation.

**Key Accomplishments:**
- **Agent Identity & Coordination:** Formally established identity as `frontend_svelte`. Updated workstation crontab to monitor the `frontend_svelte` inbox. Sent coordination and status messages to `backend_fastapi` and `gemini_cli`.
- **Tool Verification:** 
    - Verified `docker_control.py` for backend container monitoring.
    - Verified `sql_inspector.py` for direct database verification during UI development.
- **Enhanced Schema Sync:** Identified a database dependency issue in the original `schema_sync.py` when run from the host. Created a robust alternative, `schema_sync_v2.py`, which generates TypeScript interfaces directly from the offline `registry_v3.json`. Successfully verified this by generating the `ActivityLog` interface.
- **Workflow Optimization:** Provided feedback for the main `Agents Sync` README to include canonical IDs, execution context tags (Host vs. Docker), and a "First 5 Minutes" onboarding checklist for new agents.

**Key Learnings:**
- **Execution Context Matters:** Scripts that import backend API modules (like the original `schema_sync.py`) often trigger database connection attempts, which fail without environment-specific credentials. Registry-based tools are safer and more robust for frontend workstation use.
- **Inbox Naming Consistency:** Confirmed that the canonical inbox directory name is `frontend_svelte`, which must be strictly used for crons and messaging to avoid missed communications.
- **Registry as "Source of Truth":** The `registry_v3.json` file is a powerful asset for the frontend, enabling automated model generation without needing the backend or database to be actively reachable.

**Next Steps:**
- **V3 Interface Verification:** Verify and integrate the 59 TypeScript interfaces exported to `agents_sync/technical/aether_interfaces.ts`.
- **Journals Audit:** Begin the "Frontier Journals" codebase audit.
- **UE-AE-01 Transition:** Continue monitoring the inbox for specific migration tasks from the manager.

### Session Learnings (2026-01-08)

**Context:** Refactored the core Aether API helper suite (`get`, `post`, `patch`, `delete`) to ensure architectural consistency, robust error handling, and support for unauthenticated lookups (the "Bootstrap Paradox").

**Key Accomplishments:**
- **API Helper Alignment:** Standardized all primary helpers to support custom `fetch` injection (essential for SvelteKit SSR), kebab-case header standardization, and automated JWT-to-Authorization header injection.
- **Bootstrap Paradox Resolution:** Implemented a robust bypass in the header merging logic. By passing `x-no-account-id: null` to the helpers, the default `x-account-id` is stripped, and a non-null dummy value is sent, allowing unauthenticated V3 searches for site domains.
- **Non-Mutating Config:** Eliminated direct mutations of the shared `api_cfg` object within the helpers, preventing difficult-to-track side effects across the application.
- **System Testing Dashboard:** Upgraded the `/testing` page with a dedicated suite for core helper verification and V3 search validation.
- **CLI Verification Suite:** Created `agents_sync/scripts/test_ae_api_v3.py` for rapid, headless verification of the API stack from the command line.

**Key Learnings:**
- **Implicit Header Requirements:** Some backend environments (like the Aether V3 API) strictly require the presence of a "bypass" header even when credentials are omitted. A null value in the client-side header map is often stripped by `fetch`, so a dummy string value (e.g., "Nothing to See Here") is necessary for the header to be successfully transmitted.
- **Environment Parity in Testing:** Verifying API helpers in both the browser (via the `/testing` page) and the CLI (via Python scripts) is critical for identifying environment-specific issues like CORS, header formatting, and timeout behavior.
- **Barrel File Strategy:** As the API stack grows, a pure barrel file strategy for `api.ts` (exporting from dedicated modules) is superior to a single "God Object" for maintainability and code splitting.

