Implements a full proof-of-concept for syncing IDAA's Novi AMS membership
groups to Mailman 3 mailing lists via a cron-triggered reconciliation approach.
Key changes:
- methods: rewrote sync engine around confirmed Novi API shape — group-based
member fetch (/groups/{guid}/members + /customers/{uuid}), respects
Active=false and UnsubscribeFromEmails=true flags
- methods: mirror_novi_group_to_mailman_list() diffs Novi group against
Mailman roster and subscribes/unsubscribes accordingly (full mirror)
- methods: mirror_all_configured_mappings() iterates novi_mailman_sync
config array in IDAA site cfg_json — this is the cron target
- router: replaced old /sync endpoint with POST /sync (all mappings) and
POST /sync/group/{guid} (single mapping); removed webhook endpoint
(sync is cron-based, not event-driven)
- router: added GET/POST/DELETE endpoints for list member inspection
and manual subscribe/unsubscribe
- tests: two new e2e scripts covering connection checks and full member
lifecycle; old webhook integration test archived
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- New router: app/routers/api_v3_actions_event_exhibit.py
- GET /v3/action/event_exhibit/{exhibit_id}/tracking_export
- Full V3 auth (x-aether-api-key + account context)
- Multi-tenant ownership check via check_account_access
- Permission gate: leads_api_access flag OR manager-level access
- Returns CSV or XLSX file attachment (return_file=false for JSON)
- Flattens responses_json custom Q&A columns; strips HTML from exhibitor_notes
- Exports all records regardless of hidden/enabled state
- Registered in registry.py under prefix /v3/action/event_exhibit
- New E2E test: tests/e2e/test_e2e_v3_action_event_exhibit_tracking_export.py
- 7/7 tests passing against dev-api.oneskyit.com
- Docs: GUIDE__AE_API_V3_for_Frontend.md — new Section 7 covering endpoint
usage, columns, leads_api_access dual-purpose (3rd-party API + UI export gate)
- Docs: tests/README.md — added test to table and when-to-run matrix
Root cause: child model root_validators (Vision ID anti-leakage guard) strip
integer IDs before they can be serialized into the INSERT dict, causing MariaDB
to reject the INSERT with 'Field does not have a default value' (1364).
Fix: re-inject resolved_parent_id into data_to_insert after validated_obj.dict()
in post_child_obj(). This is safe — the integer was already verified against the
DB before model validation.
Affected (were all broken since ~2026-01-27):
- journal/{id}/journal_entry/
- event/{id}/event_session/
- event/{id}/event_person/
- event/{id}/event_registration/
- event/{id}/event_presenter/
- event/{id}/event_presentation/
- event/{id}/event_location/
- event/{id}/event_track/
- event/{id}/event_device/
- event/{id}/event_abstract/
- event/{id}/event_badge/ (different symptom: NULL FK)
Tests: add nested create lifecycle regression tests to test_e2e_v3_demo_parity.py
- POST + Vision check + DELETE for journal/journal_entry and event/event_session
- All 9 checks passing (7s)
Docs: update tests/README.md with accurate demo_parity description and
a 'When to Run Tests' matrix to prevent future gaps in coverage.
This commit introduces a new end-to-end test script to validate the recent model refactoring changes.
The test suite performs two primary checks for a list of target objects:
- **ID Vision Compliance:** Verifies that all primary and foreign key fields are returned as string IDs, ensuring adherence to the V3 ID Vision standard.
- **Excluded Fields Stripping:** Attempts a PATCH operation with fields explicitly listed in and verifies that these fields are not updated in the database, confirming the mechanism functions as intended.
This test is essential for ensuring the stability and correctness of the API's interaction with the refactored models.
Address critical data visibility issues for Event Files and enhance frontend documentation.
This commit resolves the persistent problem where top-level hosted file convenience fields
(e.g., , , ) were
returning as in V3 Event File API responses, even when .
Key changes include:
- Refactored Pydantic model:
- Removed redundant definitions from top-level hosted file convenience fields,
allowing direct mapping from SQL view columns.
- Simplified to focus solely on conditionally loading the nested
object, as top-level fields are now populated directly by Pydantic
from the view.
- Added comprehensive comments to clarify data flow, Pydantic's behavior, and the
expected origin of these convenience fields from SQL views.
- Updated :
- Introduced a new section detailing how to retrieve Event File data, including the
use of to get both top-level convenience fields and a nested
object.
- Clarified all ID references as random string IDs.
- Renumbered the troubleshooting section.
- Copied updated guide to .
- Continued ID Vision compliance audit, ensuring consistent handling of random string IDs
across various core and event models (Account, Address, Contact, DataStore, Event Badge Template).
- Consolidated ID Vision E2E tests and updated related documentation.
- Minor updates to and
to support Event File data retrieval with .
1. Added 'account_id' and 'account_id_random' to searchable_fields for event_exhibit_tracking.
2. Updated tests/README.md with descriptions for the latest E2E test scripts (Demo Parity, Event Actions, Zoom).
1. Added fallback mechanism to Event_File_Base to resolve string IDs from integers when views return partial data.\n2. Added 'a2pPIT_W28o' as a permanent regression test target.\n3. Hardened lu_file_purpose_id stripping.
1. Hardened all demo models to set non-string ID fields to None, ensuring full Vision Standard compliance.\n2. Added status_id_random to common field schema.\n3. Verified account_id availability in exhibit tracking.\n4. Added comprehensive E2E parity test suite for demo objects.\n5. Fixed NameError by importing root_validator.
Harden bootstrap_db_config to prioritize .env settings for core infrastructure (DB/SMTP) and only use DB values if placeholders are detected or values have explicitly changed. Added test_e2e_email_send.py for functional SMTP verification.
Updated sql_select, sql_update, and sql_delete to use explicit 'is not None' checks for record_id. This prevents falsy ID values (like 0) from triggering generic table scans or failing to filter, which was causing the config bootstrap to accidentally load record ID 1 when ID 0 was requested.
- Reduced api_crud.py (1843 -> 143 lines) by extracting V1 registry and logic.
- Reduced hosted_file.py (1596 -> 361 lines) by moving storage and media logic to methods.
- Created lib_media.py for specialized video/image processing.
- Created api_crud_methods.py for legacy template handlers.
- Created legacy_v1.py for the legacy object registry.
- Fixed subdirectory_path bug in Hosted File creation.
- Verified full File Lifecycle via consolidated E2E suite.
- Combined 10+ one-off tests into 4 primary functional suites (Search, Auth, Lifecycle, Vision).
- Archived original scripts to tests/archive/.
- Updated README with the new standardized inventory.
- Applied clean output formatting across the new suite.
- Moved legacy/redundant tests to tests/archive/.
- Relocated root-level debug scripts to tests/integration/.
- Updated tests/README.md with final organized inventory.
- Cleaned up root directory from one-off reproduction scripts.
- Added id_random, account_id_random, created_on, and updated_on to searchable whitelists.
- Standardized field coverage for Core and Other (Archive/HostedFile) modules.
- Added Developer Handshake comments to prevent future whitelist/model desync.
- Verified via new E2E registry test suite.
- Introduced WS_Message_V3 standardized Pydantic model and WS_Manager_V3.
- Implemented /v3/ws/ endpoint with granular Redis routing to solve "noisy neighbor" scaling issues.
- Added presence tracking using Redis Sets for group coordination.
- Comprehensive test suite added (unit and integration) covering models, manager, and routing logic.
- Documentation: Created V3 Frontend WebSocket Guide and Project design spec.
- Updated main Frontend API guide and tests README with new standards.
- Migrated event_device and event_session models to the V3 Vision ID pattern (string-based public IDs).
- Added root_validator for automatic id_random mapping and integer stripping.
- Implemented fields_to_exclude_from_db to protect database updates from convenience/view fields.
- Fixed description_json type in Journal_Base for correct JSON parsing.
- Added E2E verification tests for event_device and event_session V3 endpoints.
- Update GET /v3/data_store/code/{code} to support 'limit' query parameter.
- Refactor return logic: returns single object if limit=1, otherwise returns a list.
- Clean up formatting in GUIDE__V3_FRONTEND_API.md and sync to agents_sync.
- Finalize unified E2E test script: tests/e2e/test_e2e_v3_data_store_lookup.py.
- Added GET /v3/data_store/code/{code} with hierarchical context-aware fallback.
- Implemented ID Vision standard in Data_Store_Base (string IDs, internal int exclusion).
- Enhanced Data_Store_Base robustness to handle stringified 'NULL' values from the database.
- Fixed legacy router bugs by removing undefined parameters (inc_event_cfg, inc_event_location).
- Corrected type hints and resolved UnboundLocalError in data_store methods.
- Updated Frontend Integration Guide with Section 8: Data Store V3.
- Added unified E2E test script: tests/e2e/test_e2e_v3_data_store_lookup.py.
- Overhauled README.md to serve as a unified system index and WIP tracker.
- Standardized documentation filenames (ARCH__, GUIDE__, PLAN__) for better discoverability.
- Archived completed project plans and scopes.
- Fixed regressions in unit tests (errors, models, email) caused by V3 architectural updates.
- Ensured unit tests remain non-destructive and environment-independent via mocking.
- Refactor SQL CRUD to use engine.connect() context managers for thread safety
- Optimize connection pooling in lib_sql_core
- Clean up app/routers/api.py to fix duplicate definitions and OpenAPI KeyError
- Add 'default_qry_str' to searchable_fields for Event, Session, Presentation, Presenter, Badge, and Journal
- Add 'event_location_name' to searchable_fields for Event Session
- Verified 20/20 E2E success via repro_intermittent_errors.py
- Add POST /api/authenticate_passcode to verify site access codes
- Refactor sign_jwt to support arbitrary role flags (super, admin, etc.)
- Update dependencies_v3 to extract role flags from JWT payloads
- Add E2E test for passcode auth verification
- Patched request_jwt to strip privileged IDs when signing with public keys
- Updated AccountContext and V3 dependencies to preserve JWT payloads for guests
- Whitelisted Archive, Post, Event, and other core objects for public read access
- Added 'default_qry_str' to Event searchable fields
- Added test_e2e_jwt_guest_auth.py for security verification
- Created LOCAL_DEVELOPMENT_GUIDE.md and DEPLOYMENT_GUIDE_MANUAL.md from legacy txt files.
- Migrated country/time_zone data and requirements.txt to documentation/reference_data/.
- Removed redundant admin/documentation/ and admin/data_files/ directories.
- Enhanced app/lib_schema_v3.py to explicitly capture 'required' fields from DB 'NOT NULL' constraint.
- Added verification tests for schema logic and standalone DB connectivity.
