Commit Graph

95 Commits

Author SHA1 Message Date
Scott Idem
b590bc09a0 fix: require root_url on email_auth_key_url; correct frontend guide for user auth endpoints
- Make root_url a required query param on GET /v3/action/user/{id}/email_auth_key_url
  (previously Optional[str]=None, which produced a malformed link in the emailed URL)
- Update GUIDE__AE_API_V3_for_Frontend.md: document root_url as required, add magic link
  URL format, note valid_email=True side effect, add 404 error, expand 403 conditions
  for authenticate, add 400 for verify_password when no password is set
- Add test_e2e_v3_user_action_routes.py and test_e2e_v3_user_auth_routes.py to tests/README.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-25 12:34:49 -04:00
Scott Idem
3d89e95c24 fix(P2): add OperationalError retry to sql_insert, sql_select, sql_insert_or_update
All three were missing the transient-connection retry that sql_update and
run_sql_select already had. On OperationalError (stale/dropped connection),
each now retries once with a fresh engine.connect() without disposing the pool.

IntegrityError (duplicate key, FK violation, NOT NULL) continues to return
None without retrying — the same data would fail again and None signals a
data conflict to callers, distinct from False (error) or an int (success).

sql_insert_or_update retry is safe because ON DUPLICATE KEY UPDATE is idempotent.
sql_insert retry is safe because OperationalError means MariaDB rolled back.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 19:41:26 -04:00
Scott Idem
3db5f7c749 fix(P3): guard startup db connection with try/except in lib_sql_core
Wraps the deprecated global `db = engine.connect()` in a try/except so
a Docker startup race (MariaDB not yet ready) no longer crashes the
Gunicorn worker before it can serve any requests.

Sets db=None on failure; reconnect_db() on the lifespan bootstrap path
re-establishes it once credentials are confirmed.

TODO (P3 full): migrate lib_schema_v3.py:39 and lib_api_crud_v3.py:166
off the global db to engine.connect() context managers, then remove it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 19:28:28 -04:00
Scott Idem
ace00929f2 feat: expose DB pool_size and max_overflow as env vars (P4)
Added AE_DB_POOL_SIZE and AE_DB_POOL_MAX_OVERFLOW to config.py with
defaults matching prior hardcoded values (10/20). Wired into settings.DB
property so create_ae_engine() reads them without fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 18:08:01 -04:00
Scott Idem
c7444a8a89 fix: remove pool-nuking reconnect_db() from OperationalError retry paths
On OperationalError, sql_update and run_sql_select were calling
sql_connect() → reconnect_db() which disposes the entire connection
pool mid-flight, killing other in-flight connections under concurrency.

Removed the sql_connect() calls; the existing retry blocks already open
a fresh engine.connect() context manager, and pool_pre_ping=True handles
stale connection detection. Also drops the now-unused sql_connect import.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 17:24:47 -04:00
Scott Idem
8f1fe5d4df Fixing this:
#3 (zombie import) is genuinely a 2-line fix — remove the import from api.py:10 and move db_connection.py to trash. Zero functional change since db is only in a commented-out line.
2026-04-16 20:09:12 -04:00
Scott Idem
516865b7d8 Updated docs 2026-04-10 11:56:44 -04:00
Scott Idem
f9f588ddf2 docs: add temporary Axonius Zoom CSV Upload section (Apr 2026) 2026-04-08 12:38:36 -04:00
Scott Idem
4629e1ec63 feat(site_domain): restore access_key enforcement for FQDN lookups
- api_crud_v3: strip falsy access_key values; restrict keyless queries
  to public domains (both site_access_key and site_domain_access_key
  must be NULL/empty); 75-line recursive block replaced with ~16 lines
- lib_sql_search: expand virtual 'access_key' field into priority SQL —
  site_access_key first, site_domain_access_key as fallback
- cms.py: add site_domain_access_key to site_domain searchable_fields
- docs: update frontend guide with access key behavior and examples
- e2e test: expand to cover all valid/invalid access key scenarios (15/15)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 14:46:33 -04:00
Scott Idem
687472f4e3 feat(user): V3 action endpoints + auth bug fixes (19/19 + 22/22 tests)
New router: /v3/action/user/ (api_v3_actions_user.py)
  - POST /authenticate  — credentials in body (not query params; security fix)
  - POST /verify_password
  - POST /{user_id}/change_password  — optional current-password verification
  - GET  /{user_id}/new_auth_key
  - GET  /{user_id}/email_auth_key_url
  Registered in registry.py under /v3/action/user with V3 AccountContext auth.

Bug fixes (from audit in previous session):
  - user.py: fix broken @router.get decorator (authenticate was unreachable)
  - user.py + user_methods.py: fix AttributeError id_random → id (Vision ID)
  - user_models.py: add fields_to_exclude_from_db to User_New_Base; narrow
    collision prevention to self-reference IDs only
  - user_models.py: pre-inject hashed password in root_validator(pre=True) so
    exclude_unset=True in CRUD POST handler includes it (was writing NULL)
  - api_crud_v3.py: move sanitize_payload + account_id injection to after
    model validation (fixes FK integer collision with Vision ID constraints)

Docs: GUIDE__AE_API_V3_for_Frontend.md — new Section 7 with full migration
  table (legacy → V3), request/response docs for all 5 action endpoints,
  and V3 CRUD search equivalents for the 3 lookup routes.

Tests: tests/e2e/test_e2e_v3_user_action_routes.py — 19 tests, 19/19 pass.
  Legacy tests/e2e/test_e2e_v3_user_auth_routes.py — 22/22 still pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 21:54:09 -04:00
Scott Idem
91434968f7 docs+site_domain: Add guidance for restoring access_key validation in site_domain lookup; stage recent user/auth changes and frontend guide updates 2026-03-25 19:33:53 -04:00
Scott Idem
cffde249d3 fix(models): migrate Archive_Content_Base to Vision ID pattern
- Replace integer `id` (alias archive_content_id) with Vision string fields:
  `id: Optional[str]` and `archive_content_id: Optional[str]` — both always
  hold the random string ID, never the DB integer.
- Add `root_validator(pre=True)` (map_v3_ids) that maps id_random /
  archive_content_id_random → id and archive_content_id, with collision
  prevention to reject any integer that arrives in these fields.
- Remove old `archive_content_id_lookup` integer validator (superseded by
  sanitize_payload + root_validator).
- Keep `id_random` (alias archive_content_id_random) in responses for
  backward compatibility; add id, archive_content_id, id_random to
  fields_to_exclude_from_db so they never appear in INSERT/UPDATE payloads.

Generic CRUD layer safety net (post_obj + post_child_obj):
- After building resp_data on create, swap any integer {obj_type}_id with
  the corresponding {obj_type}_id_random value — catches models not yet
  migrated to Vision IDs.
- Fix return_obj=False fallback to return obj_id as the random string.

Docs: add Section 3D to GUIDE__AE_API_V3_for_Frontend.md documenting the
Vision ID convention — {obj_type}_id is always the random string; the
_id_random suffix is a legacy artifact that frontend code should phase out.

Fixes: POST /v3/crud/archive/{id}/archive_content/ returning integer ID,
breaking the subsequent PATCH flow (422 min_length validation failure).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 17:40:27 -04:00
Scott Idem
b9742cfcd8 feat(routers): migrate hosted_file hash lookup to V3 actions
Ported the legacy '/hosted_file/hash/{hash}' endpoint to the V3 actions router.
The new endpoint '/v3/action/hosted_file/hash/{hosted_file_hash}' supports:
- ID Vision: returns random string IDs instead of internal integers
- Local Check: verifies physical file existence on disk (check_for_local=True)
- Deduplication: enables frontend to check for existing files before upload

Also added PROJECT document for AE Hosted Files migration tracking.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-25 13:05:09 -04:00
Scott Idem
c7f1341b1e docs(lookup): expand Section 4 with override model, group key invariants
Documents the root cause of the timezone collapse bug and how to avoid it
in future data imports. Covers:
- group as the dedup identity key (not a display label), per lookup type
- correct way to add/update/remove account and object overrides
- hard invariant for time_zone: group must equal name
- verification query to catch bad seed data before it ships
- frontend keying guidance: use group, not id or name

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 17:56:50 -04:00
Scott Idem
15b5084df3 Quick lookup project for time zones. 2026-03-23 17:50:59 -04:00
Scott Idem
a0767b1c69 Saving documentation updates and clean up. 2026-03-18 16:16:58 -04:00
Scott Idem
356f4b8efc Docs: Modernize main README, archive legacy/deprecated guides, and mark completed security/project docs (March 2026 review) 2026-03-18 16:16:20 -04:00
Scott Idem
74ad69bc63 Saving updated to do list 2026-03-18 14:35:39 -04:00
Scott Idem
d7b86cc186 docs(todo): add Bitbucket app password → API token migration deadline
App passwords inactive 2026-06-09.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 19:24:05 -04:00
Scott Idem
950e34cabd docs: mark Novi-Mailman Bridge POC complete and update notes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-17 18:17:18 -04:00
Scott Idem
29579fd9f1 feat: add V3 action endpoint for event exhibit tracking export
- New router: app/routers/api_v3_actions_event_exhibit.py
  - GET /v3/action/event_exhibit/{exhibit_id}/tracking_export
  - Full V3 auth (x-aether-api-key + account context)
  - Multi-tenant ownership check via check_account_access
  - Permission gate: leads_api_access flag OR manager-level access
  - Returns CSV or XLSX file attachment (return_file=false for JSON)
  - Flattens responses_json custom Q&A columns; strips HTML from exhibitor_notes
  - Exports all records regardless of hidden/enabled state

- Registered in registry.py under prefix /v3/action/event_exhibit

- New E2E test: tests/e2e/test_e2e_v3_action_event_exhibit_tracking_export.py
  - 7/7 tests passing against dev-api.oneskyit.com

- Docs: GUIDE__AE_API_V3_for_Frontend.md — new Section 7 covering endpoint
  usage, columns, leads_api_access dual-purpose (3rd-party API + UI export gate)

- Docs: tests/README.md — added test to table and when-to-run matrix
2026-03-16 16:50:32 -04:00
Scott Idem
e608696ec8 Docs: WS guide - client_id is UUID v4 persisted in localStorage, not Date.now() 2026-03-11 19:03:16 -04:00
Scott Idem
c7c14e8047 Docs: fix WS guide - client_id/group_id are opaque strings, not required Vision IDs 2026-03-11 19:02:06 -04:00
Scott Idem
8f0e6c16bc TODO: queue WS V3 frontend test after IDAA work 2026-03-11 16:01:16 -04:00
Scott Idem
32b519c507 V3 WebSocket: wire auth dependency, add heartbeat presence refresh, update frontend guide (wss://, auth query params, schema clarifications) 2026-03-11 15:21:19 -04:00
Scott Idem
8c7263fdbf Update docs: correct guide links in README, add March 11 session notes to TODO__Agents 2026-03-11 15:14:33 -04:00
Scott Idem
a20c436013 Migrate clip/convert to V3 actions; add background clip support, redirect legacy route; update frontend guide 2026-03-11 14:51:08 -04:00
Scott Idem
f110c2eecb docs: Expand TODO with Novi-Mailman bridge details and session notes
- Add confirmed Novi API patterns (auth, field names, endpoints)
- List remaining unknowns and data_store credential setup requirements
- Add pydantic/SQLAlchemy migration notes
- Summarize completed operational hardening and BuildKit cache work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 19:30:01 -04:00
Scott Idem
f1c8958a7a feat: Scaffold Novi-Mailman Bridge integration
- app/methods/e_novi_mailman_methods.py: full sync engine, per-member
  sync helper, webhook handler, and Mailman 3 REST subscribe/unsubscribe
- app/routers/api_v3_actions_e_novi_mailman.py: test_connection, list
  inspection, full sync trigger, and Novi webhook receiver endpoints
- registry.py: registered at /v3/action/e_novi_mailman
- TODO: marked as scaffolded, pending Novi field verification + data_store setup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 19:16:16 -04:00
Scott Idem
fc3277086f feat: Add BuildKit pip cache, unlock fastapi pin, mark Locking TODO complete
- Dockerfile: enable BuildKit syntax, use --mount=type=cache for pip to speed up rebuilds
- requirements.txt: relax fastapi==0.115.5 → fastapi>=0.115.5
- TODO: mark Locking task as complete

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 18:53:54 -04:00
Scott Idem
32560d2257 feat: Operational hardening — healthcheck, config refactor, requirements lock
- Add GET /health route (DB + Redis ping, 200/503) with Dockerfile HEALTHCHECK directive
- Replace config.py stub with real pydantic BaseSettings reading directly from env vars;
  remove external config file mount from docker-compose
- Add requirements.lock (pip freeze snapshot for bit-identical builds)
- Untrack config.py globally but allow app/config.py via .gitignore negation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-10 18:44:58 -04:00
Scott Idem
d35f374a45 Renamed the todo file for agents. 2026-03-10 16:26:51 -04:00
Scott Idem
9d89d4c8e4 fix: exclude account_id and virtual fields from archive_content DB writes
- Adds fields_to_exclude_from_db to Archive_Content_Base to prevent SQL errors on non-existent columns.
- Updates documentation for V3 Create/Update patterns and the x-ae-ignore-extra-fields header.
- Propagates account_id_random to hosted file and media processing methods.
2026-02-24 11:30:17 -05:00
Scott Idem
f518d7a433 Saving notes 2026-02-20 19:46:03 -05:00
Scott Idem
48fc97cf46 feat: add priority filtering and sort stability to V3 Lookup System 2026-02-20 17:18:21 -05:00
Scott Idem
6bfbff309a feat: implement V3 Uniform Lookup System with hierarchical overrides and site-based whitelisting 2026-02-20 14:48:50 -05:00
Scott Idem
2b2a2bc00f Saving notes about new lookup tables. 2026-02-19 19:19:59 -05:00
Scott Idem
6a023a82f5 Saving notes about reviewing the SQL VIEWs. 2026-02-19 17:56:08 -05:00
Scott Idem
1db71f85a5 Saving updates to notes. Less reference to the _random. 2026-02-19 16:20:25 -05:00
Scott Idem
17a627a981 feat: Implement Event File Hosted Data Fix and API Guide Update
Address critical data visibility issues for Event Files and enhance frontend documentation.

This commit resolves the persistent problem where top-level hosted file convenience fields
(e.g., , , ) were
returning as  in V3 Event File API responses, even when .

Key changes include:
- Refactored  Pydantic model:
    - Removed redundant  definitions from top-level hosted file convenience fields,
      allowing direct mapping from SQL view columns.
    - Simplified  to focus solely on conditionally loading the nested
       object, as top-level fields are now populated directly by Pydantic
      from the  view.
    - Added comprehensive comments to clarify data flow, Pydantic's behavior, and the
      expected origin of these convenience fields from SQL views.
- Updated :
    - Introduced a new section detailing how to retrieve Event File data, including the
      use of  to get both top-level convenience fields and a nested
       object.
    - Clarified all ID references as random string IDs.
    - Renumbered the troubleshooting section.
- Copied updated guide to .
- Continued ID Vision compliance audit, ensuring consistent handling of random string IDs
  across various core and event models (Account, Address, Contact, DataStore, Event Badge Template).
- Consolidated ID Vision E2E tests and updated related documentation.
- Minor updates to  and
  to support Event File data retrieval with .
2026-02-19 15:22:17 -05:00
Scott Idem
577d784fb8 Serious notes about security updates. 2026-02-13 19:22:33 -05:00
Scott Idem
2266f149f7 security(v3): harden multi-tenant isolation and enhance failure feedback 2026-02-13 18:45:20 -05:00
Scott Idem
61e17f1efa Updating the documentation for things. Hopefully improvements to Gemini CLI. 2026-02-11 17:59:15 -05:00
Scott Idem
8270f7ff7a fix(v3-actions): implement from_hosted_file and harden vision IDs
1. Implemented specialized 'from_hosted_file' action for Event Files.\n2. Fixed ValueError in Pydantic models by removing default/default_factory conflict.\n3. Hardened integer stripping to strictly enforce Vision Standards.\n4. Updated documentation for the new action route.
2026-02-06 16:23:18 -05:00
Scott Idem
4aadb4ec1c Update related to object aliases. 2026-02-06 14:20:19 -05:00
Scott Idem
12d725f468 feat(v3-standardization): implement ID Vision for Event models and update docs 2026-02-05 17:37:52 -05:00
Scott Idem
cc5af1c2e2 feat(api-v3): implement temporary ?key= access pattern and update guide
- Added ?key= query param support for unauthenticated direct downloads.
- Fixed site table column bug (auth_key -> access_key).
- Updated GUIDE__V3_FRONTEND_API.md with temporary auth documentation.
- Ensured valid keys bypass the 403 Machine Auth requirement.
2026-02-03 18:03:03 -05:00
Scott Idem
07609bae9a feat(hosted-file): implement hash-based download action and flexible auth
- Adds GET /v3/action/hosted_file/hash/{sha256}/download for direct content-addressable storage access.
- Updates V3 authentication dependencies to support 'api_key' in the query parameter (alias 'api_key').
- Implements auth_method: 'api_key' for machine-to-machine requests that provide a valid key but no user/account context.
- Updates GUIDE__V3_FRONTEND_API.md with the new endpoint and auth options.
2026-02-03 13:44:07 -05:00
Scott Idem
bcd466edc7 feat(event-file): implement atomic V3 upload action for event files
- Creates api_v3_actions_event_file.py with a specialized /upload endpoint.
- Handles physical storage (hosted_file), generic linking, and event association (event_file) in one request.
- Implements intelligent ID resolution to prevent duplicate event associations for the same physical file.
- Updates documentation in GUIDE__V3_FRONTEND_API.md.
2026-02-03 12:54:17 -05:00
Scott Idem
ea117bf268 feat(hosted-file): implement intelligent ID resolution for V3 download action
- Updates download_file_action to automatically resolve container IDs (like event_file) to the underlying hosted_file.
- Updates GUIDE__V3_FRONTEND_API.md to document the 'ID Vision' standard for downloads.
- Resolves 404 errors observed when frontend passed event_file IDs to the hosted_file download endpoint.
2026-02-03 12:05:04 -05:00