Commit Graph

1076 Commits

Author SHA1 Message Date
Scott Idem
19e64135ca Permissive Update: Implement x-ae-ignore-extra-fields header support for nested routes 2026-01-19 16:48:48 -05:00
Scott Idem
7db937f8af Vision ID: Standardize Site Domain and Journal objects with string-only IDs and searchable mapping 2026-01-19 15:57:00 -05:00
Scott Idem
2dbf47d874 Security: Implement JWT verification in V3 and prevent numeric ID signing 2026-01-19 14:41:20 -05:00
Scott Idem
cad0d2e867 Security: Enforce mandatory API Keys for V3, fix search logic, and update frontend guide 2026-01-19 14:11:13 -05:00
Scott Idem
9e0f94964e Bug fix for trying to use the wrong hosted file and tmp paths or src. Also saving documentation for the new MCP AE DB field manager. 2026-01-16 14:40:12 -05:00
Scott Idem
31fd384704 Docs: Consolidate admin documentation and migrate reference data
- Created LOCAL_DEVELOPMENT_GUIDE.md and DEPLOYMENT_GUIDE_MANUAL.md from legacy txt files.
- Migrated country/time_zone data and requirements.txt to documentation/reference_data/.
- Removed redundant admin/documentation/ and admin/data_files/ directories.
- Enhanced app/lib_schema_v3.py to explicitly capture 'required' fields from DB 'NOT NULL' constraint.
- Added verification tests for schema logic and standalone DB connectivity.
2026-01-16 10:06:51 -05:00
Scott Idem
acd770962b Refactor: Modularize logging and finalize lifespan integration 2026-01-15 17:31:32 -05:00
Scott Idem
eccd71f450 Refactor: Modularize database logic and extract core CRUD operations 2026-01-15 17:16:48 -05:00
Scott Idem
5ece1d34e3 Refactor: Relocate bootstrap and validation logic into lifespan context manager 2026-01-15 17:10:42 -05:00
Scott Idem
3f276a42e1 Refactor: Modularize configuration and implement robust DB bootstrap 2026-01-15 16:59:18 -05:00
Scott Idem
16c79aca39 Cleanup: Finalize modularization of app/main.py 2026-01-15 16:45:10 -05:00
Scott Idem
2227432970 Refactor: Modularize middleware and router registration in app/main.py 2026-01-15 16:36:19 -05:00
Scott Idem
d321b94395 chore(tests): organize test scripts and beautify account creation email
- Moved scattered Python test scripts from root and 'admin/development/' to 'tests/'.
- Beautified the HTML email body for account creation links in 'app/methods/person_methods.py' with a modern responsive design.
2026-01-15 14:38:00 -05:00
Scott Idem
f0711f27b4 fix(email): resolve SMTP authentication failure and improve configuration resilience
- Fixed a bug where missing 'id=0' in the 'cfg' table caused SMTP authentication to fail by defaulting to placeholder credentials.
- Updated 'app/lib_email.py' to explicitly validate SMTP server and port settings before connecting, preventing crashes with 'please run connect() first'.
- Added email fallback logic in 'app/methods/person_methods.py' to use 'user_email' or 'primary_email' if the primary contact email is missing.
- Aligned 'app/config.py.default' with the production structure, explicitly re-adding 'SMTP' and 'FILES_PATH' dictionaries.
- Added comprehensive unit tests in 'tests/test_email_configuration.py' to verify configuration handling.
2026-01-15 13:19:58 -05:00
Scott Idem
34a752d455 feat(api-v3): implement permissive updates, automatic ID resolution, and structured error reporting
- Added 'x-ae-ignore-extra-fields' header to support stripping unknown fields in POST/PATCH.
- Added automatic resolution of '*_id_random' strings to integer IDs in 'sanitize_payload'.
- Refactored 'post_obj' to return structured (field -> message) validation errors in 'meta.details'.
- Updated 'mk_resp' to support non-string 'details' in response metadata.
- Added 'tests/verify_feedback_fixes.py' to validate logic changes.

Ref: V3 API Refinement Feedback from mcp_agent.
2026-01-14 19:11:56 -05:00
Scott Idem
19a9890dd9 Better bug fix for working SQL test. 2026-01-13 14:21:56 -05:00
Scott Idem
f9a51e243f More clean up of old routes 2026-01-13 14:15:17 -05:00
Scott Idem
6346d4ccd6 Commenting out a bunch of old routes. Hopefully none of them are used by anything still out there.... 2026-01-13 14:12:07 -05:00
Scott Idem
ed3dda6cf5 Bug fixes for SQL testing 2026-01-13 13:57:18 -05:00
Scott Idem
8927f07bcf Added some extra print debugs for now. 2026-01-12 20:30:45 -05:00
Scott Idem
4b86432381 Enhance V3 CRUD: Implement Error Bubbling and Dry-Run Validation.
- Updated app/db_sql.py to capture SQL exceptions in thread-local storage for later retrieval.
- Implemented format_db_error() in app/lib_api_crud_v3.py to clean up raw MariaDB error strings.
- Added POST /v3/crud/{obj_type}/validate endpoint for dry-run payload validation.
- Updated main and nested routers to bubble up validation and database errors into the response 'meta.details' field.
- Added tests/test_v3_error_bubbling.py to verify formatting logic.
2026-01-09 16:57:54 -05:00
Scott Idem
3885cc6aba Refactor V3 CRUD: Extract schema introspection logic.
- Created app/lib_schema_v3.py to isolate database and Pydantic model introspection.
- Updated app/routers/api_crud_v3.py to use get_object_schema_info(), completing the modularization.
- Finalized refactoring plan documentation in documentation/REFACTOR_API_CRUD_V3.md.
2026-01-09 16:29:10 -05:00
Scott Idem
812181acb5 Refactor V3 CRUD: Extract nested child routes into separate router.
- Created app/routers/api_crud_v3_nested.py to handle all parent-child relational routes.
- Updated app/routers/api_crud_v3.py to include the nested router, significantly reducing file size.
- Documented Phase 2 completion in documentation/REFACTOR_API_CRUD_V3.md.
2026-01-09 16:23:14 -05:00
Scott Idem
8459b57e1b Refactor V3 CRUD: Extract helper functions and unify sanitization logic.
- Created app/lib_api_crud_v3.py to house core security, filtering, and sanitization logic.
- Implemented reusable sanitize_payload() to generically strip virtual lookup fields (*_id_random) and view-only fields (fields_to_exclude_from_db).
- Updated app/routers/api_crud_v3.py to use the new library and consolidated sanitization across all Create/Update endpoints.
- Documented Phase 1 completion in documentation/REFACTOR_API_CRUD_V3.md.
2026-01-09 16:16:44 -05:00
Scott Idem
2ff211f2c2 Update API documentation and finalize model validators/mappings.
- Added comprehensive docstrings to api_crud_v3.py explaining multi-tenancy, sanitization, and soft-delete logic.
- Finalized Address and Contact models/mappings with correct validators and field maps.
- Consolidated test suite in tests/ directory.
2026-01-09 15:52:00 -05:00
Scott Idem
4c83e02c4a Update V3 CRUD router and object definitions.
- Added 'external_person_id' to Post searchable fields.
- Updated api_crud_v3.py to respect 'fields_to_exclude_from_db' model attribute.
- Cleaned up old verification scripts (moved to tests/).
2026-01-09 15:36:50 -05:00
Scott Idem
1c0922ace2 Enhance API robustness: Add model validators, view-field filtering, and test suite.
- Added validators to Person_Base, Journal_Base, Journal_Entry_Base, Address_Base, and Contact_Base to handle null values and unsafe lookups.
- Implemented 'fields_to_exclude_from_db' ClassVar in Journal models to prevent view-only fields from causing DB errors.
- Updated Contact object map to align with DB schema.
- Added comprehensive test suite in 'tests/' directory (model validation, filtering logic).
- Updated GEMINI.md with progress.
2026-01-09 15:36:28 -05:00
Scott Idem
29b4d5ae4b Fix Person creation issues and enhance V3 CRUD robustness.
- Added Pydantic validators to Person_Base to handle null values for given_name and allow_auth_key, ensuring database NOT NULL constraints are met.
- Updated api_crud_v3.py (POST and PATCH) to filter out virtual *_id_random fields from data payloads before database operations to prevent "Unknown column" errors.
- Updated GEMINI.md with session progress.
2026-01-09 14:30:45 -05:00
Scott Idem
802c75bad9 V3: Standardize Primary AE Objects and Synchronize Search Whitelists.
- Synchronized searchable_fields (V3 whitelists) across all Primary and Active AE objects (Identity, People, Events, Journals, Posts, Archives, Business).
- Standardized Pydantic models for core objects to include the 10 common fields (id, id_random, enable, hide, priority, sort, group, notes, created_on, updated_on).
- Fixed field aliases and uncommented valid database columns in User_Base and Organization_Base.
- Pruned non-existent fields from searchable lists in legacy or config-specific definitions (account_cfg, user_role, log_client_viewing).
- Added system discovery and validation tools:
    - ae_object_info.py: AE object status and metadata browser.
    - export_all_interfaces.py: E2E TypeScript interface generator.
    - Verification scripts for searchable field consistency.
- Updated Jan 8 milestone progress and platform roadmap in GEMINI.md.
2026-01-08 12:24:34 -05:00
Scott Idem
59d5b81da0 Arch: Finalize V3 Auth modularization and Unified Agent spec.
- Integrated zero-dependency Auth models and dependencies_v3.py.
- Successfully resolved circular dependency boot loops.
- Verified site_domain search exception via verify_v3_exceptions.py.
- Refined Unified Agent Architecture with Storage Layer and API-driven access details.
- Updated project roadmap and milestones in GEMINI.md.
2026-01-07 19:07:21 -05:00
Scott Idem
d4805ebb09 Cleanup: Remove unused imports in lib_general.py after extraction. 2026-01-07 17:43:43 -05:00
Scott Idem
734576817c Refactor: Modularize lib_general.py by extracting core functionalities.
- Extracted Email functions to app/lib_email.py.
- Extracted Export functions to app/lib_export.py.
- Extracted JWT utilities to app/lib_jwt.py.
- Extracted Hash utilities to app/lib_hash.py.
- Updated app/lib_general.py to import from these new modules for backward compatibility.
- Updated V3 Frontend API Guide with latest security and site_domain exception details.
2026-01-07 17:41:04 -05:00
Scott Idem
d61dd0f00e Restored V3 search and implemented site_domain exception via dependency isolation.
- Implemented 'Isolation Mode' in api_crud_v3.py to bypass circular dependencies.
- Locally defined AccountContext and auth dependencies to ensure stable boot.
- Added site_domain lookup exception for guest users in search endpoint.
- Maintained agent_bridge disablement in main.py for stability.
2026-01-07 17:08:52 -05:00
Scott Idem
6937f9dca4 Saving these changes in a working state. Just in case. 2026-01-07 16:54:56 -05:00
Scott Idem
caf2868d02 Saving things while they work again!!! Still working on adding a special exception or something for site domain search. 2026-01-07 16:25:18 -05:00
Scott Idem
cf96d93246 fix: import SearchFilter in api_crud_v3.py to resolve NameError 2026-01-07 14:25:58 -05:00
Scott Idem
6d13b952c4 Implement V3 API security hardening and multi-tenant data isolation
- Enhanced AuthContext with role-aware fields (administrator, manager, super).
- Implemented deferred database lookups for user roles in get_v3_auth_context.
- Added global account isolation in api_crud_v3.py using check_account_access and apply_forced_account_filter.
- Hardened all V3 CRUD endpoints (GET, POST, PATCH, DELETE) and nested routes with ownership verification.
- Enforced forced account filtering at the SQL level for Listing and Searching.
- Updated documentation with details on the new security and data isolation architecture.
2026-01-07 13:34:38 -05:00
Scott Idem
270712f905 Another quick snapshot in case something breaks again. There are issues with this new agent bridge and the psutil and Gemini CLI. Not sure, but it causes problems. 2026-01-07 13:10:16 -05:00
Scott Idem
7fb2f00846 Things are currently working. At least 90% sure they are. 2026-01-07 12:24:52 -05:00
Scott Idem
c47ae47a2f Add agent_bridge.py administrative endpoints and mcp_docker_explorer.py script
- Implemented /status, /system/usage, /logs, /logs/list, /processes, and /container/metadata in agent_bridge.py.
- Added mcp_docker_explorer.py for Docker MCP integration testing.
- Enhanced administrative access checks in agent_bridge.py.
2026-01-07 12:01:48 -05:00
Scott Idem
75b771f87c feat: add 'archive_on' to searchable_fields for posts and update agent bridge auth logic 2026-01-07 11:07:45 -05:00
Scott Idem
ec4656eca9 Bug fix to make lookup_id_random_pop work again. 2026-01-07 10:02:57 -05:00
Scott Idem
836ed97d07 feat(agent): implement Agent Bridge for secure diagnostics 2026-01-06 19:00:03 -05:00
Scott Idem
6470af0a01 feat(v3): populate searchable_fields for all remaining object definitions 2026-01-06 18:18:39 -05:00
Scott Idem
c33ae332e9 refactor(sql): clean up db_sql.py by removing commented-out code and consolidating logic 2026-01-06 18:12:51 -05:00
Scott Idem
a6ec6d1b2b Saving our work just in case. 2026-01-06 18:06:45 -05:00
Scott Idem
55033d0749 feat(events): add conference to searchable fields and update progress 2026-01-06 18:03:57 -05:00
Scott Idem
868a0060dc refactor(sql): complete modularization of search builders and ID resolution 2026-01-06 17:58:34 -05:00
Scott Idem
56fe7ed953 refactor(sql): modularize Redis and ID lookup functions 2026-01-06 17:32:22 -05:00
Scott Idem
a6a5162385 refactor(sql): modularize status and where query builders 2026-01-06 17:24:47 -05:00