OSIT-AE-API-FastAPI/tests/README.md

Aether API Test Suite

This directory contains the automated and manual test scripts for the Aether FastAPI backend. The suite is organized by execution environment and scope to ensure a reliable development workflow.

📁 Directory Structure

• unit/: Isolated logic tests. These use heavy mocking to bypass database and network requirements. Fast and safe to run in any environment.
• integration/: Local environment tests. These verify component interactions, often requiring a connection to the local MariaDB/Redis instance.
• e2e/ (End-to-End): Network-based API tests. these use the requests library to call the live API endpoints at https://dev-api.oneskyit.com.
• tools/: Utility scripts for administrative tasks like registry generation, Docker exploration, and performance stress testing.
• archive/: Legacy or deprecated scripts kept for historical reference.

📜 Standardized E2E Suite (tests/e2e/)

These consolidated scripts are the primary verification tool for the V3 API.

Script	Description
test_e2e_v3_search_engine.py	Primary Search: Basic operators, Registry fields, Nested search, and Filter bypass.
test_e2e_v3_security_audit.py	Core Security: Verifies multi-tenant isolation, cross-account write blocking, and ID Vision compliance.
test_e2e_v3_auth_security.py	Primary Auth: Site bootstrap, Passcode-to-JWT, and permission boundaries.
test_e2e_v3_user_action_routes.py	V3 User Actions: Sign-in (username+password and auth-key flow), verify password, change password, new auth key, email magic link, and auth guards.
test_e2e_v3_user_auth_routes.py	Legacy User Routes: Tests the pre-V3 /user/* endpoints (change_password, new_auth_key, verify_password, lookup, email_auth_key_url, authenticate).
test_e2e_v3_actions_file_lifecycle.py	Primary Actions: Upload, Download (ID/Hash/Streaming), and physical Deletion.
test_e2e_v3_data_store_lookup.py	V3 Parity: Verifies code-based lookups and latency simulation.
test_e2e_redis_extensive.py	Redis Stress: Benchmarks bidirectional ID caching across thousands of records.
test_e2e_v3_event_vision_parity.py	Vision ID: Verifies string-ID enforcement across event models.
test_e2e_v3_cms_vision_parity.py	Vision ID: Verifies string-ID enforcement across CMS (post/comment) models.
test_e2e_v3_core_vision_parity.py	Vision ID: Verifies string-ID and polymorphic resolution across core models (Account, Person, Address, Contact, DataStore).
test_e2e_v3_demo_parity.py	Demo Parity + Nested Create Regression: Vision ID check for Badge, Exhibit, Tracking; nested create lifecycle (POST+DELETE) for journal/journal_entry and event/event_session; alias resolution. Run after any model or nested-router change.
test_e2e_v3_action_event_file.py	Event Actions: Specialized atomic upload and linking for event files.
test_e2e_v3_action_zoom.py	Zoom Integration: Verifies OAuth and ticket sync logic for Zoom Events.
test_e2e_v3_action_novi_mailman.py	Novi-Mailman Bridge — Connections: Verifies Novi AMS and Mailman 3 API credentials are valid (IDAA). Run first before the lists test.
test_e2e_v3_action_novi_mailman_lists.py	Novi-Mailman Bridge — List Operations: Full member lifecycle — read roster, subscribe, verify, unsubscribe — against mm3@idaa.org, mm3@dgrzone.com, mm3@oneskyit.com.
test_e2e_v3_action_event_exhibit_tracking_export.py	Exhibit Leads Export: Auth/permission guards, CSV column structure, XLSX bytes, and return_file mode for the V3 tracking export action.
test_e2e_v3_action_idaa_novi_verify.py	IDAA Novi Member Verify: Auth guard, 200 verified, 404 not-found, 429 rate-limit, 503 unreachable, Redis cache hit, email normalization. (not yet written — add when endpoint is stable)
test_e2e_v3_accounts.py	CRUD verification for the core Account object.
test_e2e_v3_schema.py	Network verification of the V3 metadata discovery endpoint.
test_e2e_agent_bridge.py	Verifies container diagnostics and log streaming routes.
cleanup_test_files.py	Utility to purge E2E artifacts from physical storage.

---

🔧 Tools (tests/tools/)

Script	Description
stress_list_queries.py	Read-only concurrency stress test. Fires N worker threads making R sequential requests across all V3 list endpoints. Reports per-endpoint p50/p95/max latency and error counts. CLI: --workers (default 10), --requests (default 5), --limit (default 20), --base-url (default dev API). Exit code 1 on any error.
tool_generate_registry.py	Generates the object type registry from source definitions.
tool_mcp_docker_explorer.py	Explores running Docker containers via the MCP bridge.

Stress test quick reference:

# Baseline (10 workers, 5 rounds, 400 total requests)
./environment/bin/python3 tests/tools/stress_list_queries.py

# Heavy load (35 workers, 5 rounds, 1400 total requests)
./environment/bin/python3 tests/tools/stress_list_queries.py --workers 35 --requests 5

# Target a different environment
./environment/bin/python3 tests/tools/stress_list_queries.py --base-url https://api.oneskyit.com --workers 5

---

🛠️ Shared Helpers

• mock_config_helper.py: A critical utility that mocks app.config.settings before other modules are imported. Use this in unit tests.

---

🚦 When to Run Tests

Tests exist to be used — run the relevant suite whenever you touch backend code, not just when something breaks.

Change type	Required suites
Model root_validator / ID Vision changes	test_e2e_v3_demo_parity.py, test_e2e_v3_event_vision_parity.py, test_e2e_v3_core_vision_parity.py
Nested router (api_crud_v3_nested.py) changes	test_e2e_v3_demo_parity.py
Search / filter changes	test_e2e_v3_search_engine.py
Auth / account context changes	test_e2e_v3_security_audit.py, test_e2e_v3_auth_security.py
User action route changes (sign-in, password, magic link)	test_e2e_v3_user_action_routes.py
File upload / download changes	test_e2e_v3_actions_file_lifecycle.py
Novi-Mailman bridge changes	test_e2e_v3_action_novi_mailman.py, test_e2e_v3_action_novi_mailman_lists.py
IDAA Novi member verify changes	tests/unit/test_unit_idaa_novi_verify.py, test_e2e_v3_action_idaa_novi_verify.py (e2e pending)
Event exhibit tracking export changes	test_e2e_v3_action_event_exhibit_tracking_export.py
Any backend change before frontend hand-off	All of the above

---

🧹 Maintenance Policy

1. Standardization: All E2E tests should use the standard Agent API Key (PMM4n50teUCaOMMTN8qOJA) and provide clean [✅ PASS] or [❌ FAIL] output.
2. Archiving: When a new specialized test is created, check if it can be combined into one of the "Primary" suites above. If so, combine and move the original to archive/.
3. Cleanup: Always use tests/e2e/cleanup_test_files.py after running file lifecycle tests.

🛠️ Standards & Patterns

To maintain a "nice" and readable test suite, follow these patterns in all new Python E2E scripts:

• Helper Functions: Use a print_result(label, success, message="") helper to output standardized ✅ PASS and ❌ FAIL status lines.
• Functional Isolation: Wrap test scenarios in descriptive functions (e.g., test_hierarchical_fallback()) rather than writing monolithic scripts.
• Vision Compliance: Always verify that IDs in the response are strings (Random IDs) and not integers, to ensure "ID Vision" is working.
• Performance Logging: Include a suite timer at the end of the if __name__ == "__main__": block to track execution speed.
• Clean Environment: Do not leave temporary debug or local-only scripts in the e2e/ directory.

---

🚀 How to Run

Recommended: Use the project virtual environment

./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py

Running unit tests with pytest

./environment/bin/python3 -m pytest tests/unit/ -v

pytest and pytest-asyncio are dev-only dependencies (not in requirements.txt). After rebuilding the venv (e.g. following an OS Python update), reinstall them:

./environment/bin/pip install pytest pytest-asyncio

Path Requirements

Always run test scripts from the project root directory. Most scripts include sys.path.append(os.getcwd()) to ensure local imports work correctly.

---

Development / Testing / Demo environment information

• Use snake_case (or Snake_Case or Snake_case or test_NASA_example or test_API_key)
• Aether test/demo base URL: 'http://demo.localhost:5173'
• Aether development API: 'https://dev-api.oneskyit.com'
• Aether development API "secret" key: 'nT0jPeiCfxSifkiDZur9jA'

These are IDs for records that we can use for testing. Please do not delete them. They are also used for demo purposes with clients.

Core Modules

• Aether test/demo Account: '_XY7DXtc9MY' (1) "One Sky IT Demo"
• Aether test/demo Site: '92vkYC4fVEl' (12) "One Sky IT Demo"
• Aether test/demo Site Domain: '_6jcTbnJk-o' (12) "demo.localhost:5173"
• Aether test/demo Site Domain: 'heXRgHOs4ns' (30) "sk-demo.oneskyit.com"
• Aether test/demo Site Domain: 'DASm8fP92yw' (69) "dev-demo.oneskyit.com"
• Aether test/demo Site Domain: '2i_0Za6yRPo' (2) "demo.oneskyit.com"
• Aether test/demo Person: 'QWODAPCNLQU' (49) "Osiris Idem"
• Aether test/demo Person: 'HMQRNPIXQMK' (48) "Cleo Idem"

Events Modules

• Aether test/demo Event: 'pjrcghqwert' (1) "Demo One Sky IT Conference"
• Aether test/demo Event Session: 'DOW3h7v6H42' (703) "How To Do Things"
• Aether test/demo Event Session (Digital Posters): "K8cxUIEWyQk" "The Beginning of Digital Posters!"
• Aether test/demo Event Session (Digital Posters): "1Un1xI1Rgk8" "Poster Session 99: All about posters!"
• Aether test/demo Event Presentation: '7U2eXSjR6H4' (1670) "Build a House"
• Aether test/demo Event Presenter: 'gT-hxnifb-0' (2202) "Bob The Builder"
• Aether test/demo Event File: 'OOsHXtng5mr' (2985) "1 Quick Test for macOS.mp4"
• Aether test/demo Event Badge: 'UIJT-73-63-61' (37163) "Scott Idem"
• Aether test/demo Event Person: 'ffkKxiHpOEC' (16603) "Scott Idem"
• Aether test/demo Event Badge Template: 'jgfixEpYp1B' (18) "Dev Demo 202x"
• Aether test/demo Event Badge Template: 'rzmUgsk7mkq' (19) "Dev Demo 202x Workshops"
• Aether test/demo Event Location: 'VXXY-98-46-14' (26) "Ballroom 1"
• Aether test/demo Event Location: 'FGRN-67-92-45' (298) "Ballroom AB"
• Aether test/demo Event Location: 'PQKB-15-39-81' (78) "Poster Display Station A"

Journals Module

• Aether test/demo Journal: 'BVYE-94-46-29' (42) "Testing Things"
• Aether test/demo Journal Entry: 'xRx-Y4-h3-fU' (233) "Another Journal Entry in the Test Journal"

Archives Module (IDAA Archives)

• Aether test/demo Archive: 'nAA2bHLv8RK' (1) "One Sky Test Archive"
• Aether test/demo Archive Content: 'UjKzrk-GKu5' (1) "Hosted File Test"

Posts Module (IDAA Bulletin Board)

• Aether test/demo Post:
• Aether test/demo Post:

Events Module (IDAA Recovery Meetings)

• Aether test/demo Event: '1Pkd025vvxU' (36) "IDAA Recovery Meeting Test"
• Aether test/demo Event: 'gIZgAjISkf8' (43) "IDAA Recovery Meeting Test"

IDAA and Novi AMS

Scott Idem (test 1)

• Novi (customer) UUID: "1dadf11c-b74b-4582-8a0a-7ec738a033dc"
• Novi Email: "stidem+test1@gmail.com"