221854df90
Proxies GET /customers/{uuid} to Novi AMS server-to-server so members'
browser IPs are no longer in the call path, eliminating false "Access
Denied" for users on hotel/conference WiFi, VPNs, and CDN-filtered nets.
- New router: GET /v3/action/idaa/novi_member/{uuid}
- Business logic in app/methods/idaa_novi_verify_methods.py
- Redis cache (4h TTL, key: idaa:novi_member:{uuid})
- 404 never cached (recently-joined member anti-pattern)
- Email space→+ normalization (Novi quirk)
- Display name: "FirstName L." format with Name field fallback
- Registered in registry.py under /v3/action/idaa tag
- 9 unit tests covering all response paths (200/404/429/503/unreachable,
cache hit, email normalization, display name format)
- Frontend guide (Section 12) and tests/README updated with full spec
and migration table for frontend hand-off
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
185 lines
11 KiB
Markdown
185 lines
11 KiB
Markdown
# Aether API Test Suite
|
|
|
|
This directory contains the automated and manual test scripts for the Aether FastAPI backend. The suite is organized by execution environment and scope to ensure a reliable development workflow.
|
|
|
|
## 📁 Directory Structure
|
|
|
|
- **`unit/`**: Isolated logic tests. These use heavy mocking to bypass database and network requirements. Fast and safe to run in any environment.
|
|
- **`integration/`**: Local environment tests. These verify component interactions, often requiring a connection to the local MariaDB/Redis instance.
|
|
- **`e2e/` (End-to-End)**: Network-based API tests. these use the `requests` library to call the live API endpoints at `https://dev-api.oneskyit.com`.
|
|
- **`tools/`**: Utility scripts for administrative tasks like registry generation, Docker exploration, and performance stress testing.
|
|
- **`archive/`**: Legacy or deprecated scripts kept for historical reference.
|
|
|
|
## 📜 Standardized E2E Suite (`tests/e2e/`)
|
|
|
|
These consolidated scripts are the primary verification tool for the V3 API.
|
|
|
|
| Script | Description |
|
|
| :--- | :--- |
|
|
| `test_e2e_v3_search_engine.py` | **Primary Search**: Basic operators, Registry fields, Nested search, and Filter bypass. |
|
|
| `test_e2e_v3_security_audit.py` | **Core Security**: Verifies multi-tenant isolation, cross-account write blocking, and ID Vision compliance. |
|
|
| `test_e2e_v3_auth_security.py` | **Primary Auth**: Site bootstrap, Passcode-to-JWT, and permission boundaries. |
|
|
| `test_e2e_v3_user_action_routes.py` | **V3 User Actions**: Sign-in (username+password and auth-key flow), verify password, change password, new auth key, email magic link, and auth guards. |
|
|
| `test_e2e_v3_user_auth_routes.py` | **Legacy User Routes**: Tests the pre-V3 `/user/*` endpoints (change_password, new_auth_key, verify_password, lookup, email_auth_key_url, authenticate). |
|
|
| `test_e2e_v3_actions_file_lifecycle.py` | **Primary Actions**: Upload, Download (ID/Hash/Streaming), and physical Deletion. |
|
|
| `test_e2e_v3_data_store_lookup.py` | **V3 Parity**: Verifies code-based lookups and latency simulation. |
|
|
| `test_e2e_redis_extensive.py` | **Redis Stress**: Benchmarks bidirectional ID caching across thousands of records. |
|
|
| `test_e2e_v3_event_vision_parity.py`| **Vision ID**: Verifies string-ID enforcement across event models. |
|
|
| `test_e2e_v3_cms_vision_parity.py`| **Vision ID**: Verifies string-ID enforcement across CMS (post/comment) models. |
|
|
| `test_e2e_v3_core_vision_parity.py`| **Vision ID**: Verifies string-ID and polymorphic resolution across core models (Account, Person, Address, Contact, DataStore). |
|
|
| `test_e2e_v3_demo_parity.py` | **Demo Parity + Nested Create Regression**: Vision ID check for Badge, Exhibit, Tracking; nested create lifecycle (POST+DELETE) for `journal/journal_entry` and `event/event_session`; alias resolution. **Run after any model or nested-router change.** |
|
|
| `test_e2e_v3_action_event_file.py` | **Event Actions**: Specialized atomic upload and linking for event files. |
|
|
| `test_e2e_v3_action_zoom.py` | **Zoom Integration**: Verifies OAuth and ticket sync logic for Zoom Events. |
|
|
| `test_e2e_v3_action_novi_mailman.py` | **Novi-Mailman Bridge — Connections**: Verifies Novi AMS and Mailman 3 API credentials are valid (IDAA). Run first before the lists test. |
|
|
| `test_e2e_v3_action_novi_mailman_lists.py` | **Novi-Mailman Bridge — List Operations**: Full member lifecycle — read roster, subscribe, verify, unsubscribe — against `mm3@idaa.org`, `mm3@dgrzone.com`, `mm3@oneskyit.com`. |
|
|
| `test_e2e_v3_action_event_exhibit_tracking_export.py` | **Exhibit Leads Export**: Auth/permission guards, CSV column structure, XLSX bytes, and `return_file` mode for the V3 tracking export action. |
|
|
| `test_e2e_v3_action_idaa_novi_verify.py` | **IDAA Novi Member Verify**: Auth guard, 200 verified, 404 not-found, 429 rate-limit, 503 unreachable, Redis cache hit, email normalization. (not yet written — add when endpoint is stable) |
|
|
| `test_e2e_v3_accounts.py` | CRUD verification for the core Account object. |
|
|
| `test_e2e_v3_schema.py` | Network verification of the V3 metadata discovery endpoint. |
|
|
| `test_e2e_agent_bridge.py` | Verifies container diagnostics and log streaming routes. |
|
|
| `cleanup_test_files.py` | Utility to purge E2E artifacts from physical storage. |
|
|
|
|
---
|
|
|
|
## 🔧 Tools (`tests/tools/`)
|
|
|
|
| Script | Description |
|
|
| :--- | :--- |
|
|
| `stress_list_queries.py` | **Read-only concurrency stress test.** Fires N worker threads making R sequential requests across all V3 list endpoints. Reports per-endpoint p50/p95/max latency and error counts. CLI: `--workers` (default 10), `--requests` (default 5), `--limit` (default 20), `--base-url` (default dev API). Exit code 1 on any error. |
|
|
| `tool_generate_registry.py` | Generates the object type registry from source definitions. |
|
|
| `tool_mcp_docker_explorer.py` | Explores running Docker containers via the MCP bridge. |
|
|
|
|
**Stress test quick reference:**
|
|
```bash
|
|
# Baseline (10 workers, 5 rounds, 400 total requests)
|
|
./environment/bin/python3 tests/tools/stress_list_queries.py
|
|
|
|
# Heavy load (35 workers, 5 rounds, 1400 total requests)
|
|
./environment/bin/python3 tests/tools/stress_list_queries.py --workers 35 --requests 5
|
|
|
|
# Target a different environment
|
|
./environment/bin/python3 tests/tools/stress_list_queries.py --base-url https://api.oneskyit.com --workers 5
|
|
```
|
|
|
|
---
|
|
|
|
## 🛠️ Shared Helpers
|
|
|
|
- **`mock_config_helper.py`**: A critical utility that mocks `app.config.settings` before other modules are imported. Use this in unit tests.
|
|
|
|
---
|
|
|
|
## 🚦 When to Run Tests
|
|
|
|
Tests exist to be used — run the relevant suite whenever you touch backend code, not just when something breaks.
|
|
|
|
| Change type | Required suites |
|
|
| :--- | :--- |
|
|
| Model `root_validator` / ID Vision changes | `test_e2e_v3_demo_parity.py`, `test_e2e_v3_event_vision_parity.py`, `test_e2e_v3_core_vision_parity.py` |
|
|
| Nested router (`api_crud_v3_nested.py`) changes | `test_e2e_v3_demo_parity.py` |
|
|
| Search / filter changes | `test_e2e_v3_search_engine.py` |
|
|
| Auth / account context changes | `test_e2e_v3_security_audit.py`, `test_e2e_v3_auth_security.py` |
|
|
| User action route changes (sign-in, password, magic link) | `test_e2e_v3_user_action_routes.py` |
|
|
| File upload / download changes | `test_e2e_v3_actions_file_lifecycle.py` |
|
|
| Novi-Mailman bridge changes | `test_e2e_v3_action_novi_mailman.py`, `test_e2e_v3_action_novi_mailman_lists.py` |
|
|
| IDAA Novi member verify changes | `tests/unit/test_unit_idaa_novi_verify.py`, `test_e2e_v3_action_idaa_novi_verify.py` (e2e pending) |
|
|
| Event exhibit tracking export changes | `test_e2e_v3_action_event_exhibit_tracking_export.py` |
|
|
| Any backend change before frontend hand-off | All of the above |
|
|
|
|
---
|
|
|
|
## 🧹 Maintenance Policy
|
|
|
|
1. **Standardization**: All E2E tests should use the standard Agent API Key (`PMM4n50teUCaOMMTN8qOJA`) and provide clean `[✅ PASS]` or `[❌ FAIL]` output.
|
|
2. **Archiving**: When a new specialized test is created, check if it can be combined into one of the "Primary" suites above. If so, combine and move the original to `archive/`.
|
|
3. **Cleanup**: Always use `tests/e2e/cleanup_test_files.py` after running file lifecycle tests.
|
|
|
|
## 🛠️ Standards & Patterns
|
|
|
|
To maintain a "nice" and readable test suite, follow these patterns in all new Python E2E scripts:
|
|
|
|
- **Helper Functions**: Use a `print_result(label, success, message="")` helper to output standardized `✅ PASS` and `❌ FAIL` status lines.
|
|
- **Functional Isolation**: Wrap test scenarios in descriptive functions (e.g., `test_hierarchical_fallback()`) rather than writing monolithic scripts.
|
|
- **Vision Compliance**: Always verify that IDs in the response are strings (Random IDs) and not integers, to ensure "ID Vision" is working.
|
|
- **Performance Logging**: Include a suite timer at the end of the `if __name__ == "__main__":` block to track execution speed.
|
|
- **Clean Environment**: Do not leave temporary debug or local-only scripts in the `e2e/` directory.
|
|
|
|
---
|
|
|
|
## 🚀 How to Run
|
|
|
|
### Recommended: Use the project virtual environment
|
|
```bash
|
|
./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py
|
|
```
|
|
|
|
### Running unit tests with pytest
|
|
```bash
|
|
./environment/bin/python3 -m pytest tests/unit/ -v
|
|
```
|
|
|
|
`pytest` and `pytest-asyncio` are dev-only dependencies (not in `requirements.txt`). After rebuilding the venv (e.g. following an OS Python update), reinstall them:
|
|
```bash
|
|
./environment/bin/pip install pytest pytest-asyncio
|
|
```
|
|
|
|
### Path Requirements
|
|
Always run test scripts from the **project root** directory. Most scripts include `sys.path.append(os.getcwd())` to ensure local imports work correctly.
|
|
|
|
|
|
---
|
|
|
|
## Development / Testing / Demo environment information
|
|
* Use snake_case (or Snake_Case or Snake_case or test_NASA_example or test_API_key)
|
|
* Aether test/demo base URL: 'http://demo.localhost:5173'
|
|
* Aether development API: 'https://dev-api.oneskyit.com'
|
|
* Aether development API "secret" key: 'nT0jPeiCfxSifkiDZur9jA'
|
|
|
|
These are IDs for records that we can use for testing. Please do not delete them. They are also used for demo purposes with clients.
|
|
|
|
### Core Modules
|
|
* Aether test/demo Account: '_XY7DXtc9MY' (1) "One Sky IT Demo"
|
|
* Aether test/demo Site: '92vkYC4fVEl' (12) "One Sky IT Demo"
|
|
* Aether test/demo Site Domain: '_6jcTbnJk-o' (12) "demo.localhost:5173"
|
|
* Aether test/demo Site Domain: 'heXRgHOs4ns' (30) "sk-demo.oneskyit.com"
|
|
* Aether test/demo Site Domain: 'DASm8fP92yw' (69) "dev-demo.oneskyit.com"
|
|
* Aether test/demo Site Domain: '2i_0Za6yRPo' (2) "demo.oneskyit.com"
|
|
* Aether test/demo Person: 'QWODAPCNLQU' (49) "Osiris Idem"
|
|
* Aether test/demo Person: 'HMQRNPIXQMK' (48) "Cleo Idem"
|
|
|
|
### Events Modules
|
|
* Aether test/demo Event: 'pjrcghqwert' (1) "Demo One Sky IT Conference"
|
|
* Aether test/demo Event Session: 'DOW3h7v6H42' (703) "How To Do Things"
|
|
* Aether test/demo Event Session (Digital Posters): "K8cxUIEWyQk" "The Beginning of Digital Posters!"
|
|
* Aether test/demo Event Session (Digital Posters): "1Un1xI1Rgk8" "Poster Session 99: All about posters!"
|
|
* Aether test/demo Event Presentation: '7U2eXSjR6H4' (1670) "Build a House"
|
|
* Aether test/demo Event Presenter: 'gT-hxnifb-0' (2202) "Bob The Builder"
|
|
* Aether test/demo Event File: 'OOsHXtng5mr' (2985) "1 Quick Test for macOS.mp4"
|
|
* Aether test/demo Event Badge: 'UIJT-73-63-61' (37163) "Scott Idem"
|
|
* Aether test/demo Event Person: 'ffkKxiHpOEC' (16603) "Scott Idem"
|
|
* Aether test/demo Event Badge Template: 'jgfixEpYp1B' (18) "Dev Demo 202x"
|
|
* Aether test/demo Event Badge Template: 'rzmUgsk7mkq' (19) "Dev Demo 202x Workshops"
|
|
* Aether test/demo Event Location: 'VXXY-98-46-14' (26) "Ballroom 1"
|
|
* Aether test/demo Event Location: 'FGRN-67-92-45' (298) "Ballroom AB"
|
|
* Aether test/demo Event Location: 'PQKB-15-39-81' (78) "Poster Display Station A"
|
|
|
|
### Journals Module
|
|
* Aether test/demo Journal: 'BVYE-94-46-29' (42) "Testing Things"
|
|
* Aether test/demo Journal Entry: 'xRx-Y4-h3-fU' (233) "Another Journal Entry in the Test Journal"
|
|
|
|
### Archives Module (IDAA Archives)
|
|
* Aether test/demo Archive: 'nAA2bHLv8RK' (1) "One Sky Test Archive"
|
|
* Aether test/demo Archive Content: 'UjKzrk-GKu5' (1) "Hosted File Test"
|
|
|
|
### Posts Module (IDAA Bulletin Board)
|
|
* Aether test/demo Post:
|
|
* Aether test/demo Post:
|
|
|
|
### Events Module (IDAA Recovery Meetings)
|
|
* Aether test/demo Event: '1Pkd025vvxU' (36) "IDAA Recovery Meeting Test"
|
|
* Aether test/demo Event: 'gIZgAjISkf8' (43) "IDAA Recovery Meeting Test"
|
|
|
|
### IDAA and Novi AMS
|
|
Scott Idem (test 1)
|
|
* Novi (customer) UUID: "1dadf11c-b74b-4582-8a0a-7ec738a033dc"
|
|
* Novi Email: "stidem+test1@gmail.com" |