chore(tests): consolidate E2E test suite into standardized primary scripts

- Combined 10+ one-off tests into 4 primary functional suites (Search, Auth, Lifecycle, Vision).
- Archived original scripts to tests/archive/.
- Updated README with the new standardized inventory.
- Applied clean output formatting across the new suite.

tests/README.md

@@ -10,85 +10,35 @@ This directory contains the automated and manual test scripts for the Aether Fas
 - **`tools/`**: Utility scripts for administrative tasks like registry generation or Docker exploration.
 - **`archive/`**: Legacy or deprecated scripts kept for historical reference.
 
-## 🛠️ Shared Helpers
+## 📜 Standardized E2E Suite (`tests/e2e/`)
 
-- **`mock_config_helper.py`**: A critical utility that mocks `app.config.settings` before other modules are imported. Use this in unit tests to prevent the application from trying to load real configuration files during import.
+These consolidated scripts are the primary verification tool for the V3 API.
+
+| Script | Description |
+| :--- | :--- |
+| `test_e2e_v3_search_engine.py` | **Primary Search**: Basic operators, Registry fields, Nested search, and Filter bypass. |
+| `test_e2e_v3_auth_security.py` | **Primary Auth**: Site bootstrap, Passcode-to-JWT, and permission boundaries. |
+| `test_e2e_v3_actions_file_lifecycle.py` | **Primary Actions**: Upload, Download (ID/Hash/Streaming), and physical Deletion. |
+| `test_e2e_v3_data_store_lookup.py` | **V3 Parity**: Verifies code-based lookups and latency simulation. |
+| `test_e2e_v3_event_vision_parity.py`| **Vision ID**: Verifies string-ID enforcement across event models. |
+| `test_e2e_v3_accounts.py` | CRUD verification for the core Account object. |
+| `test_e2e_v3_schema.py` | Network verification of the V3 metadata discovery endpoint. |
+| `test_e2e_agent_bridge.py` | Verifies container diagnostics and log streaming routes. |
+| `cleanup_test_files.py` | Utility to purge E2E artifacts from physical storage. |
 
 ---
 
-## 📜 Script Inventory
+## 🛠️ Shared Helpers
 
-### Unit Tests (`tests/unit/`)
-| Script | Description |
-| :--- | :--- |
-| `test_unit_email.py` | Unit tests for email logic with full SMTP mocking. |
-| `test_unit_errors.py` | Verifies the regex-based SQL error string cleanup logic. |
-| `test_unit_filtering.py` | Ensures virtual/view fields are correctly marked for DB exclusion. |
-| `test_unit_hosted_file_logic.py` | Validates Hosted File model aliasing and ID mapping. |
-| `test_unit_hosted_file_resolver.py` | Tests `lookup_id_random_pop` ID resolution logic. |
-| `test_unit_models.py` | Validates custom Pydantic validators (e.g., Person given_name). |
-| `test_unit_payload_sanitization.py` | **Primary Logic Test**: Verifies payload stripping and ID resolution. |
-| `test_unit_router_stripping.py` | Simulates automatic removal of random IDs during updates. |
-| `test_unit_schema_logic.py` | Verifies V3 schema metadata extraction logic. |
-| `test_unit_upload_files_flow.py` | Tests the logic flow of multi-file uploads. |
-| `test_unit_websockets_v3.py` | Unit tests for the V3 WebSocket manager and messages. |
-| `test_unit_websockets_v3_router.py` | Verifies the V3 WebSocket endpoint logic and routing. |
-
-### Integration Tests (`tests/integration/`)
-| Script | Description |
-| :--- | :--- |
-| `debug_auth_dependency.py` | Direct test of `get_account_context_optional` logic. |
-| `test_int_boot_diagnosis.py` | Progressively imports modules to identify circular traps. |
-| `test_int_db_connectivity.py` | Direct SQLAlchemy connectivity test (bypasses app config). |
-| `test_int_email_live.py` | Live SMTP connection test (non-mocked). |
-| `test_int_hosted_file_upload.py` | Tests route flow using FastAPI `TestClient`. |
-| `test_int_import_verification.py` | Basic check that all V3 routers are reachable. |
-| `test_int_permissive_mode.py` | Tests `x-ae-ignore-extra-fields` header logic. |
-| `test_int_schema_v3.py` | Verifies enhanced schema discovery against real DB. |
-| `test_int_v3_auth_security.py` | Verifies auth bypass rules (Site vs Account). |
-| `test_ws_v3_ping.py` | **Primary Gateway Test**: Verifies WebSocket V3 round-trip. |
-
-### E2E Tests (`tests/e2e/`)
-| Script | Description |
-| :--- | :--- |
-| `cleanup_test_files.py` | Utility to purge E2E artifacts from storage. |
-| `repro_intermittent_errors.py` | Stress test for intermittent 403/Timeout issues. |
-| `test_e2e_agent_bridge.py` | Verifies `/agent` diagnostics and log streaming. |
-| `test_e2e_jwt_guest_auth.py` | **Security Test**: Verifies safe guest token access. |
-| `test_e2e_passcode_auth.py` | **Security Test**: Verifies passcode-to-JWT flow. |
-| `test_e2e_site_bootstrap.py` | Verifies unauthenticated FQDN lookup. |
-| `test_e2e_v3_accounts.py` | CRUD verification for Account object. |
-| `test_e2e_v3_action_delete.py` | Verifies physical/record deletion via V3 Actions. |
-| `test_e2e_v3_action_download.py` | **Consolidated**: Tests ID Vision, Hash, and Partial DLs. |
-| `test_e2e_v3_action_event_file.py` | Tests atomic event_file upload action. |
-| `test_e2e_v3_action_upload.py` | Tests standard hosted_file upload action. |
-| `test_e2e_v3_data_store_lookup.py` | Verifies code lookup, delay, and POST search parity. |
-| `test_e2e_v3_event_device.py` | Vision ID parity test for Event Device. |
-| `test_e2e_v3_event_session.py` | Vision ID parity test for Event Session. |
-| `test_e2e_v3_extra_filters.py` | Tests complex filtering (enabled/all) on User/Site. |
-| `test_e2e_v3_nested_advanced.py` | Tests POST /search and view param on nested routes. |
-| `test_e2e_v3_registry_verify.py` | Verifies registry whitelist expansion. |
-| `test_e2e_v3_schema.py` | Validates V3 `/schema` endpoint over network. |
-| `test_e2e_v3_search.py` | **Primary API Test**: Verifies all search operators. |
-| `test_e2e_v3_security_exceptions.py` | Validates 403 blocks on restricted routes. |
-
-### Tools & Utilities (`tests/tools/` or root)
-| Script | Description |
-| :--- | :--- |
-| `gen_test_jwt.py` | Local script to generate test tokens. |
-| `tool_generate_registry.py` | Exports the Aether object registry as JSON. |
-| `tool_mcp_docker_explorer.py` | Model Context Protocol tool for Docker container inspection. |
+- **`mock_config_helper.py`**: A critical utility that mocks `app.config.settings` before other modules are imported. Use this in unit tests.
 
 ---
 
 ## 🧹 Maintenance Policy
 
-1.  **Redundant Tests**: If a test is consolidated into a more comprehensive suite (e.g., `test_e2e_v3_hash_download.py` -> `test_e2e_v3_action_download.py`), the older one should be moved to `archive/`.
-2.  **Naming Convention**:
-    *   `test_unit_*`: No database or network.
-    *   `test_int_*`: Uses local DB/Redis or `TestClient`.
-    *   `test_e2e_*`: Full network round-trip.
-3.  **Cleanup**: Always use `tests/e2e/cleanup_test_files.py` after running upload/download tests to keep the dev storage clean.
+1.  **Standardization**: All E2E tests should use the standard Agent API Key (`PMM4n50teUCaOMMTN8qOJA`) and provide clean `[✅ PASS]` or `[❌ FAIL]` output.
+2.  **Archiving**: When a new specialized test is created, check if it can be combined into one of the "Primary" suites above. If so, combine and move the original to `archive/`.
+3.  **Cleanup**: Always use `tests/e2e/cleanup_test_files.py` after running file lifecycle tests.
 
 ---
 
@@ -96,8 +46,8 @@ This directory contains the automated and manual test scripts for the Aether Fas
 
 ### Recommended: Use the project virtual environment
 ```bash
-./environment/bin/python3 tests/unit/test_unit_payload_sanitization.py
+./environment/bin/python3 tests/e2e/test_e2e_v3_search_engine.py
 ```
 
 ### Path Requirements
-Always run test scripts from the **project root** directory. Most scripts include `sys.path.append(os.getcwd())` to ensure local imports work correctly.
+Always run test scripts from the **project root** directory. Most scripts include `sys.path.append(os.getcwd())` to ensure local imports work correctly.

tests/e2e/test_e2e_v3_actions_file_lifecycle.py

@@ -0,0 +1,90 @@
+import requests
+import io
+import json
+import os
+
+# --- Configuration ---
+API_BASE = "https://dev-api.oneskyit.com/v3/action"
+API_KEY = "PMM4n50teUCaOMMTN8qOJA"
+ACCOUNT_ID = "Q8lR8Ai8hx2FjbQ3C_EH1Q" # OSIT
+# Linking target for test files
+LINK_TYPE = "archive_content"
+LINK_ID = "bZOa7CtUm0E"
+
+def get_headers():
+    return {
+        "X-Aether-API-Key": API_KEY,
+        "x-account-id": ACCOUNT_ID,
+        "x-no-account-id": "bypass"
+    }
+
+def test_file_lifecycle():
+    print(f"--- Starting Hosted File Lifecycle Test ---")
+    
+    # 1. UPLOAD
+    print("\n[Step 1] Uploading test file...")
+    test_content = b"Lifecycle Test Content: " + os.urandom(8).hex().encode()
+    files = [("file_list", ("lifecycle_test.txt", io.BytesIO(test_content), "text/plain"))]
+    data = {"account_id": ACCOUNT_ID, "link_to_type": LINK_TYPE, "link_to_id": LINK_ID}
+    
+    up_resp = requests.post(f"{API_BASE}/hosted_file/upload", headers=get_headers(), files=files, data=data)
+    if up_resp.status_code != 200:
+        print(f"  ❌ Upload Failed: {up_resp.text}")
+        return False
+    
+    file_info = up_resp.json()['data'][0]
+    file_id = file_info['id']
+    file_hash = file_info['hash_sha256']
+    subdir = file_hash[:2]
+    expected_path = f"/srv/aether_api/srv/ae_hosted_files/{subdir}/{file_hash}.file"
+    print(f"  ✅ Uploaded: {file_id} (Hash: {file_hash[:10]}...)")
+    print(f"  🔍 Expected physical path: {expected_path}")
+
+    # 2. DOWNLOAD (Direct)
+    print("\n[Step 2] Downloading by ID...")
+    dl_resp = requests.get(f"{API_BASE}/hosted_file/{file_id}/download", headers=get_headers())
+    if dl_resp.status_code == 200 and dl_resp.content == test_content:
+        print(f"  ✅ Downloaded content matches original.")
+    else:
+        print(f"  ❌ Download Failed or content mismatch. Status: {dl_resp.status_code}")
+        print(f"     Original: {test_content}")
+        print(f"     Received: {dl_resp.content}")
+        return False
+
+    # 3. DOWNLOAD (Hash + Query Auth)
+    print("\n[Step 3] Downloading by Hash (Query Auth)...")
+    hash_url = f"{API_BASE}/hosted_file/hash/{file_hash}/download?api_key={API_KEY}"
+    h_dl_resp = requests.get(hash_url)
+    if h_dl_resp.status_code == 200:
+        print(f"  ✅ Hash download successful.")
+    else:
+        print(f"  ❌ Hash download failed: {h_dl_resp.status_code}")
+        return False
+
+    # 4. DELETE (Clean Cleanup)
+    print("\n[Step 4] Deleting test file (rm_orphan=true)...")
+    del_params = {"rm_orphan": "true", "method": "delete"}
+    del_resp = requests.delete(f"{API_BASE}/hosted_file/{file_id}", headers=get_headers(), params=del_params)
+    if del_resp.status_code == 200:
+        print(f"  ✅ Deletion request successful.")
+    else:
+        print(f"  ❌ Deletion failed: {del_resp.text}")
+        return False
+
+    # 5. VERIFY DELETED
+    print("\n[Step 5] Verifying record is gone...")
+    check_resp = requests.get(f"https://dev-api.oneskyit.com/v3/crud/hosted_file/{file_id}", headers=get_headers())
+    if check_resp.status_code == 404:
+        print(f"  ✅ Success: Record purged from DB.")
+    else:
+        print(f"  ❌ Failure: Record still exists after deletion.")
+        return False
+
+    return True
+
+if __name__ == "__main__":
+    if test_file_lifecycle():
+        print("\n🎉 HOSTED FILE LIFECYCLE VERIFIED!")
+    else:
+        print("\n❌ LIFECYCLE TEST FAILED.")
+        exit(1)

tests/e2e/test_e2e_v3_auth_security.py

@@ -0,0 +1,65 @@
+import requests
+import json
+import time
+
+# --- Configuration ---
+API_ROOT = "https://dev-api.oneskyit.com"
+API_KEY = "PMM4n50teUCaOMMTN8qOJA"
+SITE_ID = "ltOdfNtjZLo" 
+PASSCODE = "10241024"
+FQDN = "dev-app.oneskyit.com"
+
+def print_result(label, success, message=""):
+    status = "✅ PASS" if success else "❌ FAIL"
+    print(f"[{status}] {label} {message}")
+
+def test_site_bootstrap():
+    """Tests unauthenticated FQDN lookup (Bootstrap Exception)."""
+    print("\n--- Testing Site Bootstrap (Unauth) ---")
+    url = f"{API_ROOT}/v3/crud/site_domain/search"
+    query = {"and": [{"field": "fqdn", "op": "eq", "value": FQDN}]}
+    # NO AUTH HEADERS
+    resp = requests.post(url, json=query)
+    print_result("Bootstrap lookup (site_domain)", resp.status_code == 200)
+
+def test_passcode_to_jwt():
+    """Tests site-specific passcode authentication."""
+    print("\n--- Testing Passcode -> JWT Flow ---")
+    url = f"{API_ROOT}/api/authenticate_passcode"
+    payload = {"site_id": SITE_ID, "passcode": PASSCODE}
+    resp = requests.post(url, json=payload)
+    
+    success = resp.status_code == 200
+    token = resp.json().get('data', {}).get('jwt') if success else None
+    print_result("Passcode Auth", success and token is not None)
+    return token
+
+def test_security_boundaries(token):
+    """Tests that a site-token cannot access private journals."""
+    print("\n--- Testing Security Boundaries ---")
+    url = f"{API_ROOT}/v3/crud/journal/search"
+    headers = {"X-Aether-API-Key": API_KEY}
+    params = {"jwt": token}
+    
+    # site-scoped JWT should NOT be able to search global journals
+    resp = requests.post(url, headers=headers, params=params, json={"q": "%"})
+    print_result("Access Blocked (site-jwt -> journal)", resp.status_code == 403)
+
+def test_machine_auth_exception():
+    """Tests that restricted routes fail without API Key."""
+    print("\n--- Testing Machine Auth Exceptions ---")
+    url = f"{API_ROOT}/v3/crud/journal/search"
+    # No headers, no key
+    resp = requests.post(url, json={"q": "%"})
+    print_result("Unauth block (journal)", resp.status_code == 403)
+
+if __name__ == "__main__":
+    print(f"Starting Consolidated Auth & Security E2E Suite")
+    
+    test_site_bootstrap()
+    token = test_passcode_to_jwt()
+    if token:
+        test_security_boundaries(token)
+    test_machine_auth_exception()
+    
+    print("\nSuite completed.")

tests/e2e/test_e2e_v3_event_vision_parity.py

@@ -0,0 +1,70 @@
+import requests
+import json
+
+# --- Configuration ---
+BASE_URL = "https://dev-api.oneskyit.com/v3/crud"
+# Standard Agent API Key
+API_KEY = "PMM4n50teUCaOMMTN8qOJA"
+
+# Test Targets: (Object Type, Valid ID Random)
+TARGETS = [
+    ("event_device", "GZvFjgIIZQg"),
+    ("event_session", "F0PZd1bNcuD")
+]
+
+def get_headers():
+    return {
+        "Content-Type": "application/json",
+        "X-Aether-API-Key": API_KEY,
+        "x-no-account-id": "bypass"
+    }
+
+def verify_vision_parity(obj_type, record_id):
+    """
+    Verifies that the object returns ONLY string IDs for all ID fields (Vision Standard).
+    """
+    print(f"--- Testing {obj_type}: {record_id} ---")
+    url = f"{BASE_URL}/{obj_type}/{record_id}"
+    
+    try:
+        response = requests.get(url, headers=get_headers())
+        print(f"  Status: {response.status_code}")
+        
+        if response.status_code == 200:
+            data = response.json().get('data', {})
+            failures = []
+            
+            # Check all fields ending in _id (except external_id)
+            for key, val in data.items():
+                if key == "id" or (key.endswith("_id") and not key.endswith("external_id")):
+                    if val is not None and not isinstance(val, str):
+                        failures.append(f"{key} is {type(val).__name__} ({val})")
+            
+            if not failures:
+                print(f"  ✅ Success: All ID fields are strings.")
+                return True
+            else:
+                print(f"  ❌ Failure: Integer leakage detected:")
+                for f in failures:
+                    print(f"    - {f}")
+                return False
+        else:
+            print(f"  ❌ Error: {response.text[:200]}")
+            return False
+            
+    except Exception as e:
+        print(f"  💥 Exception: {e}")
+        return False
+
+if __name__ == "__main__":
+    print(f"Starting Aether V3 Event Vision Parity Tests\n")
+    
+    results = []
+    for obj_type, record_id in TARGETS:
+        results.append(verify_vision_parity(obj_type, record_id))
+        print("-" * 40)
+
+    if all(results):
+        print("\n🎉 ALL VISION PARITY TESTS PASSED!")
+    else:
+        print("\n❌ SOME TESTS FAILED.")

tests/e2e/test_e2e_v3_search_engine.py

@@ -0,0 +1,80 @@
+import requests
+import json
+import time
+
+# --- Configuration ---
+API_BASE = "https://dev-api.oneskyit.com/v3/crud"
+API_KEY = "PMM4n50teUCaOMMTN8qOJA"
+ACCOUNT_ID = "nqOzejLCDXM" # Standard Test Account
+
+def get_headers(no_account=False):
+    headers = {
+        "Content-Type": "application/json",
+        "X-Aether-API-Key": API_KEY
+    }
+    if no_account:
+        headers["x-no-account-id"] = "bypass"
+    else:
+        headers["x-account-id"] = ACCOUNT_ID
+    return headers
+
+def print_result(label, success, message=""):
+    status = "✅ PASS" if success else "❌ FAIL"
+    print(f"[{status}] {label} {message}")
+
+def test_basic_operators():
+    """Tests contains, startswith, endswith logic."""
+    print("\n--- Testing Basic Search Operators ---")
+    query = {"and": [{"field": "name", "op": "contains", "value": "Journal"}]}
+    resp = requests.post(f"{API_BASE}/journal/search", headers=get_headers(), json=query)
+    print_result("Operator: contains", resp.status_code == 200)
+
+    query = {"and": [{"field": "name", "op": "startswith", "value": "A"}]}
+    resp = requests.post(f"{API_BASE}/journal/search", headers=get_headers(), json=query)
+    print_result("Operator: startswith", resp.status_code == 200)
+
+def test_registry_fields():
+    """Tests searching by newly added registry fields (created_on, id_random)."""
+    print("\n--- Testing Registry-Expanded Fields ---")
+    query = {"and_filters": [{"field": "created_on", "op": "gt", "value": "2020-01-01"}]}
+    resp = requests.post(f"{API_BASE}/journal/search", headers=get_headers(), json=query)
+    print_result("Field: created_on", resp.status_code == 200)
+
+    # Get a valid ID for exact match test
+    res = requests.get(f"{API_BASE}/journal/", headers=get_headers(), params={"limit": 1})
+    if res.status_code == 200 and res.json().get("data"):
+        valid_id = res.json()["data"][0]["id"]
+        query = {"and_filters": [{"field": "id_random", "op": "eq", "value": valid_id}]}
+        resp = requests.post(f"{API_BASE}/journal/search", headers=get_headers(), json=query)
+        print_result(f"Field: id_random ({valid_id})", resp.status_code == 200)
+
+def test_nested_search():
+    """Tests POST /search on child objects."""
+    print("\n--- Testing Nested Advanced Search ---")
+    parent_id = "--ghJX-ztEM" # Valid person
+    url = f"{API_BASE}/person/{parent_id}/journal/search"
+    query = {"and_filters": [{"field": "name", "op": "like", "value": "%"}]}
+    resp = requests.post(url, headers=get_headers(), json=query)
+    print_result("Nested Search (person -> journal)", resp.status_code == 200)
+
+def test_extra_filters():
+    """Tests enabled=all and hidden=all bypass filters."""
+    print("\n--- Testing Extra Filters (enabled/hidden) ---")
+    # Using User object as it often has disabled records
+    resp = requests.get(f"{API_BASE}/user/?enabled=all&hidden=all", headers=get_headers())
+    print_result("Bypass Filters (enabled=all)", resp.status_code == 200)
+
+if __name__ == "__main__":
+    print(f"Starting Consolidated Search Engine E2E Suite")
+    print(f"Target: {API_BASE}")
+    
+    start_time = time.time()
+    try:
+        test_basic_operators()
+        test_registry_fields()
+        test_nested_search()
+        test_extra_filters()
+    except Exception as e:
+        print(f"💥 Suite Error: {e}")
+    
+    print(f"\nSuite completed in {time.time() - start_time:.2f}s")