Docs: Modernize main README, archive legacy/deprecated guides, and mark completed security/project docs (March 2026 review)
This commit is contained in:
183
README.md
183
README.md
@@ -1,67 +1,96 @@
|
|||||||
# Aether API v3.00.x (FastAPI)
|
|
||||||
|
|
||||||
The **Aether API** is a high-performance, multi-tenant backend infrastructure built using the Python **FastAPI** framework. It serves as the central data and logic hub for the Aether Platform, supporting both legacy applications and modern V3/V4 standards.
|
# Aether API v3.x (FastAPI)
|
||||||
|
|
||||||
|
The **Aether API** is a high-performance, multi-tenant backend for the Aether Platform, built on Python **FastAPI**. It powers both legacy and modern (V3/V4) applications, and is now fully containerized for robust, scalable deployment.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🏗️ Architecture Overview
|
## 🏗️ Architecture Overview
|
||||||
|
|
||||||
The API is currently in a transitional state between legacy (V1/V2) patterns and the modern **V3 CRUD Architecture**.
|
The API is in transition from legacy (V1/V2) to the modern **V3 CRUD Architecture**. All new development follows V3 standards.
|
||||||
|
|
||||||
### **V3 CRUD (Modern)**
|
|
||||||
|
### V3 CRUD (Modern)
|
||||||
- **Path:** `/v3/crud/`
|
- **Path:** `/v3/crud/`
|
||||||
- **Core Principles:**
|
- **Principles:**
|
||||||
- **`id_random` Primary:** All public communication uses URL-safe string IDs. Internal integer IDs are hidden.
|
- **String IDs:** All public APIs use `id_random` (URL-safe string IDs); internal integer IDs are hidden.
|
||||||
- **Nested URL Structure:** Enforces parent-child relationships (e.g., `/v3/crud/journal/{id}/entry/`).
|
- **Nested URLs:** Parent-child relationships enforced in URL structure.
|
||||||
- **Nested Advanced Search:** Full support for POST-based search on nested objects.
|
- **Advanced Search:** POST-based, recursive, with standardized operators.
|
||||||
- **Granular Dependencies:** Uses specialized FastAPI dependencies for Account Context, Pagination, Filtering, and Serialization.
|
- **Schema Discovery:** Dynamic model/database introspection at `/v3/crud/{obj_type}/schema`.
|
||||||
- **Advanced Search:** POST-based search with recursive logic and standardized operators.
|
- **Granular Dependencies:** Specialized FastAPI dependencies for account context, pagination, filtering, serialization.
|
||||||
- **Schema Discovery:** Dynamic introspection of database and Pydantic models via `/v3/crud/{obj_type}/schema`.
|
|
||||||
|
|
||||||
### **V3 Actions**
|
|
||||||
|
### V3 Actions
|
||||||
- **Path:** `/v3/action/`
|
- **Path:** `/v3/action/`
|
||||||
- Handles complex binary operations and atomic business logic separately from standard metadata CRUD.
|
- Handles complex/atomic business logic and binary operations outside standard CRUD.
|
||||||
- **Key Features:**
|
- **Features:**
|
||||||
- **Atomic Event Uploads:** Marriage of physical storage and complex event relations in one request.
|
- **Atomic Event Uploads:** File storage + event relations in one request.
|
||||||
- **Content-Addressable Downloads:** Direct file retrieval by SHA256 hash for high-performance local caching.
|
- **Content-Addressable Downloads:** SHA256-based file retrieval for high-performance caching.
|
||||||
- **Intelligent ID Resolution:** Standard download endpoints now automatically resolve container IDs (e.g., event_file) to underlying binaries.
|
- **Intelligent ID Resolution:** Download endpoints auto-resolve container IDs.
|
||||||
|
|
||||||
### **Legacy API (V1/V2)**
|
|
||||||
- **Path:** `/`, `/api/`, `/crud/`, `/v2/crud/`
|
### Legacy API (V1/V2)
|
||||||
- Maintained for backward compatibility but currently being systematically audited and deprecated.
|
- **Paths:** `/`, `/api/`, `/crud/`, `/v2/crud/`
|
||||||
- **Deprecation System:** Accessing legacy routes triggers a `!!! DEPRECATED ROUTE ACCESSED` warning in logs.
|
- Maintained for backward compatibility, but being systematically deprecated. Accessing legacy routes triggers a warning in logs.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
||||||
## 🛠️ Core Technologies
|
## 🛠️ Core Technologies
|
||||||
- **Framework:** FastAPI (v0.95.1)
|
- **Framework:** FastAPI (v0.95.1+)
|
||||||
- **Database:** MariaDB (Remote Master) + SQLAlchemy (v1.4.52)
|
- **Database:** MariaDB (Docker, shared) + SQLAlchemy (v1.4.52)
|
||||||
- **Caching/ID Resolution:** Redis
|
- **Caching/ID Resolution:** Redis
|
||||||
- **Security:** JWT (JSON Web Tokens) + API Key Machine Authorization
|
- **Security:** JWT (JSON Web Tokens), API Key Machine Auth
|
||||||
- **Logging:** Structured logging with module-level isolation and rotation.
|
- **Logging:** Structured, module-level, with rotation
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🚀 Getting Started
|
|
||||||
|
|
||||||
### **Local Development**
|
## 🚀 Quick Start
|
||||||
1. **Environment:** Requires Python 3.9+.
|
|
||||||
2. **Setup:**
|
The Aether API is designed for containerized deployment as part of the unified Aether Docker environment. For full-stack orchestration, see the documentation in the `aether_container_env` project.
|
||||||
```bash
|
|
||||||
virtualenv environment
|
### Prerequisites
|
||||||
source environment/bin/activate
|
- Docker & Docker Compose (for containerized use)
|
||||||
pip install -r requirements.txt
|
- Python 3.9+ (for local-only development)
|
||||||
```
|
|
||||||
3. **Run:**
|
### Local Development (Optional)
|
||||||
```bash
|
You can run the API locally for debugging:
|
||||||
uvicorn app.main:app --host 0.0.0.0 --port 5005 --reload
|
```bash
|
||||||
```
|
virtualenv environment
|
||||||
- **Documentation:** [GUIDE__LOCAL_DEVELOPMENT.md](documentation/GUIDE__LOCAL_DEVELOPMENT.md)
|
source environment/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
uvicorn app.main:app --host 0.0.0.0 --port 5005 --reload
|
||||||
|
```
|
||||||
|
See [GUIDE__LOCAL_DEVELOPMENT.md](documentation/GUIDE__LOCAL_DEVELOPMENT.md) for details.
|
||||||
|
|
||||||
|
### Docker Usage
|
||||||
|
The API is run and managed via Docker Compose as part of the full Aether stack. Refer to the `aether_container_env` project for orchestration, environment setup, and advanced deployment instructions.
|
||||||
|
|
||||||
|
### Service Endpoints (Default Ports)
|
||||||
|
- **API Docs:** https://dev-api.oneskyit.com/docs
|
||||||
|
- **Frontend:** http://localhost:8888
|
||||||
|
- **phpMyAdmin:** http://localhost:8081 (if enabled)
|
||||||
|
- **Logs (Dozzle):** http://localhost:8881
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🗄️ Database & Backups
|
||||||
|
|
||||||
|
All database operations are managed via Docker scripts in `aether_container_env/`:
|
||||||
|
- **Backup:** `./backup_db.sh` (saves to `backups/`)
|
||||||
|
- **Restore:** `./restore_db.sh [backup_file.gz]`
|
||||||
|
- **Export:** `./export_db.sh` (conference-ready backup)
|
||||||
|
- **Automated Import:** Drop file in `backups/import/` and run `./check_and_import.sh`
|
||||||
|
|
||||||
|
See [GUIDE__DEPLOYMENT_MANUAL.md](documentation/GUIDE__DEPLOYMENT_MANUAL.md) for full deployment and backup/restore instructions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### **Deployment**
|
|
||||||
- The API is deployed via **Docker Compose** within the **Aether Docker Environment** (`aether_container_env`).
|
|
||||||
- **Configuration (Docker)**: All settings (Database, SMTP, Ports) are managed via the master `.env` file in the `aether_container_env/` directory. No local `.env` file is required in this repository.
|
|
||||||
- **Manual Deployment:** [GUIDE__DEPLOYMENT_MANUAL.md](documentation/GUIDE__DEPLOYMENT_MANUAL.md)
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -82,28 +111,66 @@ The API is currently in a transitional state between legacy (V1/V2) patterns and
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
||||||
## 🧪 Testing Suite
|
## 🧪 Testing Suite
|
||||||
The project maintains an exhaustive test suite under the `tests/` directory.
|
Tests are under `tests/`:
|
||||||
- **Unit Tests:** `tests/unit/` (Mocked logic).
|
- **Unit:** `tests/unit/` (mocked logic)
|
||||||
- **Integration Tests:** `tests/integration/` (Local DB/Redis connectivity).
|
- **Integration:** `tests/integration/` (DB/Redis connectivity)
|
||||||
- **E2E Tests:** `tests/e2e/` (Network-based API validation).
|
- **E2E:** `tests/e2e/` (API validation)
|
||||||
- **Documentation:** [tests/README.md](tests/README.md)
|
- **Docs:** [tests/README.md](tests/README.md)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🚧 Current Status & Work in Progress
|
|
||||||
|
|
||||||
### **Active Workstreams**
|
## 🚧 Status & Work in Progress
|
||||||
- **[Backend] API Deprecation:** Systematic pruning of orphaned routers and methods (ID: 111523094).
|
|
||||||
- **[ID Vision]:** Phase 2 complete. String-ID standardization extended to Page, Post, Person, Journal, Contact, and User models.
|
### Active Workstreams
|
||||||
- **[V3 Migration]:** Implementation of atomic event actions and hash-based retrieval for high-performance Launcher caching complete.
|
- **API Deprecation:** Pruning orphaned routers/methods
|
||||||
|
- **ID Vision:** String-ID standardization (Phase 2 complete)
|
||||||
|
- **V3 Migration:** Atomic event actions, hash-based file retrieval
|
||||||
|
|
||||||
|
### Known Issues
|
||||||
|
- **Badge Rendering:** Corrupted numeric `id` fields in `event_badge_template` can cause template load failures
|
||||||
|
- **Websockets:** Legacy modules need unification and stability improvements
|
||||||
|
- **Intermittent Timeouts:** Some E2E tests occasionally reproduce 403s/timeouts on nested GET calls
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
### **Known Bugs / Issues**
|
|
||||||
- **Badge Rendering:**Corrupted numeric `id` fields in `event_badge_template` table causing template load failures in Svelte 5 views.
|
|
||||||
- **Websockets:** Legacy `websockets.py` and `websockets_redis.py` require unification and stability improvements.
|
|
||||||
- **Intermittent Timeouts:** Some E2E tests occasionally reproduce 403s/Timeouts on nested GET calls (investigating).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📜 Release Snapshot
|
## 📜 Release Snapshot
|
||||||
Current Baseline: **`release/2026-01-28-v3_prod-snapshot`** (Stable v3.0.99).
|
Current Baseline: **`release/2026-01-28-v3_prod-snapshot`** (Stable v3.0.99)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔐 Security & Access
|
||||||
|
- **SSH Required:** All git operations now require SSH (Bitbucket app passwords deprecated June 2026). See your Gitea or Bitbucket account for adding SSH keys.
|
||||||
|
- **Never commit secrets:** `.env` and credentials are git-ignored.
|
||||||
|
- **JWT Key:** Ensure `AE_API_JWT_KEY` is unique and high-entropy in production.
|
||||||
|
- **.env precedence:** API uses `.env` credentials for core infra (SMTP/DB) over DB settings.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🧑💻 Management & Operations
|
||||||
|
- **Restart API:** `docker compose restart ae_api`
|
||||||
|
- **Restart Frontend:** `docker compose restart ae_app`
|
||||||
|
- **Rebuild everything:** `docker compose up -d --build`
|
||||||
|
- **Logs:** http://localhost:8881 (Dozzle)
|
||||||
|
- **phpMyAdmin:** http://localhost:8081 (if enabled)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🏠 Directory Map (Key Mounts)
|
||||||
|
- `conf/` — Nginx/Gunicorn config templates
|
||||||
|
- `logs/` — Centralized logs
|
||||||
|
- `srv/` — Data/source code mounts
|
||||||
|
- `scripts/` — Automation scripts
|
||||||
|
- `backups/` — MariaDB snapshots
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 Notes
|
||||||
|
- For multi-stack setups, ensure unique `AE_NETWORK_NAME` and `CONTAINER_` prefixes in `.env`.
|
||||||
|
- All stacks must connect to `aether_shared_net` for shared DB/Redis.
|
||||||
|
- See Docker env README and CHEATSHEET for advanced orchestration and troubleshooting.
|
||||||
158
documentation/archive/GUIDE__DEPLOYMENT_MANUAL.md
Normal file
158
documentation/archive/GUIDE__DEPLOYMENT_MANUAL.md
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
# DEPRECATED: Manual Server Deployment Guide (Non-Docker)
|
||||||
|
|
||||||
|
> **Notice (March 2026):**
|
||||||
|
> This manual deployment guide is deprecated. The standard and supported method for deploying the Aether API is now via Docker Compose, as described in the main README and the `aether_container_env` documentation. Use this guide only for legacy or advanced manual setups.
|
||||||
|
|
||||||
|
# Manual Server Deployment Guide (Non-Docker)
|
||||||
|
|
||||||
|
This guide describes the manual process for deploying the Aether API on a Linux server using Nginx, Gunicorn, and Systemd.
|
||||||
|
|
||||||
|
## 1. Initial Server Setup
|
||||||
|
|
||||||
|
### Clone the Repository
|
||||||
|
```bash
|
||||||
|
sudo git clone https://scott_idem@bitbucket.org/oneskyit/one-sky-it-api-fastapi.git /srv/http/dev_fastapi.oneskyit.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configure Permissions
|
||||||
|
```bash
|
||||||
|
sudo mkdir admin/log
|
||||||
|
sudo chown http:http -R /srv/http/dev_fastapi.oneskyit.com/
|
||||||
|
sudo chmod 775 -R /srv/http/dev_fastapi.oneskyit.com/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Environment Preparation
|
||||||
|
```bash
|
||||||
|
cd /srv/http/dev_fastapi.oneskyit.com/
|
||||||
|
git switch development
|
||||||
|
|
||||||
|
virtualenv environment
|
||||||
|
source environment/bin/activate
|
||||||
|
pip install -U -r admin/requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
## 2. Gunicorn Configuration (Systemd)
|
||||||
|
|
||||||
|
### Socket Configuration (`/etc/systemd/system/gunicorn.socket`)
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=gunicorn socket
|
||||||
|
|
||||||
|
[Socket]
|
||||||
|
ListenStream=/run/gunicorn.sock
|
||||||
|
User=http
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=sockets.target
|
||||||
|
```
|
||||||
|
|
||||||
|
### Service Configuration (`/etc/systemd/system/gunicorn.service`)
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=gunicorn daemon
|
||||||
|
Requires=gunicorn.socket
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=notify
|
||||||
|
User=root
|
||||||
|
Group=root
|
||||||
|
RuntimeDirectory=gunicorn
|
||||||
|
WorkingDirectory=/srv/http/dev_fastapi.oneskyit.com
|
||||||
|
Environment="PATH=/srv/http/dev_fastapi.oneskyit.com/environment/bin"
|
||||||
|
ExecStart=/srv/http/dev_fastapi.oneskyit.com/environment/bin/gunicorn \
|
||||||
|
--bind unix:/srv/http/dev_fastapi.oneskyit.com/gunicorn.sock \
|
||||||
|
-m 007 app.main:app \
|
||||||
|
--workers 4 \
|
||||||
|
-k uvicorn.workers.UvicornWorker \
|
||||||
|
--access-logfile admin/log/access.log \
|
||||||
|
--error-logfile admin/log/error.log \
|
||||||
|
--capture-output --keep-alive 5
|
||||||
|
|
||||||
|
ExecReload=/bin/kill -s HUP $MAINPID
|
||||||
|
KillMode=mixed
|
||||||
|
TimeoutStopSec=5
|
||||||
|
PrivateTmp=true
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
```
|
||||||
|
|
||||||
|
### Activation
|
||||||
|
```bash
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable gunicorn.socket
|
||||||
|
sudo systemctl start gunicorn.socket
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Nginx Configuration
|
||||||
|
|
||||||
|
Create a site configuration file at `/etc/nginx/sites-available/dev_fastapi.oneskyit.com`:
|
||||||
|
|
||||||
|
```nginx
|
||||||
|
server {
|
||||||
|
access_log /var/log/nginx/access_dev_fastapi.oneskyit.com.log;
|
||||||
|
|
||||||
|
listen 443 ssl;
|
||||||
|
listen [::]:443 ssl http2;
|
||||||
|
server_name dev-fastapi.oneskyit.com;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/oneskyit.com-0001/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/oneskyit.com-0001/privkey.pem;
|
||||||
|
|
||||||
|
client_max_body_size 4096M;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_set_header Host $http_host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
|
||||||
|
proxy_redirect off;
|
||||||
|
proxy_buffering off;
|
||||||
|
|
||||||
|
proxy_pass http://unix:/run/gunicorn.sock;
|
||||||
|
}
|
||||||
|
|
||||||
|
# WebSocket Support
|
||||||
|
location ~ ^/(ws|ws_redis) {
|
||||||
|
proxy_set_header Host $http_host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
proxy_redirect off;
|
||||||
|
proxy_buffering off;
|
||||||
|
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection "upgrade";
|
||||||
|
|
||||||
|
proxy_pass http://unix:/run/gunicorn.sock;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
listen [::]:80;
|
||||||
|
server_name dev-fastapi.oneskyit.com;
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Enable and restart Nginx:
|
||||||
|
```bash
|
||||||
|
sudo ln -s /etc/nginx/sites-available/dev_fastapi.oneskyit.com /etc/nginx/sites-enabled/
|
||||||
|
sudo systemctl restart nginx.service
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Troubleshooting
|
||||||
|
```bash
|
||||||
|
# Check status
|
||||||
|
sudo systemctl status gunicorn.socket
|
||||||
|
sudo systemctl status gunicorn.service
|
||||||
|
sudo systemctl status nginx.service
|
||||||
|
|
||||||
|
# List active units
|
||||||
|
systemctl list-units --type=service --state=running
|
||||||
|
```
|
||||||
61
documentation/archive/PLAN__SECURITY_HARDENING.md
Normal file
61
documentation/archive/PLAN__SECURITY_HARDENING.md
Normal file
@@ -0,0 +1,61 @@
|
|||||||
|
# Project: API Security Hardening (V3)
|
||||||
|
|
||||||
|
**Status:** Complete / Archived (Reviewed March 18, 2026)
|
||||||
|
**Date:** Jan 18, 2026
|
||||||
|
**Owner:** Scott / Aether API Team
|
||||||
|
|
||||||
|
> This plan was fully implemented and reviewed on March 18, 2026. All critical and high vulnerabilities described have been addressed in the current codebase. See dependencies in `app/routers/dependencies_v3.py` and JWT logic in `app/lib_jwt.py` for details.
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
This project aims to close a critical security vulnerability in the Aether API V3 dependencies where the `x_no_account_id` header allows unauthorized "Superuser/Bypass" access without validation. Additionally, it addresses the lack of cryptographic verification for JWTs in the V3 CRUD layer.
|
||||||
|
|
||||||
|
## 2. The Vulnerabilities
|
||||||
|
|
||||||
|
### A. The "Bypass Header" (Critical)
|
||||||
|
* **Issue:** The `get_account_context_optional` dependency in `app/routers/dependencies_v3.py` accepts a header `x_no_account_id`. If present, it sets `auth_method = 'bypass'` and grants full administrative privileges (`super=True`).
|
||||||
|
* **Note:** This header may be sent as `x_no_account_id` or `x-no-account-id`. FastAPI's `Header` logic should handle the conversion, but validation must be explicit.
|
||||||
|
* **Current State:** There is **no** validation of this header. Sending `x_no_account_id: anything` works.
|
||||||
|
* **Risk:** Total system compromise via simple header injection.
|
||||||
|
|
||||||
|
### B. JWT Verification (High)
|
||||||
|
* **Issue:** The system validates users by looking up the `x_account_id` string in the database (Redis/MariaDB).
|
||||||
|
* **Current State:** It does **not** cryptographically verify the JWT signature or expiration in the V3 dependency chain.
|
||||||
|
* **Risk:** Account impersonation if an Account ID is leaked.
|
||||||
|
|
||||||
|
## 3. Implementation Plan
|
||||||
|
|
||||||
|
### Phase 1: Secure the Bypass Header (Immediate)
|
||||||
|
**Goal:** Restrict "Bypass Mode" to clients possessing a valid, active API Key.
|
||||||
|
|
||||||
|
1. **Refactor Dependencies:**
|
||||||
|
* Update `get_account_context_optional` to accept `x_aether_api_key`.
|
||||||
|
* **Logic Change:** If `x_no_account_id` header is detected:
|
||||||
|
* **Require** `x_aether_api_key`.
|
||||||
|
* **Lookup** key in `api_key` table.
|
||||||
|
* **Verify** `enable = 1` and `now()` is between `enable_from` and `enable_to`.
|
||||||
|
* **Failure Behavior:** If key is invalid/missing, log warning and deny 'bypass' status (fall back to 'guest').
|
||||||
|
|
||||||
|
2. **Verification:**
|
||||||
|
* Test `curl` with header only -> Expect 403/Forbidden.
|
||||||
|
* Test `curl` with header + valid Key -> Expect 200/OK.
|
||||||
|
* Test Frontend File Download (uses `x_no_account_id_token` param) -> Expect 200/OK (No regression).
|
||||||
|
|
||||||
|
### Phase 2: JWT Verification (Subsequent)
|
||||||
|
**Goal:** Replace DB-lookup auth with cryptographic JWT validation.
|
||||||
|
|
||||||
|
1. **Audit:** Confirm `app/lib_jwt.py` logic is sound.
|
||||||
|
2. **Middleware/Dependency:** Integrate `decode_jwt` into the `get_account_context` flow.
|
||||||
|
3. **Transition:** Allow dual-mode (DB lookup OR JWT) for a transition period if necessary, or cut over if frontend sends valid JWTs.
|
||||||
|
|
||||||
|
## 4. Impact Analysis
|
||||||
|
* **Frontend (SvelteKit):**
|
||||||
|
* Audit confirmed no usage of `x_no_account_id` **header** in active source code.
|
||||||
|
* Usage of `x_no_account_id_token` (Query Param) is **safe** and distinct from the header logic.
|
||||||
|
* **Scripts/External Tools:**
|
||||||
|
* Any external scripts using the "Bypass" header must be updated to send a valid API Key.
|
||||||
|
|
||||||
|
## 5. Action Items
|
||||||
|
- [x] Create `PROJECT_SECURITY_HARDENING.md` (This document).
|
||||||
|
- [x] Refactor `app/routers/dependencies_v3.py`.
|
||||||
|
- [x] Verify fix with `curl` tests.
|
||||||
|
- [x] Commit changes.
|
||||||
88
documentation/archive/PROJECT__Aether_API_Websockets_v3.md
Normal file
88
documentation/archive/PROJECT__Aether_API_Websockets_v3.md
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
# Project: Aether API WebSockets V3
|
||||||
|
|
||||||
|
## 1. Overview
|
||||||
|
The goal of WebSockets V3 is to provide a high-performance, scalable, and standardized real-time communication layer for the Aether Platform. This version focuses on efficient message routing using Redis granular Pub/Sub, integration with the **Vision ID** (string-based) pattern, and strict data validation via Pydantic.
|
||||||
|
|
||||||
|
The primary use case is **Group Coordination**: allowing a "controller" client to send commands or messages to one or more "worker" clients within the same group.
|
||||||
|
|
||||||
|
## 2. Analysis of Previous Versions
|
||||||
|
|
||||||
|
### V1: `websockets.py` (Memory-Based)
|
||||||
|
* **Mechanism**: Maintained a list of `WebSocket` objects in a Python list (`active_connections`).
|
||||||
|
* **Limitation**: Did not scale across multiple Docker containers. Clients on instance A could not communicate with clients on instance B.
|
||||||
|
* **Feature**: Basic support for `direct`, `group`, and `broadcast`.
|
||||||
|
|
||||||
|
### V2: `websockets_redis.py` (Global Pub/Sub)
|
||||||
|
* **Mechanism**: Uses `redis.asyncio` to publish all messages to a single `channel:ws`.
|
||||||
|
* **Limitation**: **"Noisy Neighbor" Problem**. Every API instance receives *every* message sent across the entire platform and must filter them in Python code (`if data.get('target') == 'group'`). This wastes CPU and network bandwidth at scale.
|
||||||
|
* **Feature**: Solved multi-instance connectivity.
|
||||||
|
|
||||||
|
## 3. V3 Architecture: Granular Pub/Sub
|
||||||
|
|
||||||
|
### Granular Redis Channels
|
||||||
|
V3 will move filtering from Python to Redis by using specific channel names. A client will subscribe only to the channels relevant to them:
|
||||||
|
1. **Client Channel**: `ws:client:{client_id_random}` (For Direct Messages)
|
||||||
|
2. **Group Channel**: `ws:group:{group_id_random}` (For Group Messages)
|
||||||
|
3. **Global Channel**: `ws:broadcast` (For System-wide Messages)
|
||||||
|
|
||||||
|
### Vision ID Integration
|
||||||
|
* All IDs in the WebSocket path and payload will be string-based `id_random` values.
|
||||||
|
* Path format: `/v3/ws/group/{group_id_random}/client/{client_id_random}`
|
||||||
|
|
||||||
|
### Standardized Message Schema
|
||||||
|
All V3 messages will follow a strict Pydantic model to ensure consistency between different device types.
|
||||||
|
|
||||||
|
```python
|
||||||
|
class WS_Message_V3(BaseModel):
|
||||||
|
version: str = "3"
|
||||||
|
msg_type: str # 'msg', 'cmd', 'heartbeat', 'presence'
|
||||||
|
target: str # 'direct', 'group', 'broadcast', 'echo'
|
||||||
|
from_id: str # client_id_random
|
||||||
|
to_id: Optional[str] # target client_id_random (for direct)
|
||||||
|
group_id: Optional[str] # target group_id_random (for group)
|
||||||
|
cmd: Optional[str] # Specific command string
|
||||||
|
msg: Optional[str] # Human-readable message
|
||||||
|
payload: Dict[str, Any] # Flexible JSON data
|
||||||
|
sent_at: datetime
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Backend Implementation Plan
|
||||||
|
|
||||||
|
### Phase 1: Library Layer (`app/lib_websockets_v3.py`)
|
||||||
|
* Define the `WS_Message_V3` model.
|
||||||
|
* Implement `WS_Manager_V3` to handle Redis connections and channel string generation.
|
||||||
|
* Add presence tracking using Redis Sets (`SADD` on connect, `SREM` on disconnect).
|
||||||
|
|
||||||
|
### Phase 2: Router Layer (`app/routers/websockets_v3.py`)
|
||||||
|
* Implement the `/v3/ws/...` endpoint.
|
||||||
|
* **Receiver Loop**: Receives JSON from client -> Validates -> Publishes to correct Redis channel.
|
||||||
|
* **Sender Loop**: Listens to multiple Redis channels -> Forwards messages to the client.
|
||||||
|
|
||||||
|
### Phase 3: Integration
|
||||||
|
* Register the router in `app/routers/registry.py`.
|
||||||
|
* Ensure legacy endpoints (`/ws/group/...`) remain functional in `websockets_redis.py`.
|
||||||
|
|
||||||
|
## 5. Frontend Integration & Changes
|
||||||
|
|
||||||
|
The frontend will need several updates to support the V3 protocol:
|
||||||
|
|
||||||
|
1. **Connection URL**: Update connection logic to use the `/v3/` prefix.
|
||||||
|
* *Old*: `ws://api.domain.com/ws/group/{id}/client/{id}`
|
||||||
|
* *New*: `ws://api.domain.com/v3/ws/group/{id}/client/{id}`
|
||||||
|
2. **Payload Wrapping**: All outgoing messages must be wrapped in the `WS_Message_V3` structure.
|
||||||
|
* Instead of sending raw text or simple JSON, send the structured object.
|
||||||
|
3. **Targeting Logic**:
|
||||||
|
* To send to the group, set `target: "group"`.
|
||||||
|
* To send to one specific device, set `target: "direct"` and provide `to_id`.
|
||||||
|
4. **Heartbeats**: The frontend should ideally send a `msg_type: "heartbeat"` every 30-60 seconds to keep the connection alive and update presence in Redis.
|
||||||
|
5. **Response Handling**: Incoming messages will now have a consistent shape, making it easier to route data to internal app state or components.
|
||||||
|
|
||||||
|
## 6. Security & Safety
|
||||||
|
* **API Key Verification**: WebSocket handshakes should optionally verify the `X-Aether-API-Key` during the upgrade request.
|
||||||
|
* **Isolation**: V3 will use its own Redis database or a strict prefixing strategy to ensure messages never bleed into legacy channels.
|
||||||
|
* **Error Handling**: Standardize the close codes (e.g., 4000 for invalid message schema).
|
||||||
|
|
||||||
|
## 7. Verification Plan
|
||||||
|
* Create `tests/e2e/test_e2e_v3_websockets.py`.
|
||||||
|
* Use `websockets` python library to simulate multiple concurrent clients.
|
||||||
|
* Test cross-instance communication (if possible in the test environment).
|
||||||
Reference in New Issue
Block a user