OSIT-AE-API-FastAPI/documentation/V3_HOSTED_FILE_MIGRATION_PLAN.md

Aether V3: Hosted File System Migration Plan

1. Overview

The goal of this project is to migrate the existing hosted_file and hosted_file_link logic into the CRUD V3 Architecture. This involves splitting the system into a Standard Record Layer (metadata) and a Specialized Action Layer (binary handling).

2. Core Requirements

• Relational Integrity: Fully utilize hosted_file_link for all object associations.
• Deduplication: Automatic filesystem and DB hash-checks before creating new records.
• Cleanup: Intelligent "Orphan" removal logic via rm_orphan flag.
• Flexible Auth: Support uploads/downloads with JWT, without JWT (Guest), and via URL-key fallback (bypass API Key requirement).
• Binary Support: High-performance streaming, byte-range seeking, and multi-file POST handling.
• Developer DX: Integrated delay_ms simulation and extension whitelisting.

---

3. Implementation Phases (Bite-Sized Chunks)

Phase 1: V3 Metadata Baseline

Status: Ready to start

• Whitelist hosted_file and hosted_file_link in obj_type_kv_li.
• Verify standard V3 Search works for files (filtering by account, hash, etc.).
• Enable PATCH /v3/crud/hosted_file/{id} for metadata updates (title, description).
• Implement "Fake Delete" using standard DELETE ...?method=hide.

Phase 2: V3 Action Router Scaffolding

• Create app/routers/api_v3_actions_hosted_file.py.
• Implement delay_ms middleware/logic for action routes.
• Implement specialized Extension Validator.

Phase 3: Enhanced Binary Actions

• Download Action: Port the streamer logic to /v3/action/hosted_file/{id}/download.
  • Add URL-param fallback for API Key/Auth bypass.
• Upload Action: Implement /v3/action/hosted_file/upload.
  • Support both single and List[UploadFile].
  • Implement the Hash-Lookup-Before-Write logic.

Phase 4: Relational Cleanup & Linking

• Relational Delete Logic:
  • Implement DELETE /v3/action/hosted_file/{id}.
  • Support method parameter: hide, disable, delete (hard).
  • Orphan Check: Logic to count remaining links; if rm_orphan=True and count is 0, physically remove file and parent record.
• Fake Delete (Test Mode):
  • Specialized mode for testing frontend workflows without data loss.
  • Logic:
    1. Verify hosted_file record existence.
    2. Verify physical file existence on server.
    3. Verify hosted_file_link existence.
    4. Return 200 OK success response without executing the actual SQL DELETE or os.unlink.

---

4. Technical Architecture

Standard CRUD Routes (Metadata)

Method	Endpoint	Description
POST	/v3/crud/hosted_file/search	Find files by hash, name, or account.
PATCH	/v3/crud/hosted_file/{id}	Update title, description, or notes.
DELETE	/v3/crud/hosted_file/{id}	Soft-delete (Hide) the file record.

Specialized Action Routes (Binary)

Method	Endpoint	Description
POST	/v3/action/hosted_file/upload	Upload 1+ files; handles deduplication.
GET	/v3/action/hosted_file/{id}/download	Stream binary data; supports range seeking.
DELETE	/v3/action/hosted_file/{id}	Removes link; optionally deletes orphan file.

---

5. Testing & Verification Strategy

For every chunk, we will create/update:

1. Logic Test: Unit test for the internal method (e.g., lookup_file_hash).
2. E2E Test: Live network test against the dev API to verify real record creation and file persistence.
3. Security Test: Verification of the "Bypass" modes (Site Key / URL Key).