diff --git a/documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md b/documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md new file mode 100644 index 0000000..4ba5b6a --- /dev/null +++ b/documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md @@ -0,0 +1,116 @@ +# Project: V3 Uniform Lookup & Identity Agnostic Resolution +> **Status:** 📝 Planning (Drafted Feb 19, 2026) +> **Goal:** Standardize all `lu_*` tables into a hierarchical, identity-agnostic system supporting Global Defaults, Account Overrides, and Object Overrides. + +## 1. Executive Summary +The Aether V3 Lookup System solves the problem of varied data standards (e.g., ISO-2, ISO-3, Numeric codes) and multi-tenant customization needs. It adopts a "Deduplicated Inheritance" pattern: +- **Global Defaults:** System-wide standard records (provided by Aether). +- **Account Overrides:** Client-specific labels, sorting, or disabled states. +- **Object Overrides:** Context-specific overrides (e.g., unique labels for a specific Event or Person). + +## 2. Theoretical Architecture + +### 2.1 Standardized Schema Components +To ensure consistency across the ecosystem, all lookup tables will migrate toward a baseline "V3 Minimal Structure." + +**Minimal Baseline Fields:** +- `id`: Internal Integer (Primary Key). +- `id_random`: Public String (ID Vision compliance). +- `account_id`: Account context (NULL = Global). +- `for_type` / `for_id`: Polymorphic Context (Optional). +- `group`: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). *Note: Replaces 'code' to avoid collision with industry standard codes.* +- `name`: Primary display label. +- `description`: Detailed explanation. +- `enable`: Binary (1 = Active, 0 = Disabled). *Crucial for Negative Overrides.* +- `hide`: UI Visibility flag. +- `sort`: Ordering priority. +- `created_on` / `updated_on`: Audit timestamps. + +### 2.2 Hierarchical Selection (The "Ranked" Query) +To return a complete list where overrides replace defaults without duplication, we use **Window Functions** (`ROW_NUMBER() OVER`). + +**SQL Pattern (Scenario: Account Context):** +```sql +SELECT * FROM ( + SELECT *, + ROW_NUMBER() OVER ( + PARTITION BY `group` + ORDER BY + (for_type = :for_type AND for_id = :for_id) DESC, + account_id DESC, + created_on DESC + ) as rank_priority + FROM `v_lu_example` + WHERE ( + (for_type = :for_type AND for_id = :for_id) + OR account_id = :account_id + OR account_id IS NULL + ) +) AS ranked_records +WHERE rank_priority = 1 + AND enable = 1 -- Negative Override enforcement +ORDER BY sort ASC, name ASC; +``` + +### 2.3 Identity Agnostic Resolution +To support external systems using different coding standards (e.g., ISO Numeric vs Alpha-2), the V3 API will implement an "Any-Match" resolver. + +**Resolution Logic:** +- Accepts a query string `q`. +- Scans all "Identity" fields (e.g., `alpha_2_code`, `alpha_3_code`, `numeric_code`, `group`). +- Returns the highest-priority hierarchical match found across the "Identity" fields. + +### 2.4 Negative Overrides (The "Taiwan" Problem) +To "delete" or hide a global default for a specific account, the account creates an override record with `enable = 0`. Because the V3 query filters by `enable = 1` **after** the hierarchical ranking, the disabled override "shadows" the default, effectively removing the item from that account's view. + +### 2.5 Policy-Driven Whitelisting (The "Timezone" Problem) +For tables with high-density global data (e.g., Timezones, Languages), accounts can implement an **Opt-In Policy** to avoid the bloat of managing hundreds of negative overrides. +- **Mechanism:** A configuration record (e.g., `data_store` JSON or `account_cfg` flag) defines a list of "Active Groups". +- **API Logic:** If a whitelist is defined for the account, the API injects an `AND group IN (:whitelist)` filter into the ranked query. +- **Outcome:** A client can curate a list of 5 active timezones out of 400 without creating manual shadow records for the remaining 395. + +--- + +## 3. Prioritization Examples +*Context: User in Account 5 (OSIT) viewing Event 99. Searching for "US".* + +| Priority | Group | Account ID | For Type | For ID | Name | Outcome | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| **1 (Winner)** | **US** | 5 | `event` | 99 | **US of America** | **Object Override** wins. | +| 2 (Shadowed)| US | 5 | NULL | NULL | USA | Account preference ignored. | +| 3 (Shadowed)| US | NULL | NULL | NULL | United States | Global default ignored. | +| 4 (Ignored) | FR | NULL | NULL | NULL | France | Different group. | +| 5 (Removed) | TW | 5 | NULL | NULL | Taiwan | `enable=0` makes this vanish. | + +--- + +## 4. Implementation Roadmap + +### Phase 1: Normalization & Views +- [ ] Audit all `lu_*` tables for current schema. +- [ ] Create/Update `v_lu_*` views to map legacy columns to the uniform `group` structure. +- [ ] Ensure `id_random` is available for all lookups. + +### Phase 2: V3 Lookup Router +- [ ] Create `app/routers/lookup_v3.py`. +- [ ] Implement `GET /v3/lookup/{lu_type}/list` with Rank/Override logic. +- [ ] Implement **Whitelist Support** via account context policies. +- [ ] Implement `GET /v3/lookup/{lu_type}/resolve?q=VALUE`. + +### Phase 3: Migration (Batch 1: High Priority) +- [ ] `lu_country` (Complexity: Multi-Identity codes) +- [ ] `lu_country_subdivision` +- [ ] `lu_time_zone` (Complexity: Requires Whitelist Phase 2) + +### Phase 4: Migration (Batch 2: Contextual) +- [ ] `lu_post_topic` +- [ ] `lu_user_status` +- [ ] `lu_file_purpose` + +## 5. Key Learnings & Decisions +- **Deduplication:** The `PARTITION BY` strategy ensures the frontend never receives two records for the same logical lookup item. +- **Fail-Closed:** Account isolation must be maintained; a user should never see another account's overrides. +- **Group vs Code:** Using `group` as the internal Aether cluster key avoids naming collisions with industry standard codes (ISO, etc). + +--- +*Created by Gemini CLI for Scott Idem*