Saving notes about new lookup tables.
This commit is contained in:
Scott Idem
116
documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md
Normal file
116
documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md
Normal file
@@ -0,0 +1,116 @@
|
||||
# Project: V3 Uniform Lookup & Identity Agnostic Resolution
|
||||
> **Status:** 📝 Planning (Drafted Feb 19, 2026)
|
||||
> **Goal:** Standardize all `lu_*` tables into a hierarchical, identity-agnostic system supporting Global Defaults, Account Overrides, and Object Overrides.
|
||||
|
||||
## 1. Executive Summary
|
||||
The Aether V3 Lookup System solves the problem of varied data standards (e.g., ISO-2, ISO-3, Numeric codes) and multi-tenant customization needs. It adopts a "Deduplicated Inheritance" pattern:
|
||||
- **Global Defaults:** System-wide standard records (provided by Aether).
|
||||
- **Account Overrides:** Client-specific labels, sorting, or disabled states.
|
||||
- **Object Overrides:** Context-specific overrides (e.g., unique labels for a specific Event or Person).
|
||||
|
||||
## 2. Theoretical Architecture
|
||||
|
||||
### 2.1 Standardized Schema Components
|
||||
To ensure consistency across the ecosystem, all lookup tables will migrate toward a baseline "V3 Minimal Structure."
|
||||
|
||||
**Minimal Baseline Fields:**
|
||||
- `id`: Internal Integer (Primary Key).
|
||||
- `id_random`: Public String (ID Vision compliance).
|
||||
- `account_id`: Account context (NULL = Global).
|
||||
- `for_type` / `for_id`: Polymorphic Context (Optional).
|
||||
- `group`: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). *Note: Replaces 'code' to avoid collision with industry standard codes.*
|
||||
- `name`: Primary display label.
|
||||
- `description`: Detailed explanation.
|
||||
- `enable`: Binary (1 = Active, 0 = Disabled). *Crucial for Negative Overrides.*
|
||||
- `hide`: UI Visibility flag.
|
||||
- `sort`: Ordering priority.
|
||||
- `created_on` / `updated_on`: Audit timestamps.
|
||||
|
||||
### 2.2 Hierarchical Selection (The "Ranked" Query)
|
||||
To return a complete list where overrides replace defaults without duplication, we use **Window Functions** (`ROW_NUMBER() OVER`).
|
||||
|
||||
**SQL Pattern (Scenario: Account Context):**
|
||||
```sql
|
||||
SELECT * FROM (
|
||||
SELECT *,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY `group`
|
||||
ORDER BY
|
||||
(for_type = :for_type AND for_id = :for_id) DESC,
|
||||
account_id DESC,
|
||||
created_on DESC
|
||||
) as rank_priority
|
||||
FROM `v_lu_example`
|
||||
WHERE (
|
||||
(for_type = :for_type AND for_id = :for_id)
|
||||
OR account_id = :account_id
|
||||
OR account_id IS NULL
|
||||
)
|
||||
) AS ranked_records
|
||||
WHERE rank_priority = 1
|
||||
AND enable = 1 -- Negative Override enforcement
|
||||
ORDER BY sort ASC, name ASC;
|
||||
```
|
||||
|
||||
### 2.3 Identity Agnostic Resolution
|
||||
To support external systems using different coding standards (e.g., ISO Numeric vs Alpha-2), the V3 API will implement an "Any-Match" resolver.
|
||||
|
||||
**Resolution Logic:**
|
||||
- Accepts a query string `q`.
|
||||
- Scans all "Identity" fields (e.g., `alpha_2_code`, `alpha_3_code`, `numeric_code`, `group`).
|
||||
- Returns the highest-priority hierarchical match found across the "Identity" fields.
|
||||
|
||||
### 2.4 Negative Overrides (The "Taiwan" Problem)
|
||||
To "delete" or hide a global default for a specific account, the account creates an override record with `enable = 0`. Because the V3 query filters by `enable = 1` **after** the hierarchical ranking, the disabled override "shadows" the default, effectively removing the item from that account's view.
|
||||
|
||||
### 2.5 Policy-Driven Whitelisting (The "Timezone" Problem)
|
||||
For tables with high-density global data (e.g., Timezones, Languages), accounts can implement an **Opt-In Policy** to avoid the bloat of managing hundreds of negative overrides.
|
||||
- **Mechanism:** A configuration record (e.g., `data_store` JSON or `account_cfg` flag) defines a list of "Active Groups".
|
||||
- **API Logic:** If a whitelist is defined for the account, the API injects an `AND group IN (:whitelist)` filter into the ranked query.
|
||||
- **Outcome:** A client can curate a list of 5 active timezones out of 400 without creating manual shadow records for the remaining 395.
|
||||
|
||||
---
|
||||
|
||||
## 3. Prioritization Examples
|
||||
*Context: User in Account 5 (OSIT) viewing Event 99. Searching for "US".*
|
||||
|
||||
| Priority | Group | Account ID | For Type | For ID | Name | Outcome |
|
||||
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
|
||||
| **1 (Winner)** | **US** | 5 | `event` | 99 | **US of America** | **Object Override** wins. |
|
||||
| 2 (Shadowed)| US | 5 | NULL | NULL | USA | Account preference ignored. |
|
||||
| 3 (Shadowed)| US | NULL | NULL | NULL | United States | Global default ignored. |
|
||||
| 4 (Ignored) | FR | NULL | NULL | NULL | France | Different group. |
|
||||
| 5 (Removed) | TW | 5 | NULL | NULL | Taiwan | `enable=0` makes this vanish. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Roadmap
|
||||
|
||||
### Phase 1: Normalization & Views
|
||||
- [ ] Audit all `lu_*` tables for current schema.
|
||||
- [ ] Create/Update `v_lu_*` views to map legacy columns to the uniform `group` structure.
|
||||
- [ ] Ensure `id_random` is available for all lookups.
|
||||
|
||||
### Phase 2: V3 Lookup Router
|
||||
- [ ] Create `app/routers/lookup_v3.py`.
|
||||
- [ ] Implement `GET /v3/lookup/{lu_type}/list` with Rank/Override logic.
|
||||
- [ ] Implement **Whitelist Support** via account context policies.
|
||||
- [ ] Implement `GET /v3/lookup/{lu_type}/resolve?q=VALUE`.
|
||||
|
||||
### Phase 3: Migration (Batch 1: High Priority)
|
||||
- [ ] `lu_country` (Complexity: Multi-Identity codes)
|
||||
- [ ] `lu_country_subdivision`
|
||||
- [ ] `lu_time_zone` (Complexity: Requires Whitelist Phase 2)
|
||||
|
||||
### Phase 4: Migration (Batch 2: Contextual)
|
||||
- [ ] `lu_post_topic`
|
||||
- [ ] `lu_user_status`
|
||||
- [ ] `lu_file_purpose`
|
||||
|
||||
## 5. Key Learnings & Decisions
|
||||
- **Deduplication:** The `PARTITION BY` strategy ensures the frontend never receives two records for the same logical lookup item.
|
||||
- **Fail-Closed:** Account isolation must be maintained; a user should never see another account's overrides.
|
||||
- **Group vs Code:** Using `group` as the internal Aether cluster key avoids naming collisions with industry standard codes (ISO, etc).
|
||||
|
||||
---
|
||||
*Created by Gemini CLI for Scott Idem*
|
||||
Reference in New Issue
Block a user