Scott.Idem/OSIT-AE-API-FastAPI
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: implement V3 Uniform Lookup System with hierarchical overrides and site-based whitelisting

Browse Source
This commit is contained in:
Scott Idem
2026-02-20 14:48:50 -05:00
parent 2b2a2bc00f
commit 6bfbff309a
8 changed files with 390 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
documentation/AGENT_TODO.md
View File

@@ -14,7 +14,7 @@
- [x] Audit File/Exhibit Models (File, Template, Tracking).
- [x] Whitelist `account_id` in all Event search definitions.
- [x] Audit Relational "Low-Priority" Models (Address, Contact, DataStore).
- [ ] Audit Lookup Fields (Exclude all `lu_*_id` integers from public output).
- [x] Audit Lookup Fields (Uniform V3 System Phase 1 Complete).
- [ ] Verify SQL Views join in all required `_random` IDs for performance.
- [ ] **Step 2:** Coordination (Verify Frontend uses `x-account-id` instead of token).

89
documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md
View File

@@ -1,5 +1,5 @@
# Project: V3 Uniform Lookup & Identity Agnostic Resolution
> **Status:** 📝 Planning (Drafted Feb 19, 2026)
> **Status:** 🏗️ Implementation (Phase 1 Complete - Feb 20, 2026)
> **Goal:** Standardize all `lu_*` tables into a hierarchical, identity-agnostic system supporting Global Defaults, Account Overrides, and Object Overrides.
## 1. Executive Summary
@@ -18,7 +18,7 @@ To ensure consistency across the ecosystem, all lookup tables will migrate towar
- `id_random`: Public String (ID Vision compliance).
- `account_id`: Account context (NULL = Global).
- `for_type` / `for_id`: Polymorphic Context (Optional).
- `group`: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). *Note: Replaces 'code' to avoid collision with industry standard codes.*
- `group`: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). *Note: Must be populated for hierarchy to work.*
- `name`: Primary display label.
- `description`: Detailed explanation.
- `enable`: Binary (1 = Active, 0 = Disabled). *Crucial for Negative Overrides.*
@@ -29,7 +29,7 @@ To ensure consistency across the ecosystem, all lookup tables will migrate towar
### 2.2 Hierarchical Selection (The "Ranked" Query)
To return a complete list where overrides replace defaults without duplication, we use **Window Functions** (`ROW_NUMBER() OVER`).
**SQL Pattern (Scenario: Account Context):**
**Finalized SQL Pattern:**
```sql
SELECT * FROM (
SELECT *,
@@ -37,80 +37,71 @@ SELECT * FROM (
PARTITION BY `group`
ORDER BY
(for_type = :for_type AND for_id = :for_id) DESC,
account_id DESC,
(account_id = :account_id) DESC,
created_on DESC
) as rank_priority
FROM `v_lu_example`
WHERE (
(for_type = :for_type AND for_id = :for_id)
OR account_id = :account_id
OR account_id IS NULL
)
) AS ranked_records
FROM `v_lu_v3_example`
WHERE (for_type = :for_type AND for_id = :for_id)
OR account_id = :account_id
OR account_id IS NULL
) AS ranked
WHERE rank_priority = 1
AND enable = 1 -- Negative Override enforcement
ORDER BY sort ASC, name ASC;
```
### 2.3 Identity Agnostic Resolution
To support external systems using different coding standards (e.g., ISO Numeric vs Alpha-2), the V3 API will implement an "Any-Match" resolver.
To support external systems using different coding standards, the V3 API implements a multi-field "Any-Match" resolver.
**Resolution Logic:**
- Accepts a query string `q`.
- Scans all "Identity" fields (e.g., `alpha_2_code`, `alpha_3_code`, `numeric_code`, `group`).
- Returns the highest-priority hierarchical match found across the "Identity" fields.
- Iterates through `searchable_fields` defined in the lookup registry.
- Returns the highest-priority hierarchical match.
### 2.4 Negative Overrides (The "Taiwan" Problem)
To "delete" or hide a global default for a specific account, the account creates an override record with `enable = 0`. Because the V3 query filters by `enable = 1` **after** the hierarchical ranking, the disabled override "shadows" the default, effectively removing the item from that account's view.
### 2.5 Policy-Driven Whitelisting (The "Timezone" Problem)
For tables with high-density global data (e.g., Timezones, Languages), accounts can implement an **Opt-In Policy** to avoid the bloat of managing hundreds of negative overrides.
- **Mechanism:** A configuration record (e.g., `data_store` JSON or `account_cfg` flag) defines a list of "Active Groups".
- **API Logic:** If a whitelist is defined for the account, the API injects an `AND group IN (:whitelist)` filter into the ranked query.
- **Outcome:** A client can curate a list of 5 active timezones out of 400 without creating manual shadow records for the remaining 395.
### 2.4 Negative Overrides (The "Shadowing" Pattern)
To "delete" a global default for a specific account, the account creates an override record with the same `group` but `enable = 0`. The ranking query assigns `rank_priority = 1` to the account record, and the outer `WHERE enable = 1` then excludes it, effectively hiding the item for that account.
---
## 3. Prioritization Examples
*Context: User in Account 5 (OSIT) viewing Event 99. Searching for "US".*
## 3. Implementation Standards
| Priority | Group | Account ID | For Type | For ID | Name | Outcome |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **1 (Winner)** | **US** | 5 | `event` | 99 | **US of America** | **Object Override** wins. |
| 2 (Shadowed)| US | 5 | NULL | NULL | USA | Account preference ignored. |
| 3 (Shadowed)| US | NULL | NULL | NULL | United States | Global default ignored. |
| 4 (Ignored) | FR | NULL | NULL | NULL | France | Different group. |
| 5 (Removed) | TW | 5 | NULL | NULL | Taiwan | `enable=0` makes this vanish. |
### 3.1 Normalization Rules
- **Underscore Naming:** All database columns must use underscores (e.g., `alpha_3_code`) instead of hyphens to ensure clean Python attribute access.
- **Group Population:** The `group` field must be physically populated in the table with a stable identity string (ISO codes, standard names).
- **Physical Name Field:** While specialized fields (like `english_short_name`) may exist, a physical `name` field is required for Generic CRUD compatibility.
### 3.2 View Requirements
- **Standard Joins:** All `v_lu_v3_*` views must join the `account` table to provide `account_id_random`.
- **Hybrid Naming:** (Proposed) Views may use `COALESCE` to provide a default `name` from specialized fields if the physical `name` column is empty.
---
## 4. Implementation Roadmap
### Phase 1: Normalization & Views
- [ ] Audit all `lu_*` tables for current schema.
- [ ] Create/Update `v_lu_*` views to map legacy columns to the uniform `group` structure.
- [ ] Ensure `id_random` is available for all lookups.
### Phase 1: Normalization & Infrastructure
- [x] Audit first batch of `lu_*` tables.
- [x] Establish `Lookup_Base` Pydantic models.
- [x] Implement hierarchical `get_lookup_list_v3` logic.
- [x] Register `/v3/lookup/` router.
### Phase 2: V3 Lookup Router
- [ ] Create `app/routers/lookup_v3.py`.
- [ ] Implement `GET /v3/lookup/{lu_type}/list` with Rank/Override logic.
- [ ] Implement **Whitelist Support** via account context policies.
- [ ] Implement `GET /v3/lookup/{lu_type}/resolve?q=VALUE`.
### Phase 2: Migration (Batch 1: High Priority)
- [x] `lu_v3_country` (Group: `alpha_2_code`)
- [x] `lu_v3_country_subdivision` (Group: `code`)
- [x] `lu_v3_time_zone` (Group: `name`)
### Phase 3: Migration (Batch 1: High Priority)
- [ ] `lu_country` (Complexity: Multi-Identity codes)
- [ ] `lu_country_subdivision`
- [ ] `lu_time_zone` (Complexity: Requires Whitelist Phase 2)
### Phase 4: Migration (Batch 2: Contextual)
### Phase 3: Migration (Batch 2: Contextual)
- [ ] `lu_post_topic`
- [ ] `lu_user_status`
- [ ] `lu_file_purpose`
### Phase 4: Advanced Features
- [ ] Implement **Whitelist Policies** via `data_store` JSON.
- [ ] Implement Batch Update tools for managers.
## 5. Key Learnings & Decisions
- **Deduplication:** The `PARTITION BY` strategy ensures the frontend never receives two records for the same logical lookup item.
- **Fail-Closed:** Account isolation must be maintained; a user should never see another account's overrides.
- **Group vs Code:** Using `group` as the internal Aether cluster key avoids naming collisions with industry standard codes (ISO, etc).
- **Deduplication:** The `PARTITION BY group` is the cornerstone of the system. Without a populated `group` field, the hierarchy collapses.
- **Generic CRUD Compatibility:** We decided to maintain a physical `name` field to allow the Aether V3 UI to handle all lookups with zero custom code.
- **Fail-Closed:** Account isolation is enforced at the query level via the `account_id` filter.
---
*Created by Gemini CLI for Scott Idem*