OSIT-AE-API-FastAPI/documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md

Project: V3 Uniform Lookup & Identity Agnostic Resolution

Status: 📝 Planning (Drafted Feb 19, 2026) Goal: Standardize all lu_* tables into a hierarchical, identity-agnostic system supporting Global Defaults, Account Overrides, and Object Overrides.

1. Executive Summary

The Aether V3 Lookup System solves the problem of varied data standards (e.g., ISO-2, ISO-3, Numeric codes) and multi-tenant customization needs. It adopts a "Deduplicated Inheritance" pattern:

• Global Defaults: System-wide standard records (provided by Aether).
• Account Overrides: Client-specific labels, sorting, or disabled states.
• Object Overrides: Context-specific overrides (e.g., unique labels for a specific Event or Person).

2. Theoretical Architecture

2.1 Standardized Schema Components

To ensure consistency across the ecosystem, all lookup tables will migrate toward a baseline "V3 Minimal Structure."

Minimal Baseline Fields:

• id: Internal Integer (Primary Key).
• id_random: Public String (ID Vision compliance).
• account_id: Account context (NULL = Global).
• for_type / for_id: Polymorphic Context (Optional).
• group: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). Note: Replaces 'code' to avoid collision with industry standard codes.
• name: Primary display label.
• description: Detailed explanation.
• enable: Binary (1 = Active, 0 = Disabled). Crucial for Negative Overrides.
• hide: UI Visibility flag.
• sort: Ordering priority.
• created_on / updated_on: Audit timestamps.

2.2 Hierarchical Selection (The "Ranked" Query)

To return a complete list where overrides replace defaults without duplication, we use Window Functions (ROW_NUMBER() OVER).

SQL Pattern (Scenario: Account Context):

SELECT * FROM (
    SELECT *, 
           ROW_NUMBER() OVER (
               PARTITION BY `group` 
               ORDER BY 
                  (for_type = :for_type AND for_id = :for_id) DESC,
                  account_id DESC, 
                  created_on DESC
           ) as rank_priority
    FROM `v_lu_example`
    WHERE (
        (for_type = :for_type AND for_id = :for_id)
        OR account_id = :account_id 
        OR account_id IS NULL
    )
) AS ranked_records
WHERE rank_priority = 1
  AND enable = 1 -- Negative Override enforcement
ORDER BY sort ASC, name ASC;

2.3 Identity Agnostic Resolution

To support external systems using different coding standards (e.g., ISO Numeric vs Alpha-2), the V3 API will implement an "Any-Match" resolver.

Resolution Logic:

• Accepts a query string q.
• Scans all "Identity" fields (e.g., alpha_2_code, alpha_3_code, numeric_code, group).
• Returns the highest-priority hierarchical match found across the "Identity" fields.

2.4 Negative Overrides (The "Taiwan" Problem)

To "delete" or hide a global default for a specific account, the account creates an override record with enable = 0. Because the V3 query filters by enable = 1 after the hierarchical ranking, the disabled override "shadows" the default, effectively removing the item from that account's view.

2.5 Policy-Driven Whitelisting (The "Timezone" Problem)

For tables with high-density global data (e.g., Timezones, Languages), accounts can implement an Opt-In Policy to avoid the bloat of managing hundreds of negative overrides.

• Mechanism: A configuration record (e.g., data_store JSON or account_cfg flag) defines a list of "Active Groups".
• API Logic: If a whitelist is defined for the account, the API injects an AND group IN (:whitelist) filter into the ranked query.
• Outcome: A client can curate a list of 5 active timezones out of 400 without creating manual shadow records for the remaining 395.

---

3. Prioritization Examples

Context: User in Account 5 (OSIT) viewing Event 99. Searching for "US".

Priority	Group	Account ID	For Type	For ID	Name	Outcome
1 (Winner)	US	5	event	99	US of America	Object Override wins.
2 (Shadowed)	US	5	NULL	NULL	USA	Account preference ignored.
3 (Shadowed)	US	NULL	NULL	NULL	United States	Global default ignored.
4 (Ignored)	FR	NULL	NULL	NULL	France	Different group.
5 (Removed)	TW	5	NULL	NULL	Taiwan	enable=0 makes this vanish.

---

4. Implementation Roadmap

Phase 1: Normalization & Views

• Audit all lu_* tables for current schema.
• Create/Update v_lu_* views to map legacy columns to the uniform group structure.
• Ensure id_random is available for all lookups.

Phase 2: V3 Lookup Router

• Create app/routers/lookup_v3.py.
• Implement GET /v3/lookup/{lu_type}/list with Rank/Override logic.
• Implement Whitelist Support via account context policies.
• Implement GET /v3/lookup/{lu_type}/resolve?q=VALUE.

Phase 3: Migration (Batch 1: High Priority)

• lu_country (Complexity: Multi-Identity codes)
• lu_country_subdivision
• lu_time_zone (Complexity: Requires Whitelist Phase 2)

Phase 4: Migration (Batch 2: Contextual)

• lu_post_topic
• lu_user_status
• lu_file_purpose

5. Key Learnings & Decisions

• Deduplication: The PARTITION BY strategy ensures the frontend never receives two records for the same logical lookup item.
• Fail-Closed: Account isolation must be maintained; a user should never see another account's overrides.
• Group vs Code: Using group as the internal Aether cluster key avoids naming collisions with industry standard codes (ISO, etc).

---

Created by Gemini CLI for Scott Idem