OSIT-AE-API-FastAPI/documentation/PROJECT__V3_UNIFORM_LOOKUP_SYSTEM.md

Project: V3 Uniform Lookup & Identity Agnostic Resolution

Status: 🏗️ Implementation (Phase 2 & Whitelist Complete - Feb 20, 2026) Goal: Standardize all lu_* tables into a hierarchical, identity-agnostic system supporting Global Defaults, Account Overrides, and Object Overrides.

1. Executive Summary

The Aether V3 Lookup System solves the problem of varied data standards (e.g., ISO-2, ISO-3, Numeric codes) and multi-tenant customization needs. It adopts a "Deduplicated Inheritance" pattern:

• Global Defaults: System-wide standard records (provided by Aether).
• Account Overrides: Client-specific labels, sorting, or disabled states.
• Object Overrides: Context-specific overrides (e.g., unique labels for a specific Event or Person).

2. Theoretical Architecture

2.1 Standardized Schema Components

To ensure consistency across the ecosystem, all lookup tables will migrate toward a baseline "V3 Minimal Structure."

Minimal Baseline Fields:

• id: Internal Integer (Primary Key).
• id_random: Public String (ID Vision compliance).
• account_id: Account context (NULL = Global).
• for_type / for_id: Polymorphic Context (Optional).
• group: The primary business key/cluster key (e.g., 'US', 'ACTIVE'). Note: Must be populated for hierarchy to work.
• name: Primary display label.
• description: Detailed explanation.
• enable: Binary (1 = Active, 0 = Disabled). Crucial for Negative Overrides.
• hide: UI Visibility flag.
• priority: Boolean/TinyInt (1 = High priority, 0 = Normal).
• sort: Ordering priority.
• created_on / updated_on: Audit timestamps.

2.2 Hierarchical Selection (The "Ranked" Query)

To return a complete list where overrides replace defaults without duplication, we use Window Functions (ROW_NUMBER() OVER).

Finalized SQL Pattern:

SELECT * FROM (
    SELECT *, 
           ROW_NUMBER() OVER (
               PARTITION BY `group` 
               ORDER BY 
                  (for_type = :for_type AND for_id = :for_id) DESC,
                  (account_id = :account_id) DESC, 
                  created_on DESC
           ) as rank_priority
    FROM `v_lu_v3_example`
    WHERE (for_type = :for_type AND for_id = :for_id)
       OR account_id = :account_id 
       OR account_id IS NULL
) AS ranked
WHERE rank_priority = 1
  AND enable = 1 -- Negative Override enforcement
ORDER BY sort ASC, name ASC;

2.3 Identity Agnostic Resolution

To support external systems using different coding standards, the V3 API implements a multi-field "Any-Match" resolver.

Resolution Logic:

• Accepts a query string q.
• Iterates through searchable_fields defined in the lookup registry.
• Returns the highest-priority hierarchical match.

2.4 Negative Overrides (The "Shadowing" Pattern)

To "delete" a global default for a specific account, the account creates an override record with the same group but enable = 0. The ranking query assigns rank_priority = 1 to the account record, and the outer WHERE enable = 1 then excludes it, effectively hiding the item for that account.

---

3. Implementation Standards

3.1 Normalization Rules

• Underscore Naming: All database columns must use underscores (e.g., alpha_3_code) instead of hyphens to ensure clean Python attribute access.
• Group Population: The group field must be physically populated in the table with a stable identity string (ISO codes, standard names).
• Physical Name Field: While specialized fields (like english_short_name) may exist, a physical name field is required for Generic CRUD compatibility.

3.2 View Requirements

• Standard Joins: All v_lu_v3_* views must join the account table to provide account_id_random.
• Hybrid Naming: Views may use COALESCE to provide a default name from specialized fields if the physical name column is empty.

3.3 Whitelist Policy (Phase 2)

To avoid manual management of hundreds of negative overrides, accounts can define an opt-in whitelist.

• Storage: site.cfg_json -> lookup_policy.
• Schema: { "lookup_policy": { "country": ["US", "CA"] } }
• Logic: When site_id is passed to the lookup API, results are filtered to only include groups present in the whitelist.

---

4. Implementation Roadmap

Phase 1: Normalization & Infrastructure

• Audit first batch of lu_* tables.
• Establish Lookup_Base Pydantic models.
• Implement hierarchical get_lookup_list_v3 logic.
• Register /v3/lookup/ router.

Phase 2: Migration (Batch 1: High Priority)

• lu_v3_country (Group: alpha_2_code)
• lu_v3_country_subdivision (Group: code)
• lu_v3_time_zone (Group: name)

Phase 3: Migration (Batch 2: Contextual) - ON HOLD FOR V3.1

• lu_post_topic
• lu_user_status
• lu_file_purpose

Phase 4: Advanced Features

• Implement Whitelist Policies via site.cfg_json.
• Implement Batch Update tools for managers.

5. Key Learnings & Decisions

• Deduplication: The PARTITION BY group is the cornerstone of the system. Without a populated group field, the hierarchy collapses.
• Generic CRUD Compatibility: We decided to maintain a physical name field to allow the Aether V3 UI to handle all lookups with zero custom code.
• Fail-Closed: Account isolation is enforced at the query level via the account_id filter.
• Type-Safe Auth: Discovered that load_site_obj returns Random IDs for accounts; updated router context comparison to use account_id_random strings for 403 authorization checks.

---

Created by Gemini CLI for Scott Idem