# Model Registry V2 — Design Document

> Status: Planning / Pre-implementation
> Goal: Unified, provider-agnostic model management with clean role-based routing

---

## Problem Statement

The current system has two classes of models with different treatment:

| Type | How configured | How selected |
|---|---|---|
| Claude, Gemini | Hardcoded built-ins (`claude_cli`, `gemini_api`) | Backend toggle string ("claude"/"gemini") |
| Local (Ollama, Open WebUI) | Configured via `/settings/local` | Backend toggle string "local" |

This breaks down when you want:
- Multiple Gemini API keys (e.g. one per Google account)
- Claude via direct API key instead of OAuth CLI
- OpenRouter or other hosted providers alongside local models
- Role assignments to span all provider types uniformly
- A chat toggle that shows "which model" not "which service"

---

## Proposed Architecture

### Core concept: Providers + Credentials + Models + Roles

```
Providers (built-in, fixed set)
  └─ Anthropic       ← has a catalog of Claude model IDs
  └─ Google          ← has a catalog of Gemini model IDs
  └─ Local Host      ← OpenAI-compatible endpoint (user adds these)

Credentials (user-configured, per provider)
  └─ Anthropic       ← Claude CLI (OAuth, default) or API key
  └─ Google          ← one or more API keys (one per Google account)
  └─ Local Host      ← api_key stored on the host record (existing)

Model Entries (user-registered — "I want to use this model")
  └─ Provider + model ID + credential = one usable model entry
  └─ Same model ID with two different accounts = two model entries

Role Assignments (unified — any model entry can fill any role)
  └─ chat:        primary → backup_1 → backup_2
  └─ orchestrator: primary → backup_1
  └─ distill:     primary
  └─ (etc.)
```

### Backend toggle redesign

**Current:** cycles service type strings — `auto → claude → gemini → local`
**New:** cycles through the chat role's assigned models — `Primary → Backup 1 → Backup 2`

The toggle displays the active model's label (e.g. "Sonnet 4.6" / "Gemini 2.5 Flash" / "Gemma 4 E4B"). Auto defaults to Primary.

This means the toggle is context-free — it just picks a slot — and all the "what model, what provider, what credentials" logic lives in the registry.

---

## Data Model (V2 Schema)

Stored in `home/{user}/model_registry.json`.

```json
{
  "version": 2,

  "providers": {
    "anthropic": {
      "catalog": [
        {"id": "claude-opus-4-7",    "label": "Claude Opus 4.7",    "context_k": 200},
        {"id": "claude-sonnet-4-6",  "label": "Claude Sonnet 4.6",  "context_k": 200},
        {"id": "claude-haiku-4-5",   "label": "Claude Haiku 4.5",   "context_k": 200}
      ],
      "credentials": [
        {"id": "cli", "label": "Claude CLI (OAuth)", "type": "cli"}
      ]
    },
    "google": {
      "catalog": [
        {"id": "gemini-2.5-pro",   "label": "Gemini 2.5 Pro",   "context_k": 1000},
        {"id": "gemini-2.5-flash", "label": "Gemini 2.5 Flash", "context_k": 1000},
        {"id": "gemini-2.0-flash", "label": "Gemini 2.0 Flash", "context_k": 1000},
        {"id": "gemini-1.5-pro",   "label": "Gemini 1.5 Pro",   "context_k": 2000}
      ],
      "accounts": [
        {"id": "osit", "label": "One Sky IT (scott.idem@oneskyit.com)", "api_key": "AIza..."}
      ]
    }
  },

  "hosts": [
    {
      "id": "h1",
      "label": "Gaming Laptop",
      "api_url": "http://192.168.x.x:3000",
      "api_key": "",
      "host_type": "openwebui"
    }
  ],

  "models": [
    {
      "id": "m1",
      "label": "Sonnet 4.6 (CLI)",
      "type": "claude_cli",
      "provider": "anthropic",
      "model_name": "claude-sonnet-4-6",
      "credential_id": "cli",
      "context_k": 200,
      "tags": ["chat", "persona"]
    },
    {
      "id": "m2",
      "label": "Gemini 2.5 Flash (OSIT)",
      "type": "gemini_api",
      "provider": "google",
      "model_name": "gemini-2.5-flash",
      "account_id": "osit",
      "context_k": 1000,
      "tags": ["orchestrator", "research"]
    },
    {
      "id": "m3",
      "label": "Gemma 4 E4B",
      "type": "local_openai",
      "provider": "local",
      "host_id": "h1",
      "model_name": "gemma4:e4b",
      "context_k": 72,
      "tags": ["fast", "local"]
    }
  ],

  "roles": {
    "chat":        {"primary": "m1", "backup_1": "m2", "backup_2": "m3"},
    "orchestrator":{"primary": "m2", "backup_1": "m3"},
    "distill":     {"primary": "m1"}
  }
}
```

### Key differences from V1

| V1 | V2 |
|---|---|
| Built-ins (`claude_cli`, `gemini_api`) are hardcoded constants | All models are registry entries — built-ins become auto-populated defaults |
| Single Gemini API key in `auth.json` | `providers.google.accounts[]` — list of accounts |
| Role assignments only work with local models in UI | All models in all roles |
| Host list only for local | Host list stays for local; `providers` section for cloud |
| `type` field existed but only `local_openai` was user-configurable | `type` fully determines dispatch for all models |

---

## Resolution Logic (updated)

`get_model_for_role(username, role)` stays the same interface. Internally:

1. Walk `roles[role].primary → backup_1 → backup_2 → backup_3 → backup_4`
2. For each slot: resolve the model entry → merge in credentials
3. If no registry entry for a role: fall back to `.env` defaults, then hardcoded

`_resolve_model(registry, model_id)` gains new merge cases:
- `type == "claude_cli"` → merge in credential from `providers.anthropic.credentials`
- `type == "gemini_api"` → merge in `api_key` from `providers.google.accounts[account_id]`
- `type == "local_openai"` → merge host fields (existing logic, unchanged)

### Backend toggle → dispatch

```
UI sends: slot = "primary" | "backup_1" | "backup_2" | null (auto)
```

`llm_client.complete()` resolves the slot against the `chat` role, gets a full model config, dispatches by `type`. No more `"claude"/"gemini"/"local"` string matching.

---

## Routing Code Changes

### `llm_client.complete()`
- **Remove:** `model: str | None` → service type string
- **Add:** `slot: str | None = None` → role slot override ("primary"/"backup_1"/etc.)
- Dispatch table: `type` → handler
  - `claude_cli`   → `_claude()` (unchanged)
  - `claude_api`   → `_claude_api()` (new, direct Anthropic API — future phase)
  - `gemini_cli`   → `_gemini()` (unchanged)
  - `gemini_api`   → `_gemini_api()` (new, replaces current hardcoded gemini_api built-in)
  - `local_openai` → `_local()` (unchanged)

### `orchestrator_engine.py` / `openai_orchestrator.py`
- Get orchestrator model via `get_model_for_role(username, "orchestrator")`
- Already works — `openai_orchestrator.py` runs when type is `local_openai`
- `orchestrator_engine.py` (Gemini) runs when type is `gemini_api`

### Chat router (`routers/chat.py`)
- Accept `slot` instead of `model` from UI
- Pass to `llm_client.complete(slot=slot)`

---

## Settings UI Redesign

### New page structure

```
/settings/models     ← unified model registry (replaces /settings/local)
  Section 1: Cloud Providers
    Anthropic
      - credential: Claude CLI (OAuth) [default, always there]
      - + Add API Key (future)
      - model catalog [editable list of available Claude models]
    Google
      - accounts: [osit key ●●●●, + Add account]
      - model catalog [editable list of available Gemini models]
  Section 2: Local Hosts
    [existing host cards, unchanged]
  Section 3: Models  
    [unified list — all registered model entries across all providers]
    + Add Model (provider picker first, then model + credential/account dropdowns)

/settings/roles      ← standalone page (or promoted to /settings/models bottom)
  Role Assignments
    chat:         [primary ▾] [backup 1 ▾] [backup 2 ▾]
    orchestrator: [primary ▾] [backup 1 ▾]
    distill:      [primary ▾]
    (all dropdowns show all models from all providers)
```

### Backend toggle in chat UI

Replace the `claude → gemini → local → auto` cycle with:

```
[Model label] ▾ (clickable cycles through chat role slots)
```

- Shows the label of the currently active chat model
- Click cycles: Primary → Backup 1 → Backup 2 → Primary
- Slots with no model assigned are skipped
- Color: same purple/amber/slate theme, based on provider type (optional)

---

## Migration

V1 → V2 is handled in `_load()`:

1. Detect `version == 1` (or missing)
2. Synthesize `providers.anthropic` catalog from hardcoded defaults
3. Synthesize `providers.google` — migrate API key from `auth.json` as first account
4. Convert built-in role assignments (`claude_cli` / `gemini_api`) to new model entry IDs
5. Existing `hosts[]` and `local_openai` models carry over unchanged
6. Write `version: 2` and save

No data loss. Old `local_llm.json` migration path still works (V0 → V1 → V2).

---

## Phases

### Phase 1 — Data model + backend routing (no UI changes yet)
- Extend schema to V2 in `model_registry.py`
- Migration from V1 on first load
- Update `_resolve_model()` to handle `gemini_api` + account lookup
- Update `llm_client.complete()` to accept `slot` parameter
- Update `routers/chat.py` to pass `slot` instead of backend string
- Keep backend toggle UI working (map old strings to slots temporarily)
- **Deliverable:** routing works with multi-account Gemini, no UI changes needed yet

### Phase 2 — Cloud provider UI
- Add Anthropic and Google sections to `/settings/local` (rename to `/settings/models`)
- Google accounts: add/remove API keys with labels
- Editable model catalog for Anthropic + Google (add/remove model IDs from the list)
- Model entry creation: provider picker → model dropdown (from catalog) → account/credential picker
- **Deliverable:** can register cloud models in the UI just like local models

### Phase 3 — Unified role assignments + toggle redesign
- Promote role assignments to standalone `/settings/roles` page (or `/settings/models` bottom)
- All models from all providers appear in role selects
- Chat UI toggle: replace service-type cycle with slot cycle, show model label
- **Deliverable:** end-to-end unified experience

### Phase 4 — Polish + future providers
- Claude direct API key support (optional, CLI is fine for now)
- OpenRouter as a named provider (already works as a "local" host with host_type=openai — could be promoted)
- Model catalog sync: fetch available models from Anthropic/Google API if keys are present
- Per-role "test" button in role assignments UI

---

## Open Questions

1. **Claude direct API key:** Is this needed now, or is CLI OAuth sufficient for all users?
   - Decision: CLI-only for Phase 1; add API key support in Phase 4 if needed

2. **Catalog management:** Should the Anthropic/Google catalogs be server-wide defaults
   that users can extend, or fully per-user?
   - Recommendation: ship sensible defaults in code (updated with each deploy);
     users can add custom entries if needed

3. **Toggle UX:** Cycle through slot labels ("Primary / Backup 1 / Backup 2") or cycle
     through model labels ("Sonnet 4.6 / Gemini 2.5 Flash / Gemma 4")?
   - Model labels are more useful — clearer what you're switching to

4. **Orchestrator mode toggle:** Does agent mode also respect the slot toggle, or is it
     always "use orchestrator role"?
   - Keep orchestrator role separate; the UI toggle only affects `chat` role