Scott.Idem/Cortex-Inara
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: model registry Phase 3 — slot-based backend toggle

Browse Source 
Backend toggle now cycles through chat role models by label instead of
cycling service type strings (auto/claude/gemini/local).

- model_registry: get_model_for_slot() — resolves a specific priority
  slot without walking the fallback chain
- llm_client: complete() gains slot param; explicit slot selection
  dispatches directly to that model with no silent fallback
- routers/chat.py: ChatRequest.slot; GET /backend returns chat_models
  [{slot, label, type}] for the UI; _stream_chat uses resolved model
  label for the response tag when a slot is pinned
- app.js: toggle loads chat_models from /backend, cycles by label,
  sends slot in chat payload; legacy model field removed from payload
- app.js: fix Gap B — agent mode placeholder no longer says "Gemini
  tool loop"; now says "orchestrator"
- DESIGN doc: updated to reflect phases 1+2 complete, catalog-as-code
  decision, Gap A/B documented, Phase 3 implementation details

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Scott Idem
2026-04-27 21:43:08 -04:00
parent 3bc6b45f9f
commit 962d58d2e2
5 changed files with 248 additions and 296 deletions
Download Patch File Download Diff File Expand all files Collapse all files

327
documentation/DESIGN__Model_Registry_V2.md
View File

@@ -1,62 +1,71 @@
# Model Registry V2 — Design Document
> Status: Planning / Pre-implementation
> Status: Phase 3 in progress
> Goal: Unified, provider-agnostic model management with clean role-based routing
---
## Problem Statement
The current system has two classes of models with different treatment:
The original system had two classes of models with different treatment:
| Type | How configured | How selected |
|---|---|---|
| Claude, Gemini | Hardcoded built-ins (`claude_cli`, `gemini_api`) | Backend toggle string ("claude"/"gemini") |
| Local (Ollama, Open WebUI) | Configured via `/settings/local` | Backend toggle string "local" |
This breaks down when you want:
- Multiple Gemini API keys (e.g. one per Google account)
- Claude via direct API key instead of OAuth CLI
- OpenRouter or other hosted providers alongside local models
- Role assignments to span all provider types uniformly
- A chat toggle that shows "which model" not "which service"
This breaks down when you want multiple Gemini API keys, OpenRouter alongside local models,
role assignments spanning all provider types, or a toggle that shows which model is active
instead of which service.
---
## Proposed Architecture
## Architecture
### Core concept: Providers + Credentials + Models + Roles
```
Providers (built-in, fixed set)
└─ Anthropic ← has a catalog of Claude model IDs
└─ Google ← has a catalog of Gemini model IDs
└─ Anthropic ← catalog of Claude model IDs (code constants)
└─ Google ← catalog of Gemini model IDs (code constants)
└─ Local Host ← OpenAI-compatible endpoint (user adds these)
Credentials (user-configured, per provider)
└─ Anthropic ← Claude CLI (OAuth, default) or API key
Credentials (user-configured, stored in model_registry.json)
└─ Anthropic ← Claude CLI (OAuth, default) — API key support in Phase 4
└─ Google ← one or more API keys (one per Google account)
└─ Local Host ← api_key stored on the host record (existing)
└─ Local Host ← api_key stored on the host record
Model Entries (user-registered — "I want to use this model")
Model Entries (user-registered)
└─ Provider + model ID + credential = one usable model entry
└─ Same model ID with two different accounts = two model entries
Role Assignments (unified — any model entry can fill any role)
└─ chat: primary → backup_1 → backup_2
└─ chat: primary → backup_1 → backup_2
└─ orchestrator: primary → backup_1
└─ distill: primary
└─ distill: primary
└─ (etc.)
```
### Backend toggle redesign
### Catalog design decision
**Current:** cycles service type strings — `auto → claude → gemini → local`
**New:** cycles through the chat role's assigned models — `Primary → Backup 1 → Backup 2`
Catalogs (`ANTHROPIC_CATALOG`, `GOOGLE_CATALOG`) are **Python constants** in
`model_registry.py`, not stored in the per-user JSON. Updated with each code deploy.
Per-user catalog customisation is deferred to Phase 4.
The toggle displays the active model's label (e.g. "Sonnet 4.6" / "Gemini 2.5 Flash" / "Gemma 4 E4B"). Auto defaults to Primary.
### Backend toggle redesign (Phase 3)
This means the toggle is context-free — it just picks a slot — and all the "what model, what provider, what credentials" logic lives in the registry.
**Before:** cycles service type strings — `auto → claude → gemini → local`
**After:** cycles through the chat role's configured models by label:
```
Sonnet 4.6 (CLI) → Gemini 2.5 Flash → Gemma 4 E4B → (wraps)
```
- Shows the resolved model label on the toggle button
- If no chat role models are configured: shows "auto", uses existing role routing
- Click skips empty slots automatically
- Color: `claude_cli` = default, `gemini_*` = blue, `local_openai` = amber
UI sends `slot: "primary" | "backup_1" | "backup_2"` (not backend type string).
`llm_client.complete()` resolves that slot from the chat role and dispatches by `type`.
---
@@ -67,74 +76,22 @@ Stored in `home/{user}/model_registry.json`.
```json
{
"version": 2,
"providers": {
"anthropic": {
"catalog": [
{"id": "claude-opus-4-7", "label": "Claude Opus 4.7", "context_k": 200},
{"id": "claude-sonnet-4-6", "label": "Claude Sonnet 4.6", "context_k": 200},
{"id": "claude-haiku-4-5", "label": "Claude Haiku 4.5", "context_k": 200}
],
"credentials": [
{"id": "cli", "label": "Claude CLI (OAuth)", "type": "cli"}
]
"credentials": [{"id": "cli", "label": "Claude CLI (OAuth)", "type": "cli"}]
},
"google": {
"catalog": [
{"id": "gemini-2.5-pro", "label": "Gemini 2.5 Pro", "context_k": 1000},
{"id": "gemini-2.5-flash", "label": "Gemini 2.5 Flash", "context_k": 1000},
{"id": "gemini-2.0-flash", "label": "Gemini 2.0 Flash", "context_k": 1000},
{"id": "gemini-1.5-pro", "label": "Gemini 1.5 Pro", "context_k": 2000}
],
"accounts": [
{"id": "osit", "label": "One Sky IT (scott.idem@oneskyit.com)", "api_key": "AIza..."}
]
"accounts": [{"id": "a1b2", "label": "One Sky IT", "api_key": "AIza..."}]
}
},
"hosts": [
{
"id": "h1",
"label": "Gaming Laptop",
"api_url": "http://192.168.x.x:3000",
"api_key": "",
"host_type": "openwebui"
}
{"id": "h1", "label": "Gaming Laptop", "api_url": "http://...", "api_key": "", "host_type": "openwebui"}
],
"models": [
{
"id": "m1",
"label": "Sonnet 4.6 (CLI)",
"type": "claude_cli",
"provider": "anthropic",
"model_name": "claude-sonnet-4-6",
"credential_id": "cli",
"context_k": 200,
"tags": ["chat", "persona"]
},
{
"id": "m2",
"label": "Gemini 2.5 Flash (OSIT)",
"type": "gemini_api",
"provider": "google",
"model_name": "gemini-2.5-flash",
"account_id": "osit",
"context_k": 1000,
"tags": ["orchestrator", "research"]
},
{
"id": "m3",
"label": "Gemma 4 E4B",
"type": "local_openai",
"provider": "local",
"host_id": "h1",
"model_name": "gemma4:e4b",
"context_k": 72,
"tags": ["fast", "local"]
}
{"id": "m1", "type": "claude_cli", "label": "Sonnet 4.6 (CLI)", "model_name": "claude-sonnet-4-6", "provider": "anthropic", "credential_id": "cli", "context_k": 1000, "tags": []},
{"id": "m2", "type": "gemini_api", "label": "Gemini 2.5 Flash", "model_name": "gemini-2.5-flash", "provider": "google", "account_id": "a1b2", "context_k": 1000, "tags": []},
{"id": "m3", "type": "local_openai", "label": "Gemma 4 E4B", "model_name": "gemma4:e4b", "provider": "local", "host_id": "h1", "context_k": 72, "tags": []}
],
"roles": {
"chat": {"primary": "m1", "backup_1": "m2", "backup_2": "m3"},
"orchestrator":{"primary": "m2", "backup_1": "m3"},
@@ -143,168 +100,100 @@ Stored in `home/{user}/model_registry.json`.
}
```
### Key differences from V1
### Model types and dispatch
| V1 | V2 |
|---|---|
| Built-ins (`claude_cli`, `gemini_api`) are hardcoded constants | All models are registry entries — built-ins become auto-populated defaults |
| Single Gemini API key in `auth.json` | `providers.google.accounts[]` — list of accounts |
| Role assignments only work with local models in UI | All models in all roles |
| Host list only for local | Host list stays for local; `providers` section for cloud |
| `type` field existed but only `local_openai` was user-configurable | `type` fully determines dispatch for all models |
| `type` | Dispatches via | Notes |
|---|---|---|
| `claude_cli` | Claude CLI subprocess | `~/.claude/.credentials.json` OAuth |
| `gemini_cli` | Gemini CLI subprocess | |
| `gemini_api` | Currently: Gemini CLI (gap — see Phase 4) | Should use google-genai SDK |
| `local_openai` | HTTP to OpenAI-compatible endpoint | host_type controls path |
### Built-in model IDs
Always resolvable without a registry entry (used as `.env` role defaults):
`claude_cli`, `gemini_cli`, `gemini_api`
---
## Resolution Logic (updated)
## Resolution Logic
`get_model_for_role(username, role)` stays the same interface. Internally:
`get_model_for_role(username, role)` — walks `primary → backup_1 → backup_2 → backup_3 → backup_4`, returns first resolved model config with credentials merged in. Falls back to `.env` defaults, then hardcoded last-resort.
1. Walk `roles[role].primary → backup_1 → backup_2 → backup_3 → backup_4`
2. For each slot: resolve the model entry → merge in credentials
3. If no registry entry for a role: fall back to `.env` defaults, then hardcoded
`_resolve_model(registry, model_id)` gains new merge cases:
- `type == "claude_cli"` → merge in credential from `providers.anthropic.credentials`
- `type == "gemini_api"` → merge in `api_key` from `providers.google.accounts[account_id]`
- `type == "local_openai"` → merge host fields (existing logic, unchanged)
### Backend toggle → dispatch
```
UI sends: slot = "primary" | "backup_1" | "backup_2" | null (auto)
```
`llm_client.complete()` resolves the slot against the `chat` role, gets a full model config, dispatches by `type`. No more `"claude"/"gemini"/"local"` string matching.
`get_model_for_slot(username, role, slot)` — resolves *only* the named slot, no fallback chain. Used by Phase 3 explicit slot selection.
---
## Routing Code Changes
## Routing Code
### `llm_client.complete()`
- **Remove:** `model: str | None` → service type string
- **Add:** `slot: str | None = None` → role slot override ("primary"/"backup_1"/etc.)
- Dispatch table: `type` → handler
- `claude_cli` → `_claude()` (unchanged)
- `claude_api` → `_claude_api()` (new, direct Anthropic API — future phase)
- `gemini_cli` → `_gemini()` (unchanged)
- `gemini_api` → `_gemini_api()` (new, replaces current hardcoded gemini_api built-in)
- `local_openai` → `_local()` (unchanged)
### `llm_client.complete()` (Phase 3 update)
### `orchestrator_engine.py` / `openai_orchestrator.py`
- Get orchestrator model via `get_model_for_role(username, "orchestrator")`
- Already works — `openai_orchestrator.py` runs when type is `local_openai`
- `orchestrator_engine.py` (Gemini) runs when type is `gemini_api`
```
slot: str | None → resolve specific slot, no fallback (explicit selection)
model: str | None → legacy backend strings, kept for backward compat
(neither) → auto: role-based routing with full fallback chain
```
### Chat router (`routers/chat.py`)
- Accept `slot` instead of `model` from UI
- Pass to `llm_client.complete(slot=slot)`
Dispatch table (`type` → backend function):
- `claude_cli` → `_claude()`
- `gemini_cli` → `_gemini()`
- `gemini_api` → `_gemini()` ← **gap: should be `_gemini_api()` (Phase 4)**
- `local_openai` → `_local()`
### `routers/chat.py` (Phase 3 update)
- `ChatRequest` gets `slot: str | None = None`
- `GET /backend` returns `chat_models: [{slot, label, type}]` for the UI toggle
- `_stream_chat` resolves model label from slot when `req.slot` is set
### `app.js` (Phase 3 update)
- Loads `chat_models` from `GET /backend` on page init
- Toggle cycles through `chat_models` by label, sends `slot` in chat payload
- Agent mode placeholder: remove "Gemini tool loop" hardcode → "orchestrator"
---
## Settings UI Redesign
## Known Gaps (not yet implemented)
### New page structure
### Gap A — `gemini_api` dispatch in `llm_client` (Phase 4)
`_TYPE_TO_BACKEND` maps `gemini_api → "gemini"` (CLI subprocess). If a user assigns a
`gemini_api` type model to the `chat` role, it silently routes to the Gemini CLI instead
of the Google genai SDK. Fix: add `_gemini_api()` in `llm_client.py` that calls the SDK
directly, matching how `orchestrator_engine.py` does it. Needs API key from resolved config.
```
/settings/models ← unified model registry (replaces /settings/local)
Section 1: Cloud Providers
Anthropic
- credential: Claude CLI (OAuth) [default, always there]
- + Add API Key (future)
- model catalog [editable list of available Claude models]
Google
- accounts: [osit key ●●●●, + Add account]
- model catalog [editable list of available Gemini models]
Section 2: Local Hosts
[existing host cards, unchanged]
Section 3: Models
[unified list — all registered model entries across all providers]
+ Add Model (provider picker first, then model + credential/account dropdowns)
/settings/roles ← standalone page (or promoted to /settings/models bottom)
Role Assignments
chat: [primary ▾] [backup 1 ▾] [backup 2 ▾]
orchestrator: [primary ▾] [backup 1 ▾]
distill: [primary ▾]
(all dropdowns show all models from all providers)
```
### Backend toggle in chat UI
Replace the `claude → gemini → local → auto` cycle with:
```
[Model label] ▾ (clickable cycles through chat role slots)
```
- Shows the label of the currently active chat model
- Click cycles: Primary → Backup 1 → Backup 2 → Primary
- Slots with no model assigned are skipped
- Color: same purple/amber/slate theme, based on provider type (optional)
---
## Migration
V1 → V2 is handled in `_load()`:
1. Detect `version == 1` (or missing)
2. Synthesize `providers.anthropic` catalog from hardcoded defaults
3. Synthesize `providers.google` — migrate API key from `auth.json` as first account
4. Convert built-in role assignments (`claude_cli` / `gemini_api`) to new model entry IDs
5. Existing `hosts[]` and `local_openai` models carry over unchanged
6. Write `version: 2` and save
No data loss. Old `local_llm.json` migration path still works (V0 → V1 → V2).
### Gap B — Agent mode placeholder (Phase 3, quick fix)
`app.js` lines 257–258 hard-code `"Gemini tool loop"`. Should say `"orchestrator"` since
the orchestrator role can now be a local model.
---
## Phases
### Phase 1 — Data model + backend routing (no UI changes yet)
- Extend schema to V2 in `model_registry.py`
- Migration from V1 on first load
- Update `_resolve_model()` to handle `gemini_api` + account lookup
- Update `llm_client.complete()` to accept `slot` parameter
- Update `routers/chat.py` to pass `slot` instead of backend string
- Keep backend toggle UI working (map old strings to slots temporarily)
- **Deliverable:** routing works with multi-account Gemini, no UI changes needed yet
### Phase 1 — Data model + routing ✅ 2026-04-27
- V2 schema with `providers` section
- Auto migration V1→V2 (pulls gemini_api_key from auth.json → Google accounts)
- `_resolve_model()` merges account API key for `gemini_api` type
- `get_google_api_key()`, `save_cloud_model()`, `save/remove_google_account()`
- Orchestrator router uses model-resolved API key
### Phase 2 — Cloud provider UI
- Add Anthropic and Google sections to `/settings/local` (rename to `/settings/models`)
- Google accounts: add/remove API keys with labels
- Editable model catalog for Anthropic + Google (add/remove model IDs from the list)
- Model entry creation: provider picker → model dropdown (from catalog) → account/credential picker
- **Deliverable:** can register cloud models in the UI just like local models
### Phase 2 — Cloud provider UI ✅ 2026-04-27
- `/settings/models` (canonical, `/settings/local` redirects)
- Cloud Providers section: Anthropic info + Google account add/remove
- Add Model form with provider tabs (Local / Google / Anthropic)
- Provider badges on model rows (Anthropic / Google / Local)
- Settings page updated: Gemini Key section replaced by Model Registry card
### Phase 3 — Unified role assignments + toggle redesign
- Promote role assignments to standalone `/settings/roles` page (or `/settings/models` bottom)
- All models from all providers appear in role selects
- Chat UI toggle: replace service-type cycle with slot cycle, show model label
- **Deliverable:** end-to-end unified experience
### Phase 3 — Toggle redesign + routing cleanup 🔄 in progress
- `model_registry.get_model_for_slot()` — resolve a specific slot without fallback chain
- `llm_client.complete()` — add `slot` parameter
- `routers/chat.py` — `ChatRequest.slot`, extend `GET /backend`, slot label in response tag
- `app.js` — data-driven toggle cycling model labels; send `slot` not backend string
- Fix Gap B: agent mode placeholder
### Phase 4 — Polish + future providers
- Claude direct API key support (optional, CLI is fine for now)
- OpenRouter as a named provider (already works as a "local" host with host_type=openai — could be promoted)
- Model catalog sync: fetch available models from Anthropic/Google API if keys are present
- Fix Gap A: `gemini_api` dispatch in `llm_client` → direct Google genai SDK for chat
- Claude direct API key support (alternative to CLI OAuth)
- OpenRouter as a named provider (already works as local host; could be promoted)
- Per-role "test" button in role assignments UI
---
## Open Questions
1. **Claude direct API key:** Is this needed now, or is CLI OAuth sufficient for all users?
- Decision: CLI-only for Phase 1; add API key support in Phase 4 if needed
2. **Catalog management:** Should the Anthropic/Google catalogs be server-wide defaults
that users can extend, or fully per-user?
- Recommendation: ship sensible defaults in code (updated with each deploy);
users can add custom entries if needed
3. **Toggle UX:** Cycle through slot labels ("Primary / Backup 1 / Backup 2") or cycle
through model labels ("Sonnet 4.6 / Gemini 2.5 Flash / Gemma 4")?
- Model labels are more useful — clearer what you're switching to
4. **Orchestrator mode toggle:** Does agent mode also respect the slot toggle, or is it
always "use orchestrator role"?
- Keep orchestrator role separate; the UI toggle only affects `chat` role
- Per-user catalog additions (extend ANTHROPIC_CATALOG / GOOGLE_CATALOG from UI)