Files

Scott Idem a4daebdc9b feat: local LLM multi-model, session search, cron proactive types, notifications, docs overhaul

Local LLM:
- user_settings.py: per-user hosts/models config (local_llm.json)
- routers/local_llm.py + static/local_llm.html: dedicated settings page
- llm_client.py: local OpenAI-compatible backend via httpx
- config.py: LOCAL_API_URL/KEY/MODEL + per-backend timeouts
- Active model shown near backend toggle (amber hint text)

Memory distillation:
- memory_distiller.py: DISTILL_BACKEND_MID/LONG .env overrides
- scheduler.py + notification.py: notify NC Talk after mid/long distill
- notification.py: outbound channel abstraction (NC Talk, extensible)

Session search:
- routers/files.py: GET /sessions/search?q= with excerpts grouped by date
- static/index.html + app.js: search UI in file sidebar with highlight
- _esc() helper to prevent XSS in search results

Proactive cron:
- cron_runner.py: new job types — message (send directly) and brief (LLM + send)
- Both support optional per-job channel override

Channels:
- routers/nextcloud_talk.py: consolidated using notification._send_nct_message()
- routers/auth.py: local backend status in /auth/status
- routers/chat.py: /backend returns {primary, fallback, local_model} object

UI / UX:
- Copy button for user messages (matching assistant)
- Autocomplete disabled on sensitive form fields
- settings.html: local model section replaced with link to /settings/local

Docs overhaul:
- MASTER.md hub + ARCH__SYSTEM/BACKENDS/PERSONA/CHANNELS/FUTURE.md
- ARCH__Intelligence_Layer.md replaced with redirect table
- CORTEX.md trimmed to vision only; README updated
- OPEN_WEBUI_API.md added to docs/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-05 20:53:06 -04:00

3.8 KiB

Raw Blame History

Architecture: LLM Backends

How Cortex talks to AI models. Last updated: 2026-04-03

Three Backends

Backend	Used for	Auth	Config
Claude CLI	Primary chat, all user-facing responses	OAuth token from `~/.claude/.credentials.json`	`DEFAULT_MODEL` in `.env`
Gemini CLI	Fallback when Claude unavailable	Gemini CLI credentials	Auto-fallback
Local (Open WebUI)	Private/offline tasks, cost-free use	API key per user in `local_llm.json`	`/settings/local` UI

The Gemini API (google-genai SDK) is also used — but only by the orchestrator tool loop, not as a general chat backend. See ARCH__FUTURE.md for the orchestrator pattern.

Backend Selection

User toggles backend in the UI: claude → gemini → local (cycles). The active backend is stored server-side; the UI reflects it with color coding (default / green / amber).

When local is active, the active model name appears below the toggle button.

Fallback chain (automatic, on error):

claude  → gemini
gemini  → claude
local   → claude

Auth expiry on Claude triggers a UI banner + claude_auth_expired SSE event.

Claude Backend (`_claude()`)

Runs claude --print --no-session-persistence --output-format text as a subprocess.

System prompt passed via --system-prompt
Conversation history formatted as <conversation> block
Token read live from ~/.claude/.credentials.json on every call — never relies on the env var, which goes stale after claude auth login
Model override via --model flag (e.g. claude-opus-4-6)

Timeout: TIMEOUT_CLAUDE=60 seconds (.env)

Gemini CLI Backend (`_gemini()`)

Runs gemini --output-format text --extensions "" -p <prompt> as a subprocess.

--extensions "" disables all MCP extensions — prevents child processes from keeping pipes open after responding
start_new_session=True puts the process in its own group for clean os.killpg on timeout
Output is cleaned to strip CLI noise lines (loading messages, retry notices, quota warnings)

Timeout: TIMEOUT_GEMINI=120 seconds (.env)

Local Backend (`_local()`)

HTTP POST to Open WebUI's OpenAI-compatible endpoint: {api_url}/api/chat/completions.

Per-user config in home/{user}/local_llm.json:

{
  "hosts": [{"id": "...", "label": "scott_gaming", "api_url": "http://192.168.32.19:3000", "api_key": "sk-..."}],
  "models": [{"id": "...", "host_id": "...", "label": "Gemma 4 Small", "model_name": "agent-support-gemma-small"}],
  "active_model_id": "..."
}

Resolution order for active model:

User's active_model_id in local_llm.json
.env server defaults (LOCAL_API_URL / LOCAL_MODEL)
Error — user is prompted to configure at /settings/local

Timeout: TIMEOUT_LOCAL=300 seconds (.env) — local models may need to load from disk.

Manage at: /settings/local — supports multiple hosts and models per user, "Fetch from host" button to populate model list from the server.

Distillation Backends

Memory distillation runs on a schedule and uses the LLM for mid and long distill passes. By default uses the primary backend (claude). Override in .env:

DISTILL_BACKEND_MID=local   # saves API credits — Gemma handles summarization well
DISTILL_BACKEND_LONG=       # empty = use primary (claude recommended for quality)

Current Local Models (scott_gaming, 8 GB VRAM)

Model	Alias	Speed	Practical Context
Gemma 4 E4B	`agent-support-gemma-small`	~25 t/s	72k tokens
Gemma 4 26B A4B (MoE)	`agent-support-gemma-medium`	~9 t/s	50k tokens

Both support OpenAI tools / tool_choice function calling — required for the local orchestrator.

Full Open WebUI API reference: docs/OPEN_WEBUI_API.md

3.8 KiB Raw Blame History