Cortex-Inara/CLAUDE.md

# CLAUDE.md — Cortex / Inara Project

This file is loaded automatically by Claude Code when working in this directory.
Read it before touching any files.

---

## Identity & Context

- **Project:** Cortex (dispatcher) + Inara (resident agent)
- **Owner:** Scott Idem (One Sky IT / Danger Zone)
- **Machine context:** See `~/CLAUDE.md` for fleet identity (`scott_lpt` = General Manager)
- **Named after:** The 'verse-wide communications network (Firefly)

---

## Directory Map

```
Cortex_and_Inara_dev/
  cortex/                ← FastAPI service (the dispatcher)
    main.py              ← App entry point, router registration
    config.py            ← All settings (pydantic-settings, reads .env)
    persona.py           ← Two-level identity: user + persona, path resolution, ContextVars
    llm_client.py        ← Claude CLI + Gemini CLI subprocess backends
    orchestrator_engine.py ← Gemini API ReAct tool loop → Claude handoff
    context_loader.py    ← Builds system prompt from persona files (tier 1–4)
    session_store.py     ← In-memory + file session persistence
    session_logger.py    ← Writes session turns to home/{user}/persona/{name}/sessions/
    memory_distiller.py  ← Short/mid/long distill jobs (APScheduler)
    cron_runner.py       ← Cron job storage, schedule parsing, job execution
    scheduler.py         ← APScheduler setup (distill + user crons)
    event_bus.py         ← Internal SSE pub/sub (NC Talk → browser)
    auth_utils.py        ← bcrypt passwords, JWT create/decode, invite token system
    auth_middleware.py   ← SessionAuthMiddleware — JWT cookie validation on all routes
    persona_template.py  ← Bootstrap a new persona directory from string templates
    email_utils.py       ← SMTP_SSL email helpers (invite emails, future notifications)
    routers/
      chat.py            ← POST /chat (streaming SSE)
      orchestrator.py    ← POST /orchestrate, GET /orchestrate/{job_id}
      auth.py            ← GET /auth/status (Claude + Gemini CLI token checks)
      distill.py         ← POST /distill/*, GET /distill/status
      files.py           ← GET /files (persona file browser)
      nextcloud_talk.py  ← POST /webhook/nextcloud (NC Talk bot)
      google_chat.py     ← POST /webhook/google (Google Chat Add-on)
      ui.py              ← Login/logout, /{user}/{persona} UI route, /api/personas
      onboarding.py      ← /setup/{token} password step + /setup/persona creation
    tools/
      __init__.py        ← Tool registry (Gemini FunctionDeclarations + dispatcher)
      web.py             ← DuckDuckGo web_search tool
      scratch.py         ← Scratchpad tools (scratch_read/write/append/clear)
      tasks.py           ← Personal task management (task_create/list/update/complete)
      cron.py            ← Scheduled job tools (cron_list/add/remove/toggle)
      system.py          ← Local machine tools (claude_allow_dir)
    tests/               ← pytest test suite (80 tests)
    static/              ← Single-page web UI (index.html, style.css, app.js)
                           login.html — login form (dark theme, POST /login)
                           setup.html — onboarding form (password + persona creation)
    data/sessions/       ← Persisted session JSON files

  home/                  ← User and persona data (Linux home layout)
    scott/
      persona/
        inara/           ← Inara identity, memory, context, sessions
          IDENTITY.md    ← Who Inara is
          SOUL.md        ← Values, personality, voice
          PROTOCOLS.md   ← Behavioral rules
          CONTEXT_TIERS.md ← What each tier (1–4) includes in the system prompt
          USER.md        ← Scott's profile (loaded into context)
          HELP.md        ← In-app help content (rendered in UI)
          MEMORY_LONG.md ← Long-term memory (auto-distilled monthly)
          MEMORY_MID.md  ← Mid-term memory (auto-distilled weekly)
          MEMORY_SHORT.md ← Short-term memory (auto-distilled daily)
          REMINDERS.md   ← Pending reminders (auto-surfaced in context at tier 2+)
          SCRATCH.md     ← Ephemeral scratchpad
          TASKS.json     ← Personal task list
          CRONS.json     ← Scheduled jobs
          sessions/      ← Session turn logs (YYYY-MM-DD.md)
    holly/
      persona/
        tina/            ← Tina (Holly's persona) — same structure as inara/

  docs/                  ← Integration reference docs
    NEXTCLOUD_TALK_BOT.md
    OPEN_WEBUI_API.md    ← Open WebUI API: tool calling, RAG, model management

  documentation/         ← Architecture decisions and agent task list
    TODO__Agents.md      ← READ THIS FIRST — active task list
    ARCH__Intelligence_Layer.md ← Orchestrator, dev agent, knowledge architecture

  docker-compose.yml     ← Docker deployment
  .env.default           ← Reference config (copy to .env, fill in secrets)
  README.md              ← Project orientation
```

---

## Run Commands

```bash
# First-time setup or update on any machine
python3 install.py

# Restart service (after any Python change)
systemctl --user restart cortex

# Syntax check a file before restarting
python3 -m py_compile cortex/<file>.py

# Syntax check all routers
for f in cortex/routers/*.py cortex/tools/*.py cortex/orchestrator_engine.py; do
    python3 -m py_compile "$f" && echo "OK: $f"
done

# Install/update dependencies
cd cortex && .venv/bin/pip install -r requirements.txt

# Logs
journalctl --user -u cortex -f

# Web UI (local)
http://localhost:8000

# Swagger docs
http://localhost:8000/docs
```

---

## Key Design Decisions

### Two-Brain Architecture (Orchestrator / Responder)
- **Gemini API** (`orchestrator_engine.py`) — runs the ReAct tool loop; handles tool calling, planning, research
- **Claude CLI** (`llm_client.py`) — produces all user-facing responses; receives enriched context from Gemini
- **Direct chat** bypasses the orchestrator entirely — `POST /chat` goes straight to Claude (faster)
- **Orchestrated tasks** go to `POST /orchestrate` — returns a job_id, result is polled

### LLM Backends
- `llm_client.py` manages Claude CLI (`claude --print`) and Gemini CLI (`gemini -p`) subprocesses
- `orchestrator_engine.py` uses the Gemini **API** (google-genai SDK) — completely separate from the Gemini CLI
- Claude OAuth token is read live from `~/.claude/.credentials.json` (never rely on stale env var)

### Tool Strategy
- Orchestrator tools live in `cortex/tools/` — separate from the `ae_*` MCP tools
- **Do not modify** the `ae_*` MCP server to support orchestrator needs; add new tools to `cortex/tools/` instead
- Tools are registered in `cortex/tools/__init__.py` as both Gemini FunctionDeclarations and Python callables

### Context / Memory
- `context_loader.py` assembles Inara's system prompt from `inara/` files based on tier (1–4)
- Tier 1 = minimal (identity only); Tier 2 = standard (+ memory + user profile); Tier 3 = + last 2 sessions; Tier 4 = + last 7 sessions
- Memory files are written by the distiller or manually — do not delete them

### Security / Safety
- **Never `rm`** — move files to `~/tmp/gemini_trash`
- **Never commit secrets** — `.env` is gitignored; use `.env.default` as the reference
- `NEXTCLOUD_TALK_BOT_SECRET` and `GEMINI_API_KEY` live in `.env` only
- `/channels/*` and `/health` are publicly exposed (webhook auth is handled at app layer — JWT/HMAC)
- `/login`, `/logout`, `/setup/*`, `/static/*` are public — all other routes require a valid JWT session cookie
- `SessionAuthMiddleware` (`auth_middleware.py`) validates the cookie on every request; browsers are redirected to `/login`, API calls get 401
- Passwords are bcrypt-hashed and stored in `home/{username}/auth.json` — never in `.env` or the DB
- Invite tokens are one-time-use, 72-hour expiry, stored in `home/{username}/invite.json`

### Onboarding Flow
New users follow a three-step setup before reaching the chat:
1. `GET /setup/{token}` → password form → `POST /setup/{token}` sets password + session cookie
2. `GET /setup/persona` → persona creation form → `POST /setup/persona` bootstraps persona directory
3. `GET /setup/model` → OpenRouter quick-connect → `POST /setup/model` saves host + model + role assignment

Step 3 is optional (skip link goes straight to `/{user}/{persona}`). `/setup/model` also works
standalone (accessible from Settings) for existing users who haven't configured a model.

All in `cortex/routers/onboarding.py`. Model writes use `model_registry.py`: `save_host()`,
`save_model()`, `set_role(username, "chat", "primary", model_id)`.

### Documentation Philosophy
Cortex is a no-black-box system. Docs must match reality — at all times.

- **Docs first:** When planning significant changes, update `TODO__Agents.md` and the relevant
  `ARCH__*.md` to describe the intended design *before* implementing. This creates a spec to
  implement against.
- **Verify after:** Once implementation is complete, re-read the pre-written docs and confirm
  they match what was actually built. Update anything that drifted.
- **HELP.md is a user contract:** It describes what users can do. Never let it describe
  features that don't exist or omit features that do.
- **CLAUDE.md + ARCH__*.md are the developer contract:** Update them as the architecture evolves.
- **Stale docs are bugs.** If you notice drift, fix it before moving on.

### Doc update checklist (run after any significant change)

| Doc | Update when |
|---|---|
| `CLAUDE.md` | New tool, channel, router, major design change, tool count |
| `cortex/static/HELP.md` | Any user-visible feature — tools, settings, UI, API endpoints |
| `documentation/TODO__Agents.md` | Mark completed items; add new planned work |
| `documentation/MASTER.md` | New capability goes live; tool count changes |
| `documentation/ROADMAP.md` | Phase items completed or added |
| `documentation/ARCH__CHANNELS.md` | New channel, notification trigger, or scheduler job |
| `documentation/ARCH__SYSTEM.md` | New module, router, or tools/ file |
| `README.md` | Architecture diagram, channels table, or setup steps change |

---

## Adding a New Tool

1. Implement the tool function in `cortex/tools/<domain>.py`
   - Must be `async def`; use `asyncio.to_thread` for blocking calls
   - Return a plain string result
2. Add a `FunctionDeclaration` and register it in `cortex/tools/__init__.py`:
   - Import the callable
   - Add to `TOOL_CATEGORIES` (pick an existing category or create one)
   - Add to `_CALLABLES`
   - Add a `TOOL_RISK` rating (low/medium/high)
   - Add to `TOOL_ROLES` if admin-only; add to `CONFIRM_REQUIRED` if destructive
   - Add module to `_ALL_DECLARATIONS`
3. Syntax check: `python3 -m py_compile cortex/tools/<domain>.py`
4. Restart Cortex

## Managing Claude Code Directory Permissions

Claude Code prompts (or silently hangs) when it needs to read or write a directory outside
its current working directory. The `claude-allow-dir` script patches `~/.claude/settings.json`
to add auto-allow rules so Claude no longer blocks on those paths.

### Script: `~/.local/bin/claude-allow-dir`

```bash
# Allow read + write (default)
claude-allow-dir ~/OSIT_dev/aether_api_fastapi

# Read-only
claude-allow-dir ~/agents_sync r

# Write-only
claude-allow-dir /tmp w
```

Adds `Read(path/*)` and/or `Edit(path/*)` + `Write(path/*)` entries to the `permissions.allow`
array in `~/.claude/settings.json`. Idempotent — safe to run twice on the same path.
Changes take effect in the next Claude Code session (or after opening `/hooks` in the UI).

### Orchestrator tool: `claude_allow_dir`

Cortex exposes this as a Gemini tool (`cortex/tools/system.py`) so the orchestrator can add
allow rules on Inara's behalf without human intervention.

**Security note:** This tool modifies Claude Code's own permission settings. The Gemini
orchestrator calling it can grant Claude access to any directory on the machine. Keep this
in mind when evaluating orchestrator behavior — it should only be invoked when Scott has
clearly asked for a directory to be unblocked.

## Adding a New Router

1. Create `cortex/routers/<name>.py` with `router = APIRouter()`
2. Import and register in `cortex/main.py`
3. Syntax check, restart

---

## Current State (2026-05-09)

Cortex is running and stable. All channels are live:

| Channel | Status | Notes |
|---|---|---|
| Web UI | ✅ Live | `https://cortex.dgrzone.com` — PWA-installable |
| Nextcloud Talk | ✅ Live | HMAC-signed webhook, async reply |
| Google Chat | ✅ Live | Workspace Add-on, `hostAppDataAction` response format |
| Local backend | ✅ Live | Open WebUI/Ollama on scott_gaming, per-user multi-model config |
| Gemini orchestrator | ✅ Live | Gemini API tool loop → Claude response; ⚡ toggle in UI |
| Local orchestrator | ✅ Live | OpenAI-compatible ReAct loop; fires when orchestrator role → local model |
| Tool audit log | ✅ Live | Every tool call logged to `home/{user}/tool_audit/YYYY-MM-DD.jsonl` |
| Token usage tracking | ✅ Live | Per-user `home/{user}/usage.json`; summary in Settings |
| Web push | ✅ Live | VAPID push notifications; `web_push` tool; subscribe via ☰ menu |
| Proactive notifications | ✅ Live | Daily reminder check (09:00); distill/cron completions; `GET /settings/notifications` dedicated page |

Active users: scott (inara), holly (tina), brian (wintermute)

**58 orchestrator tools** across 15 domain modules:
web_search/http_fetch/web_read/http_post,
project_file_read/list + file_stat/grep/syntax_check (project-scoped),
file_read/list/write/session_read/session_search (system-scoped, admin),
shell_exec/claude_allow_dir,
cortex_restart/logs/status/update,
task_list/create/update/complete, cron_list/add/remove/toggle,
reminders_add/list/remove/clear, scratch_read/write/append/clear,
web_push/email_send/nc_talk_send/nc_talk_history,
ae_journal_list/search/entries_list/entry_read/create/update/disable/append/prepend,
ae_task_list, agent_notes_read/write/append/clear, spawn_agent,
ha_get_state/ha_get_states/ha_call_service.

Each tool has a `TOOL_RISK` rating (low/medium/high). Configure access at `/settings/tools`
(max_risk threshold + per-tool whitelist/blacklist). Risk policy stored in `home/{user}/tool_policy.json`.

See `documentation/TODO__Agents.md` for the active task list.
See `documentation/ROADMAP.md` for phases and what's next.

---

## Related Docs

| File | Purpose |
|---|---|
| `documentation/MASTER.md` | **Start here** — index, current state, all doc links |
| `documentation/TODO__Agents.md` | Active task list — read before starting work |
| `documentation/ROADMAP.md` | Phases — what's done, what's next |
| `documentation/ARCH__SYSTEM.md` | System architecture and component map |
| `documentation/ARCH__BACKENDS.md` | LLM backends, routing, per-user config |
| `documentation/ARCH__PERSONA.md` | Persona system, context tiers, memory distillation |
| `documentation/ARCH__CHANNELS.md` | Input channels — web, NC Talk, Google Chat, cron |
| `documentation/ARCH__FUTURE.md` | Planned: local orchestrator, dev agents, knowledge layer |
| `~/agents_sync/projects/CORTEX.md` | Project vision and philosophy |
| `~/agents_sync/CLAUDE.md` | Fleet coordination rules |
| `~/CLAUDE.md` | Machine identity (`scott_lpt`) |