22 Commits

Author SHA1 Message Date
Scott Idem
658c508925 feat: multi-level agent management — background agents, lifecycle tools, 3-level hierarchy
agent_manager.py (new):
- AgentRecord dataclass: agent_id, level (1/2/3), role, task, status, started,
  parent_id (lineage), finished, result, notify, _task_ref
- register() / finish() / cancel_agent() / list_agents() / get() / set_task_ref()
- Calls notification.notify() on completion when notify=True (same channel as
  reminders and cron completions)
- 24-hour pruning of completed records on each new registration

spawn_agent (tools/agents.py):
- background=True: fires asyncio.create_task(), registers in agent_manager, returns
  agent_id string immediately — sync path unchanged (no regression)
- notify=True: push/Talk notification when the background task completes
- Level enforcement: _agent_level param tracks hierarchy depth; when spawning from
  Level 2, child automatically gets spawn_agent + aider_run denied so Level 3 agents
  cannot delegate further

New lifecycle tools (tools/agents.py + __init__.py):
- agent_status(agent_id) — status, role, level, elapsed, task, result preview; user-level
- agent_list(status, limit) — all agents for current user, newest first; user-level
- agent_cancel(agent_id) — kills background task; admin-only, confirm-required

tests/test_agent_manager.py (new, 41 tests):
- agent_manager CRUD, pruning, notification hook
- spawn_agent background: returns immediately, completes async, timeout, failure
- Level enforcement: L1→L2 permits spawn, L2→L3 auto-denies; explicit tool_list path
- agent_status / agent_list / agent_cancel output formatting
- aider_run background: returns agent_id, completes async, sync path unchanged
- All tests run without browser or Cortex service (~2.5s total)
  Run: cd cortex && .venv/bin/python -m pytest tests/test_agent_manager.py -v

Docs: ARCH__FUTURE.md §13 (full design), ROADMAP.md, TODO__Agents.md, MASTER.md,
HELP.md (orchestrator description corrected, tool schema line updated to reflect
keyword routing), CLAUDE.md tool count 66→69.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-03 22:40:20 -04:00
Scott Idem
a92fd90f0d feat: Anthropic SDK backend — API key alternative to Claude CLI OAuth
Adds `anthropic_api` model type so users can authenticate with a direct
Anthropic API key instead of (or alongside) the CLI OAuth session.

- model_registry.py: `anthropic_api` type; `save/get/remove_anthropic_api_key()`
  mirroring the Google account pattern; `save_cloud_model()` now picks type
  based on credential type (cli → claude_cli, api_key → anthropic_api);
  `_resolve_model()` merges api_key from the credential entry
- llm_client.py: `_anthropic_api()` backend (AsyncAnthropic SDK); dispatch
  and fallback wiring; usage tracking
- routers/local_llm.py: Anthropic API key management routes
  (POST /settings/local/anthropic-key, /anthropic-key/{id}/remove);
  `anthropic_api` badge and edit-form credential selector
- static/local_llm.html: Anthropic Cloud Provider block now shows API key
  management (add/remove); Add Model → Anthropic tab has credential selector
  (CLI vs API key)
- requirements.txt: enable anthropic>=0.40.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 21:30:56 -04:00
Scott Idem
70665fadff feat: schedules UI, task cron type, monthly/yearly schedules, AE DB tools, integrations page
- Schedules web UI (/settings/crons): list, add, edit, pause/resume, delete jobs
- cron task type: full orchestrator tool loop on a schedule, result → notification channel
- parse_schedule: monthly/yearly formats (monthly:DD:HH:MM, yearly:MM:DD:HH:MM)
- HA inbound webhook tools toggle: orchestrator loop vs. direct LLM, configurable in UI
- ae_db_query/describe/show_view: SELECT-only Aether MariaDB access (admin, per-user creds)
- /settings/integrations: admin-only page for Aether DB credentials
- Schedules nav link added to all settings pages
- pymysql added to requirements
- Docs updated: HELP.md, MASTER.md, CLAUDE.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-13 21:06:43 -04:00
Scott Idem
76fef827c5 docs: update tool count to 62 and current state to 2026-05-12
Reflects file_diff and git_status/log/diff additions, pg.css refactor,
and reasoning level controls added this session.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-12 00:14:12 -04:00
Scott Idem
69ec2f667d feat: tool risk policy UI + wiring through all orchestrators
- New /settings/tools page: max_risk selector (low/medium/high) + per-tool
  override dropdowns (Default / Force include / Force exclude) for all 58 tools
  grouped by category with color-coded risk badges; JS updates Auto status live
- get_tools_for_role() + get_openai_tools_for_role() now accept max_risk,
  whitelist, blacklist; _apply_risk_policy() handles the filtering logic
- get_risk_policy() helper in auth_utils reads from tool_policy.json
- Risk policy wired through orchestrator.py, openai_orchestrator.py,
  orchestrator_engine.py, nextcloud_talk.py, homeassistant.py
- Tools nav link added to settings.html and notifications.html
- CLAUDE.md and ARCH__SYSTEM.md updated: tool count 50→58, risk system docs,
  tool access control three-layer model documented

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-11 22:45:04 -04:00
Scott Idem
7b443b40a4 feat: http_post tool, nc_talk_history tool, local orchestrator retry
- http_post: POST to external URLs with per-user URL prefix allowlist
  (home/{user}/http_allowlist.json); admin-only, confirm-required
- nc_talk_history: read recent NC Talk messages via Basic Auth (requires
  nc_username + nc_app_password in channels.json under nextcloud)
- openai_orchestrator: _chat_with_retry() wraps both API calls with
  exponential backoff (3 attempts, 1s/2s) on connection errors and
  transient status codes (429, 500, 502, 503, 504)
- Docs updated: CLAUDE.md, HELP.md, TODO, MASTER, ROADMAP (50 tools)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 13:38:38 -04:00
Scott Idem
3672fa1506 docs: comprehensive doc audit — sync all docs to current state
- MASTER.md: tool count 40→47, add proactive notifications + spawn_agent rows, date bump
- ROADMAP.md: mark local orchestrator/web push/proactive notifs/spawn_agent/web_read/session_read as done, date bump
- ARCH__CHANNELS.md: rewrite notification channel config section — all 4 channels, all triggers, on-demand endpoints
- ARCH__SYSTEM.md: update tools/ module list to include files, agents
- README.md: update LLM backends in architecture diagram, add browser push to channels table
- CLAUDE.md: add doc update checklist to Documentation Philosophy section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 13:13:45 -04:00
Scott Idem
17e8869d12 docs: update tool count (45→47), HELP.md, and TODO for new web/file tools
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-09 13:05:04 -04:00
Scott Idem
ff154b1ec0 docs: update CLAUDE.md, HELP.md, and TODO for notifications page + push fix
- CLAUDE.md: date → 2026-05-08, add Proactive notifications row to channel table
- HELP.md: update Notifications settings entry, expand Push Notifications section
  with channel config link, add test API endpoints to reference table
- TODO__Agents.md: mark notifications dedicated page and pywebpush fix as done

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 23:58:47 -04:00
Scott Idem
09d775b47b feat: spawn_agent tool + host max_concurrent + docs
Adds a synchronous sub-agent spawning tool that lets the orchestrator
delegate tasks to a specific role's model and tool set.

- cortex/tools/agents.py: spawn_agent(task, role, tier, timeout, max_rounds)
  - Supports local_openai and gemini_api model types
  - Per-host asyncio semaphore (keyed by host_id or model type)
  - asyncio.wait_for() enforces timeout; admin-only tool
- cortex/model_registry.py: max_concurrent field in host schema (default 3,
  clamped 1-20); backfilled on _normalize() for existing hosts
- cortex/routers/local_llm.py + local_llm.html: "Max parallel" number input
  in host add/edit forms
- cortex/tools/__init__.py: spawn_agent registered in TOOL_CATEGORIES["Agents"],
  _CALLABLES, TOOL_ROLES (admin), and _ALL_DECLARATIONS
- Docs: TOOLS.md count 44→45, spawn_agent section; HELP.md tool table updated;
  ARCH__FUTURE.md Round 2 completed items; TODO__Agents.md spawn_agent checked;
  CLAUDE.md tool count and list updated

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 22:48:21 -04:00
Scott Idem
f8f7cd75da feat: audit log, usage tracking UI, OpenAI orchestrator compaction, onboarding + docs
Tool audit log:
- Every orchestrator tool call logged to home/{user}/tool_audit/YYYY-MM-DD.jsonl
- Files panel sidebar: audit log group (collapsed), date-linked read-only table
- Admin endpoints: /api/audit/files, /api/audit/day, /api/audit/recent, /api/audit/stats
- Engine and model name recorded per entry

OpenAI orchestrator improvements:
- Context budget enforcement: 75% of model context_k (min 16k)
- Message compaction: truncates old tool results when approaching budget
- max_rounds respected per model config (intersected with server cap)

OpenRouter onboarding (setup.html, onboarding.py, app.js, settings.html):
- Step 3 of 3: /setup/model with curated model picker
- Chat banner for users on server-default model (informational, not alarmist)
- Settings quick-link card; /setup/model works standalone for existing users

Model registry + session store:
- set_role_config / get_role_config for per-role tool lists and system_append
- session_store: session rename, session name backfill endpoint

UI updates (app.js, index.html, style.css, local_llm.html):
- Role toggle in context panel
- Off-the-record mode
- Agent notes read-only viewer
- OPERATIONS.md loaded at T2+ in context

Documentation:
- HELP.md: full tool table, per-role tool sets, Agent Notes, usage tracking
- TOOLS.md: Agent Notes section, count corrected to 44
- ARCH__SYSTEM.md, ARCH__BACKENDS.md, MASTER.md updated to match reality
- CLAUDE.md: onboarding flow, documentation philosophy sections
- README.md: stack in practice, DeepSeek TUI mention, architecture diagram updated
- TODO__Agents.md: onboarding task completed with deviation notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-08 21:26:43 -04:00
Scott Idem
ddf44a2aee feat: web push notifications (VAPID)
- push_utils.py: subscription storage + send helper (auto-prunes 410 endpoints)
- routers/push.py: GET /api/push/vapid-key (public), POST/DELETE /api/push/subscribe
- sw.js: push event listener shows notification; notificationclick focuses/opens tab
- app.js: subscribe/unsubscribe flow + "Enable notifications" toggle in settings dropdown
- tools/notify.py: web_push orchestrator tool (user-level, no admin required)
- VAPID keys in .env; pywebpush added to requirements.txt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:38:58 -04:00
Scott Idem
8d4aa4094c feat: usage tracking + knowledge import script
- usage_tracker.py: daily token/call buckets per user (home/{user}/usage.json)
- Hook into local backend (OpenAI usage field) and Gemini API (usage_metadata)
- Claude/Gemini CLI backends produce no structured token data and are not tracked
- Fix CLAUDE.md stale tool count (27 → 39) and refresh tool list
- scripts/import_knowledge.py: walk markdown dirs, chunk by H2, call local LLM
  for summaries, create AE journal entries with path-derived tags; resumable via
  state file; --dry-run and --limit flags for safe testing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 20:38:31 -04:00
Scott Idem
1222f806ce docs: sync all documentation to current state (2026-04-28)
HELP.md:
- Replace stale "Agent Mode" section with accurate "Tools ()" section
  including full tool table (27 tools across 9 categories)
- Fix header controls table: distinguish Context & Memory panel (sliders)
  from Settings dropdown (☰)
- Update Settings panel section: "Backend" → "Role", add S/M/L and ⌃↵
  to Display controls list
- Update Backends section to match Role toggle terminology

TODO__Agents.md:
- Mark 5 new journal tools as complete (ae_journal_list, entry_update,
  entry_disable, entry_append, entry_prepend)
- Add completed sections for: shell_exec tool, Tools toggle decoupling,
  UI input area polish (2026-04-28)

MASTER.md:
- Date: 2026-04-27 → 2026-04-28
- Orchestrator row: "Agent mode in UI" → " Tools toggle in UI (27 tools)"

CLAUDE.md:
- Date: 2026-04-03 → 2026-04-28
- Add orchestrator row to Current State table
- Add full 27-tool list for quick reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-28 22:07:48 -04:00
Scott Idem
6c84d6ae72 docs: update README and CLAUDE.md for install script and backup
- README: add Setup/Install section (install.py), Development Workflow
  (dev-restart.sh), and Backup (restic/backup.sh) sections; fix
  .env.example reference → .env.default
- CLAUDE.md: replace stale Docker start command with install.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-08 21:36:48 -04:00
Scott Idem
a4daebdc9b feat: local LLM multi-model, session search, cron proactive types, notifications, docs overhaul
Local LLM:
- user_settings.py: per-user hosts/models config (local_llm.json)
- routers/local_llm.py + static/local_llm.html: dedicated settings page
- llm_client.py: local OpenAI-compatible backend via httpx
- config.py: LOCAL_API_URL/KEY/MODEL + per-backend timeouts
- Active model shown near backend toggle (amber hint text)

Memory distillation:
- memory_distiller.py: DISTILL_BACKEND_MID/LONG .env overrides
- scheduler.py + notification.py: notify NC Talk after mid/long distill
- notification.py: outbound channel abstraction (NC Talk, extensible)

Session search:
- routers/files.py: GET /sessions/search?q= with excerpts grouped by date
- static/index.html + app.js: search UI in file sidebar with highlight
- _esc() helper to prevent XSS in search results

Proactive cron:
- cron_runner.py: new job types — message (send directly) and brief (LLM + send)
- Both support optional per-job channel override

Channels:
- routers/nextcloud_talk.py: consolidated using notification._send_nct_message()
- routers/auth.py: local backend status in /auth/status
- routers/chat.py: /backend returns {primary, fallback, local_model} object

UI / UX:
- Copy button for user messages (matching assistant)
- Autocomplete disabled on sensitive form fields
- settings.html: local model section replaced with link to /settings/local

Docs overhaul:
- MASTER.md hub + ARCH__SYSTEM/BACKENDS/PERSONA/CHANNELS/FUTURE.md
- ARCH__Intelligence_Layer.md replaced with redirect table
- CORTEX.md trimmed to vision only; README updated
- OPEN_WEBUI_API.md added to docs/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-05 20:53:06 -04:00
Scott Idem
8e20bfbea8 feat: shared Help base, Google OAuth live, new personas, cleanup
- cortex/static/HELP.md: shared Help & Reference base served to all users
- help.html: loads shared base + appends persona-specific HELP.md if present
- inara/HELP.md: cleared (content moved to shared base)
- Google OAuth: registered scott.idem@oneskyit.com; flow now working end-to-end
- .gitignore: exclude home/**/sessions/ (runtime logs)
- New personas tracked: home/holly/persona/donut/, home/scott/persona/developer/
- Removed orphans: holly/, personas/, cortex-holly.service
- CLAUDE.md: updated current state and recently completed list to 2026-03-27

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-27 22:55:45 -04:00
Scott Idem
c2825194d4 docs: update project docs, NC Talk guide, Tina persona, and gitignore
- CLAUDE.md: add new auth/onboarding files to directory map, update
  security section (JWT/bcrypt/invite details), expand recently completed
- README.md: fix Web UI auth description, add User Management section
- TODO__Agents.md: mark NC Talk docs and auth/onboarding complete,
  update Holly onboarding plan to reflect single-instance multi-user approach
- docs/NEXTCLOUD_TALK_BOT.md: complete guide — occ commands, nginx config,
  clarify incoming vs outgoing HMAC difference, multi-user note, full
  troubleshooting table
- home/holly/persona/tina/: flesh out all four persona files with real
  content (DCC name origin, metal music, reading, foster cats, Holly's profile)
- .gitignore: exclude home/**/auth.json, invite.json, profile.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-21 00:13:35 -04:00
Scott Idem
77e770cdb2 feat: multi-user/multi-persona support with two-level home directory layout
Restructures persona storage from a flat personas/{name}/ layout to
home/{username}/persona/{name}/, mirroring Linux home directories.

Changes:
- persona.py: two ContextVars (user + persona), Linux-style name validation,
  set_context(), get_user(), get_persona(), validate(), list_users(),
  list_user_personas(); persona_path() takes (username, name)
- config.py: replaces personas_dir with home_dir + home_root()
- git mv personas/inara → home/scott/persona/inara (history preserved)
- home/holly/persona/tina/: Holly's persona stub added
- cron_runner.py: all storage functions take (username, persona) params
- tools/cron.py: stamps user + persona on jobs; APScheduler IDs are
  {user}:{persona}:{job_id} to prevent collisions across users
- memory_distiller.py: distill_short/mid/long take (username, persona);
  added missing Path + settings imports
- scheduler.py: _load_user_crons() iterates home/*/persona/* (two-level)
- routers/chat.py, orchestrator.py: user field added; set_context() called
- tests/conftest.py: home_root fixture with two-level structure;
  patches home_dir instead of personas_dir
- tests/test_persona.py: fully rewritten for two-level API
- tests/test_api_files.py: updated fixture name and path
- .env.default: documents HOME_DIR setting; scrubs stale API key
- CLAUDE.md, README.md: directory maps updated for new layout

All 80 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-20 22:35:40 -04:00
Scott Idem
aaac3e1353 feat: persist orchestrator sessions + user service + docs update
- Orchestrator now saves turns to session store so history survives page refresh
- UI session_id updated from job result; history controls attached to agent turns
- Cortex migrated from system service to systemd user service (no more sudo)
- Update README.md and CLAUDE.md with correct service commands

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-18 23:08:38 -04:00
Scott Idem
9b818aa5c7 feat: orchestrator Agent mode UI + claude_allow_dir tool + fix DDG search
- Add Agent mode toggle to web UI input row — routes through POST /orchestrate
  instead of /chat; polls for result with live tool-call count in thinking bubble
- Add cortex/tools/system.py with claude_allow_dir tool; registers in tool registry
- Fix web search: duckduckgo_search renamed to ddgs, update import + requirements.txt
- Allow WebSearch and WebFetch in ~/.claude/settings.json for Claude CLI fallback
- Add claude-allow-dir script docs and security note to CLAUDE.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-18 22:42:44 -04:00
Scott Idem
ed472ce9a0 feat: Intelligence Layer Phase 1 — orchestrator service
Adds the Gemini API orchestrator (ReAct tool loop → Claude responder):

Orchestrator engine + router:
- orchestrator_engine.py: Gemini API tool loop, Claude CLI handoff
- routers/orchestrator.py: POST /orchestrate (async job queue), GET /orchestrate/{job_id}

Tools (cortex/tools/):
- web.py: DuckDuckGo web search (no key required)
- ae_knowledge.py: ae_journal_search + ae_journal_entry_create (AE V3 API)
- ae_tasks.py: ae_task_list (reads agents_sync Kanban filesystem)
- files.py: file_read (path-allowlisted to safe dirs)

Config + deps:
- config.py: orchestrator, DuckDuckGo, and AE API settings
- requirements.txt: google-genai, duckduckgo-search
- .env.default: reference config with all new keys documented

Docs:
- CLAUDE.md, README.md, documentation/ added to repo
- Port references updated 7331 → 8000 throughout
- Default model updated to gemini-2.5-flash

Tested: ae_task_list, ae_journal_search, web_search all working end-to-end.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-18 19:37:49 -04:00