Compare commits

..

11 Commits

Author SHA1 Message Date
Scott Idem
02accefe8f feat: audit log in Files panel sidebar
Adds an "Audit Log" section (collapsed by default) at the bottom of the Files
panel showing tool_audit/YYYY-MM-DD.jsonl files for the current user.

- GET /api/audit/files  — lists available dates (newest first, any auth user)
- GET /api/audit/day    — returns entries for one date as JSON (any auth user)
- tool_audit.read_day() — reads a single day's JSONL file chronologically
- Clicking a date renders a read-only table: time / tool / status / args / result
- Status cells are colour-coded (green ok, red error, amber denied)
- Edit/Raw/Preview/Save buttons are hidden in audit view, restored on file switch
- Audit group starts collapsed; expands on click like other file groups

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 20:36:08 -04:00
Scott Idem
584ae679a6 feat: tool call audit log
Every orchestrator tool invocation is recorded to home/{user}/tool_audit/YYYY-MM-DD.jsonl.
Each entry captures: timestamp, user, tool, args (truncated), status (ok/error/denied),
result length, and a 300-char result snippet.

- tool_audit.py: JSONL writer with per-file asyncio locks; read_recent / read_recent_all_users helpers
- tools/__init__.py: hook in call_tool() — fire-and-forget record on every dispatch
- routers/audit.py: GET /api/audit/recent and /api/audit/stats (admin-only)
- tools/files.py: add home_root() to file_read allowed roots so agents can read audit JSONL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:55:59 -04:00
Scott Idem
ddf44a2aee feat: web push notifications (VAPID)
- push_utils.py: subscription storage + send helper (auto-prunes 410 endpoints)
- routers/push.py: GET /api/push/vapid-key (public), POST/DELETE /api/push/subscribe
- sw.js: push event listener shows notification; notificationclick focuses/opens tab
- app.js: subscribe/unsubscribe flow + "Enable notifications" toggle in settings dropdown
- tools/notify.py: web_push orchestrator tool (user-level, no admin required)
- VAPID keys in .env; pywebpush added to requirements.txt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:38:58 -04:00
Scott Idem
0b96772fa6 fix: show session friendly name in resume message and status bar
/history/{session_id} now returns a 'name' field alongside messages.
resumeSession() uses data.name first, then the sessionNames map, then
raw ID as fallback — so named sessions display correctly even on page
load before the sessions panel has been opened.

'Resumed session X' message also now shows the friendly name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:14:59 -04:00
Scott Idem
5d23d04e7e fix: session panel wider + two-line layout for session names
Root cause: 300px panel minus edit btn (28px) + meta (~130px) + delete
btn (28px) + gaps/padding left only ~70px (~7 chars) for the session name.

- Panel: 300px → 420px desktop, 300px → 380px mobile drawer
- Max-height: 340px → 400px
- Session item: name and meta now in a .session-body flex column, so the
  name gets full body width (panel minus two buttons) — meta lives below
- Edit mode: hides .session-body + delete, input takes the full body slot

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:07:33 -04:00
Scott Idem
7a0fbdb659 feat: session rename UX overhaul
- Edit button (✎) moved to left of row, separated from delete (×)
- Clicking ✎ hides name/meta/delete and expands input to full row width
- Button changes to ✓ (accent color) while editing
- Enter or ✓ click = save; Escape = cancel without saving
- Removed accidental-save-on-blur behavior
- Edit button: 30% opacity at rest, 75% on row hover, 100% on direct hover
- Touch devices: edit button always at 60% opacity (no hover to reveal it)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 19:00:39 -04:00
Scott Idem
508fb638ad feat: distill safeguards — rolling backups + sanity checks
Before any memory file is overwritten, _rotate_backup() keeps 2 rolling
backups: MEMORY_*.bak1.md (most recent) and MEMORY_*.bak2.md (older).

_sanity_check() now also guards against size anomalies: the new content
must be between 40% and 250% of the old file size — anything outside that
range looks like truncation or runaway output and aborts the write.
Existing checks (min length, refusal phrases) still apply.

Backup files exposed in the Files panel (ALLOWED set) so they can be
reviewed and manually restored if needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 18:54:27 -04:00
Scott Idem
0ffcd57c95 fix: multi-user distillation + datetime in context + session log labels
Distillation was silently operating on scott/inara for all users due to
ContextVar defaults. All three distill endpoints now require ?user=&persona=
query params and validate them via persona.validate(). Memory distiller
signatures changed from Optional to required positional args — no more
global settings fallback. Scheduler now iterates all users/personas instead
of hardcoding the primary user.

- context_loader: inject current date/time as first system prompt section
- session_logger: use get_user()/get_persona() from context instead of
  settings globals so Holly/Brian sessions show correct speaker labels
- memory_distiller: system prompts now reference u.title()/p.title()
  instead of settings.user_name/settings.agent_name
- distill router: Query(...) enforces params; _resolve() validates persona
- scheduler: _all_personas() helper iterates every user/persona for distill
- app.js: runDistill() now appends ?user=&persona= via _fileParams

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-05 18:44:51 -04:00
Scott Idem
8d4aa4094c feat: usage tracking + knowledge import script
- usage_tracker.py: daily token/call buckets per user (home/{user}/usage.json)
- Hook into local backend (OpenAI usage field) and Gemini API (usage_metadata)
- Claude/Gemini CLI backends produce no structured token data and are not tracked
- Fix CLAUDE.md stale tool count (27 → 39) and refresh tool list
- scripts/import_knowledge.py: walk markdown dirs, chunk by H2, call local LLM
  for summaries, create AE journal entries with path-derived tags; resumable via
  state file; --dry-run and --limit flags for safe testing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-02 20:38:31 -04:00
Scott Idem
eab92d876d refactor: split tool declarations into domain files + role config UI
tools/__init__.py shrinks from 1,137 → 250 lines. Each domain file now
owns both its callables and its FunctionDeclarations (DECLARATIONS list),
so adding a new tool only touches one file.

New TOOL_CATEGORIES dict exported from __init__ — used by the UI for
grouped tool checkboxes.

Role config UI (Settings → Model Registry → Role Assignments):
- ⚙ button per role expands an inline configure panel
- Textarea for system_append (injected into system prompt for this role)
- Grouped checkboxes for tool allow-list (all checked = no restriction)
- POST /api/models/role-config saves both fields; updates ROLE_CONFIG_DATA
  in-page so re-open reflects current state without a page reload

Backend:
- model_registry.set_role_config() writes system_append + tools to registry
- TOOL_CATEGORIES exported from tools/__init__ for UI rendering
- TOOLS.md header updated: 30 → 39 tools (ae_journal_* and cortex_* additions)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 20:40:50 -04:00
Scott Idem
49123cdd5c feat: per-role tool lists and system prompt overlays
Each role in model_registry.json can now carry two optional keys:
  system_append — injected into the system prompt at position 7 (after
                  memory, closest to the turn) for the active chat_role
  tools         — explicit tool allow-list; intersected with the user's
                  access-level filter so it can only restrict, never elevate

No changes needed for existing users — missing keys fall back to current
behavior. Add keys to a role to give it a specialty focus:

  "coder": {
    "primary": "claude_cli",
    "system_append": "You are in code-specialist mode...",
    "tools": ["web_search", "file_read", "shell_exec", "scratch_write"]
  }

Changes:
- model_registry.py: get_role_config() returns system_append + tools
- context_loader.py: role_append param appended as "--- Role Context ---"
- tools/__init__.py: get_tools_for_role/get_openai_tools_for_role accept
  optional tool_list and intersect with access-level filter
- orchestrator_engine.py: tool_list threaded through run/resume/checkpoint
- openai_orchestrator.py: tool_list threaded through run/resume/checkpoint;
  _build_client now calls get_openai_tools_for_role instead of returning
  unfiltered OPENAI_TOOL_SCHEMAS
- routers/orchestrator.py: pulls role_cfg for chat_role, passes both
  role_append and tool_list to context loader and engine

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-01 20:00:38 -04:00
43 changed files with 2762 additions and 1136 deletions

View File

@@ -226,10 +226,13 @@ Cortex is running and stable. All channels are live:
Active users: scott (inara, developer), holly (tina), brian (wintermute)
**27 orchestrator tools:** web_search, file_read, shell_exec, claude_allow_dir,
**40 orchestrator tools:** web_search, http_fetch,
file_read/list/write, shell_exec, claude_allow_dir,
cortex_restart/logs/status/update,
task_list/create/update/complete, cron_list/add/remove/toggle,
reminders_add/list/clear, scratch_read/write/append/clear,
ae_journal_list/search/entry_create/entry_update/entry_disable/entry_append/entry_prepend,
reminders_add/list/remove/clear, scratch_read/write/append/clear,
web_push, email_send, nc_talk_send,
ae_journal_list/search/entries_list/entry_read/entry_create/entry_update/entry_disable/entry_append/entry_prepend,
ae_task_list.
See `documentation/TODO__Agents.md` for the active task list.

View File

@@ -17,7 +17,8 @@ from starlette.responses import RedirectResponse, JSONResponse
from auth_utils import COOKIE_NAME, decode_token
# Paths that don't require a session cookie
_PUBLIC = {"/login", "/logout", "/health", "/manifest.json", "/sw.js", "/favicon.ico"}
_PUBLIC = {"/login", "/logout", "/health", "/manifest.json", "/sw.js", "/favicon.ico",
"/api/push/vapid-key"}
# Path prefixes that are always public (setup flow + webhooks + Google OAuth)
_PUBLIC_PREFIXES = ("/setup/", "/channels/", "/webhook/", "/auth/google")

View File

@@ -89,6 +89,12 @@ class Settings(BaseSettings):
jwt_secret: str = "change-me-in-dotenv" # override in .env: JWT_SECRET=<random>
jwt_expire_days: int = 30
# Web Push (VAPID) — for browser push notifications
# Generate once with py_vapid; see push_utils.py for key format details
vapid_public_key: str = "" # base64url-encoded uncompressed EC point (for browser)
vapid_private_key_b64: str = "" # base64-encoded PEM private key (single-line .env storage)
vapid_contact: str = "mailto:admin@example.com"
# SMTP — for sending invite emails
smtp_server: str = ""
smtp_port: int = 465

View File

@@ -1,3 +1,4 @@
from datetime import datetime
from pathlib import Path
from persona import persona_path
@@ -17,6 +18,7 @@ def load_context(
include_long: bool = True,
include_mid: bool = True,
include_short: bool = True,
role_append: str = "",
) -> str:
"""
Build the system-prompt context block for a given tier and memory toggles.
@@ -28,10 +30,17 @@ def load_context(
Tier 2 — + USER full + PROTOCOLS + memory (~5,000 tokens)
Tier 3 — + last 2 raw session logs (~15,000 tokens)
Tier 4 — + last 7 raw session logs (~50,000 tokens)
role_append — optional text injected last (closest to the turn),
sourced from the active role's system_append config.
"""
inara_dir = persona_path()
parts = []
# ── 0. Current date and time (always — injected first so it's prominent) ──
now = datetime.now().astimezone()
parts.append(f"--- System ---\nCurrent date and time: {now.strftime('%A, %Y-%m-%d at %I:%M %p %Z')}")
# ── 1. Core identity (always) ──────────────────────────────────
for filename in _CORE:
path = inara_dir / filename
@@ -107,4 +116,8 @@ def load_context(
for sf in session_files:
parts.append(f"--- Session: {sf.name} ---\n{sf.read_text()}")
# ── 7. Role-specific instructions (always last — closest to the turn) ──
if role_append and role_append.strip():
parts.append(f"--- Role Context ---\n{role_append.strip()}")
return "\n\n".join(parts)

View File

@@ -218,6 +218,19 @@ async def _local(system_prompt: str, messages: list[dict], model_cfg: dict | Non
text = data["choices"][0]["message"]["content"]
if not text or not text.strip():
raise RuntimeError("Local model returned an empty response")
usage = data.get("usage") or {}
if usage.get("prompt_tokens") is not None:
import usage_tracker
from persona import _user
asyncio.create_task(usage_tracker.record(
username=_user.get(),
backend="local",
model_name=model,
prompt_tokens=usage.get("prompt_tokens", 0),
completion_tokens=usage.get("completion_tokens", 0),
))
return text.strip()

View File

@@ -9,7 +9,7 @@ logging.basicConfig(level=logging.INFO, format="%(levelname)s:%(name)s: %(messag
from config import settings
from auth_middleware import SessionAuthMiddleware
from routers import chat, google_chat, nextcloud_talk, files, distill, auth, orchestrator
from routers import ui, onboarding, settings, help, auth_google, local_llm
from routers import ui, onboarding, settings, help, auth_google, local_llm, push, audit
@asynccontextmanager
@@ -34,6 +34,8 @@ app.include_router(files.router)
app.include_router(distill.router)
app.include_router(auth.router)
app.include_router(orchestrator.router)
app.include_router(push.router)
app.include_router(audit.router)
# Static files — must be mounted BEFORE ui.router so /static/* is matched first.
# ui.router has a wildcard /{username}/{persona} that would otherwise catch /static/style.css etc.

View File

@@ -1,9 +1,17 @@
"""
Inara tiered memory distillation.
Tiered memory distillation.
distill_short() — roll recent session logs → MEMORY_SHORT.md (no LLM)
distill_mid() — summarize MEMORY_SHORT → MEMORY_MID.md (LLM)
distill_long() — integrate MEMORY_MID → MEMORY_LONG.md (LLM)
Before any file is overwritten, two rolling backups are kept:
MEMORY_*.bak1.md — most recent backup (created just before last write)
MEMORY_*.bak2.md — backup before that
LLM responses are sanity-checked before writing. If the response looks like
a refusal, is too short, or is obviously not memory content, the distill is
aborted and the original file is left untouched.
"""
import logging
from datetime import datetime
@@ -16,6 +24,25 @@ logger = logging.getLogger(__name__)
# Rough chars-per-token estimate for budget enforcement
_CHARS_PER_TOKEN = 4
# Phrases that indicate the LLM refused or misunderstood the task
_REFUSAL_PREFIXES = (
"i'm sorry",
"i am sorry",
"i can't",
"i cannot",
"i'm unable",
"i am unable",
"as an ai",
"as a language model",
"i don't have access",
"i do not have access",
"i'm not able",
"i am not able",
)
# Minimum characters for a valid mid/long distill response
_MIN_RESPONSE_CHARS = 80
def _budget_chars(tokens: int) -> int:
return tokens * _CHARS_PER_TOKEN
@@ -25,7 +52,62 @@ def _read(path: Path) -> str:
return path.read_text() if path.exists() else ""
def distill_short(username: str | None = None, persona: str | None = None) -> dict:
def _rotate_backup(path: Path, n: int = 2) -> None:
"""Rotate up to n rolling backups of path before a write.
MEMORY_LONG.md → MEMORY_LONG.bak1.md (most recent), MEMORY_LONG.bak2.md (older)
"""
if not path.exists():
return
# Shift older backups down: bak(n-1) → bak(n), …, bak1 stays as bak1 source
for i in range(n, 1, -1):
older = path.parent / f"{path.stem}.bak{i}.md"
newer = path.parent / f"{path.stem}.bak{i - 1}.md"
if newer.exists():
older.write_text(newer.read_text())
# Current file → bak1
bak1 = path.parent / f"{path.stem}.bak1.md"
bak1.write_text(path.read_text())
def _sanity_check(response_text: str, context: str, existing_content: str = "") -> str | None:
"""Return an error string if the LLM response looks invalid, else None.
Checks:
- Minimum absolute length
- Refusal / AI preamble phrases
- Size shrinkage: new content must be at least 40% of the old (catches truncation)
- Size explosion: new content must not exceed 250% of the old (catches runaway output)
(Both bounds only apply when an existing file is present and reasonably sized.)
"""
stripped = response_text.strip()
if len(stripped) < _MIN_RESPONSE_CHARS:
return f"{context}: response too short ({len(stripped)} chars) — not writing"
first_line = stripped.lower().splitlines()[0]
if any(first_line.startswith(p) for p in _REFUSAL_PREFIXES):
return f"{context}: response looks like a refusal — not writing"
if existing_content:
old_len = len(existing_content.strip())
new_len = len(stripped)
if old_len >= _MIN_RESPONSE_CHARS * 4: # only compare when old file has real content
ratio = new_len / old_len
if ratio < 0.40:
return (
f"{context}: new content is only {ratio:.0%} of the old "
f"({new_len} vs {old_len} chars) — looks truncated, not writing"
)
if ratio > 2.50:
return (
f"{context}: new content is {ratio:.0%} of the old "
f"({new_len} vs {old_len} chars) — looks like runaway output, not writing"
)
return None
def distill_short(username: str, persona: str) -> dict:
"""
Roll the most recent session log files into MEMORY_SHORT.md.
No LLM involved — pure aggregation with budget truncation.
@@ -64,8 +146,9 @@ def distill_short(username: str | None = None, persona: str | None = None) -> di
)
out_path = inara_dir / "MEMORY_SHORT.md"
_rotate_backup(out_path)
out_path.write_text(header + body)
logger.info("distill_short: wrote %d chars from %d files", len(header) + len(body), len(parts))
logger.info("distill_short [%s/%s]: wrote %d chars from %d files", username, persona, len(header) + len(body), len(parts))
return {
"files_included": len(parts),
@@ -74,32 +157,34 @@ def distill_short(username: str | None = None, persona: str | None = None) -> di
}
async def distill_mid(username: str | None = None, persona: str | None = None) -> dict:
async def distill_mid(username: str, persona: str) -> dict:
"""
Ask the LLM to summarize MEMORY_SHORT.md → MEMORY_MID.md.
Uses DISTILL_BACKEND_MID if set (e.g. "local"), otherwise primary_backend.
Backs up the current MEMORY_MID.md before overwriting.
"""
from llm_client import complete
from persona import set_context
u = username or settings.user_name.lower()
p = persona or settings.agent_name.lower()
u, p = username, persona
set_context(u, p)
inara_dir = _persona_path(u, p)
short_content = _read(inara_dir / "MEMORY_SHORT.md")
existing_mid = _read(inara_dir / "MEMORY_MID.md")
if not short_content.strip() or "Not yet populated" in short_content:
return {"error": "MEMORY_SHORT.md is empty — run distill/short first"}
budget_tokens = settings.memory_budget_mid
persona_name = p.title()
user_name = u.title()
system_prompt = (
f"You are {settings.agent_name}'s memory distillation system. "
f"You are {persona_name}'s memory distillation system. "
"Summarize the following recent session logs into a concise mid-term memory digest. "
f"Target length: under {budget_tokens} tokens. "
"Focus on: recurring themes, important decisions made, ongoing projects, "
f"{settings.user_name}'s current state and priorities, and anything that should persist into future sessions. "
f"Write in first person as {settings.agent_name} (e.g. '{settings.user_name} and I worked on...'). "
f"{user_name}'s current state and priorities, and anything that should persist into future sessions. "
f"Write in first person as {persona_name} (e.g. '{user_name} and I worked on...'). "
"Use markdown headings. Be specific and concrete — no filler."
)
@@ -109,14 +194,20 @@ async def distill_mid(username: str | None = None, persona: str | None = None) -
role="distill",
)
err = _sanity_check(response_text, "distill_mid", existing_mid)
if err:
logger.warning(err)
return {"error": err}
now = datetime.now().strftime("%Y-%m-%d %H:%M")
header = (
f"# MEMORY_MID.md — Mid-Term Memory Digest\n\n"
f"*Auto-distilled: {now} via {backend}.*\n\n---\n\n"
)
out_path = inara_dir / "MEMORY_MID.md"
_rotate_backup(out_path)
out_path.write_text(header + response_text)
logger.info("distill_mid: wrote %d chars via %s", len(header) + len(response_text), backend)
logger.info("distill_mid [%s/%s]: wrote %d chars via %s", u, p, len(header) + len(response_text), backend)
return {
"username": u,
@@ -126,16 +217,15 @@ async def distill_mid(username: str | None = None, persona: str | None = None) -
}
async def distill_long(username: str | None = None, persona: str | None = None) -> dict:
async def distill_long(username: str, persona: str) -> dict:
"""
Ask the LLM to integrate MEMORY_MID.md into MEMORY_LONG.md.
Uses DISTILL_BACKEND_LONG if set, otherwise primary_backend.
Backs up the current MEMORY_LONG.md before overwriting.
"""
from llm_client import complete
from persona import set_context
u = username or settings.user_name.lower()
p = persona or settings.agent_name.lower()
u, p = username, persona
set_context(u, p)
inara_dir = _persona_path(u, p)
@@ -146,8 +236,9 @@ async def distill_long(username: str | None = None, persona: str | None = None)
return {"error": "MEMORY_MID.md is empty — run distill/mid first"}
budget_tokens = settings.memory_budget_long
persona_name = p.title()
system_prompt = (
f"You are {settings.agent_name}'s long-term memory curator. "
f"You are {persona_name}'s long-term memory curator. "
"You will receive the current long-term memory and a recent mid-term digest. "
f"Integrate the new information into the long-term memory. Target: under {budget_tokens} tokens. "
"Rules: preserve important historical facts; update or replace stale information; "
@@ -166,18 +257,24 @@ async def distill_long(username: str | None = None, persona: str | None = None)
role="distill",
)
err = _sanity_check(response_text, "distill_long", long_content)
if err:
logger.warning(err)
return {"error": err}
# Ensure the file has the right header if the LLM dropped it
now = datetime.now().strftime("%Y-%m-%d %H:%M")
if not response_text.lstrip().startswith("# MEMORY_LONG"):
response_text = (
f"# MEMORY_LONG.md — {settings.agent_name} Long-Term Memory\n\n"
f"# MEMORY_LONG.md — {persona_name} Long-Term Memory\n\n"
f"*Last distilled: {now} via {backend}.*\n\n---\n\n"
+ response_text
)
out_path = inara_dir / "MEMORY_LONG.md"
_rotate_backup(out_path)
out_path.write_text(response_text)
logger.info("distill_long: wrote %d chars via %s", len(response_text), backend)
logger.info("distill_long [%s/%s]: wrote %d chars via %s", u, p, len(response_text), backend)
return {
"username": u,

View File

@@ -415,6 +415,40 @@ def get_best_local_model(username: str, role: str = "chat") -> dict | None:
return None
def set_role_config(username: str, role: str, system_append: str, tools: list[str] | None) -> None:
"""Save system_append and tools allow-list for a role.
tools=None clears the allow-list (role uses all accessible tools).
tools=[] would mean no tools at all — validate in the caller if that's undesired.
"""
data = _load(username)
roles = data.setdefault("roles", {})
if role not in roles:
roles[role] = {}
roles[role]["system_append"] = system_append.strip()
if tools is None:
roles[role].pop("tools", None)
else:
roles[role]["tools"] = [t for t in tools if t]
_save(username, data)
def get_role_config(username: str, role: str) -> dict:
"""
Return supplemental config for a role: system_append and tools.
Both keys are optional in the registry — missing means "use defaults":
system_append: str — appended to the system prompt for this role
tools: list[str] | None — explicit tool allow-list (None = no restriction)
"""
registry = _load(username)
role_cfg = registry.get("roles", {}).get(role, {})
return {
"system_append": role_cfg.get("system_append", ""),
"tools": role_cfg.get("tools") or None,
}
def get_model_for_slot(username: str, role: str, slot: str) -> dict | None:
"""
Resolve a single named priority slot from a role without walking the fallback chain.

View File

@@ -45,6 +45,7 @@ async def run(
model_cfg: dict | None = None,
respond_with_final: bool = True,
user_role: str = "user",
tool_list: list[str] | None = None,
confirm_allow: set[str] | None = None,
confirm_deny: set[str] | None = None,
) -> OrchestratorResult:
@@ -71,7 +72,7 @@ async def run(
_confirm_deny = frozenset(confirm_deny or ())
effective_confirm = (CONFIRM_REQUIRED - set(_confirm_allow)) | set(_confirm_deny)
client, model_name, active_tools = _build_client(model_cfg)
client, model_name, active_tools = _build_client(model_cfg, user_role, tool_list)
sys_content = (system_prompt or "") + _TOOL_INSTRUCTION
messages: list[dict] = [{"role": "system", "content": sys_content}]
@@ -95,6 +96,7 @@ async def run(
model_cfg=model_cfg,
respond_with_final=respond_with_final,
user_role=user_role,
tool_list=tool_list,
confirm_allow=_confirm_allow,
confirm_deny=_confirm_deny,
starting_round=0,
@@ -121,7 +123,7 @@ async def run(
async def resume(checkpoint: OrchestrateCheckpoint, confirmed: bool) -> OrchestratorResult:
"""Continue an OpenAI orchestrator job that was paused at a confirmation gate."""
client, model_name, active_tools = _build_client(checkpoint.model_cfg)
client, model_name, active_tools = _build_client(checkpoint.model_cfg, checkpoint.user_role, checkpoint.tool_list)
effective_confirm = (CONFIRM_REQUIRED - set(checkpoint.confirm_allow)) | set(checkpoint.confirm_deny)
@@ -138,8 +140,7 @@ async def resume(checkpoint: OrchestrateCheckpoint, confirmed: bool) -> Orchestr
for pt in checkpoint.pending_tools:
if confirmed:
_, callables = get_tools_for_role(checkpoint.user_role)
result_str = await _execute_tool_dict(pt["name"], pt["args"], checkpoint.user_role)
result_str = await _execute_tool_dict(pt["name"], pt["args"], checkpoint.user_role, checkpoint.tool_list)
logger.info("Confirmed tool %s%d chars", pt["name"], len(result_str))
else:
result_str = "Action denied by user."
@@ -162,6 +163,7 @@ async def resume(checkpoint: OrchestrateCheckpoint, confirmed: bool) -> Orchestr
model_cfg=checkpoint.model_cfg,
respond_with_final=checkpoint.respond_with_final,
user_role=checkpoint.user_role,
tool_list=checkpoint.tool_list,
confirm_allow=checkpoint.confirm_allow,
confirm_deny=checkpoint.confirm_deny,
starting_round=checkpoint.rounds_used,
@@ -200,6 +202,7 @@ async def _run_from_messages(
confirm_allow: frozenset,
confirm_deny: frozenset,
starting_round: int = 0,
tool_list: list[str] | None = None,
) -> tuple[str, OrchestrateCheckpoint | None]:
"""
Run the OpenAI ReAct loop from the current messages state.
@@ -253,7 +256,7 @@ async def _run_from_messages(
pending_tools.append({"name": name, "args": args_parsed, "tool_call_id": tc.id})
logger.info("Tool %s blocked — confirmation required", name)
else:
result_str = await _execute_tool(name, tc.function.arguments, user_role)
result_str = await _execute_tool(name, tc.function.arguments, user_role, tool_list)
logger.info("Tool %s%d chars", name, len(result_str))
executed_results.append({"name": name, "args": args_parsed, "result": result_str, "tool_call_id": tc.id})
tool_call_log.append({"tool": name, "args": args_parsed, "result": result_str})
@@ -286,6 +289,7 @@ async def _run_from_messages(
model_cfg=model_cfg,
respond_with_final=respond_with_final,
user_role=user_role,
tool_list=tool_list,
confirm_allow=confirm_allow,
confirm_deny=confirm_deny,
rounds_used=round_num + 2,
@@ -311,7 +315,11 @@ async def _run_from_messages(
return final_response, None
def _build_client(model_cfg: dict | None) -> tuple:
def _build_client(
model_cfg: dict | None,
user_role: str = "user",
tool_list: list[str] | None = None,
) -> tuple:
"""Build AsyncOpenAI client and return (client, model_name, active_tools)."""
if not model_cfg:
raise RuntimeError("model_cfg is required for the OpenAI orchestrator")
@@ -327,12 +335,18 @@ def _build_client(model_cfg: dict | None) -> tuple:
if host_type == "openwebui":
base_url = base_url + "/api"
client = AsyncOpenAI(base_url=base_url, api_key=api_key)
return client, model_name, OPENAI_TOOL_SCHEMAS
active_tools = get_openai_tools_for_role(user_role, tool_list)
return client, model_name, active_tools
async def _execute_tool(name: str, arguments_json: str, user_role: str = "user") -> str:
async def _execute_tool(
name: str,
arguments_json: str,
user_role: str = "user",
tool_list: list[str] | None = None,
) -> str:
"""Parse tool arguments and execute with role-filtered callables."""
_, callables = get_tools_for_role(user_role)
_, callables = get_tools_for_role(user_role, tool_list)
try:
args = json.loads(arguments_json)
except json.JSONDecodeError:
@@ -344,9 +358,14 @@ async def _execute_tool(name: str, arguments_json: str, user_role: str = "user")
return f"Tool error: {e}"
async def _execute_tool_dict(name: str, args: dict, user_role: str = "user") -> str:
async def _execute_tool_dict(
name: str,
args: dict,
user_role: str = "user",
tool_list: list[str] | None = None,
) -> str:
"""Execute a tool from a pre-parsed args dict."""
_, callables = get_tools_for_role(user_role)
_, callables = get_tools_for_role(user_role, tool_list)
try:
return await call_tool(name, args, callables)
except Exception as e:

View File

@@ -26,6 +26,8 @@ from google.genai import types
from config import settings
from llm_client import complete
from tools import TOOL_DECLARATIONS, call_tool, get_tools_for_role, CONFIRM_REQUIRED
import usage_tracker
from persona import _user
logger = logging.getLogger(__name__)
@@ -44,6 +46,25 @@ Keep your summary factual and complete. Include relevant URLs, data, and specifi
If no tools are needed, return an empty summary."""
def _track_gemini_usage(response, model_name: str | None) -> None:
meta = getattr(response, "usage_metadata", None)
if not meta:
return
prompt_tokens = getattr(meta, "prompt_token_count", 0) or 0
completion_tokens = getattr(meta, "candidates_token_count", 0) or 0
if prompt_tokens or completion_tokens:
try:
asyncio.create_task(usage_tracker.record(
username=_user.get(),
backend="gemini_api",
model_name=model_name or settings.orchestrator_model,
prompt_tokens=prompt_tokens,
completion_tokens=completion_tokens,
))
except Exception:
pass
@dataclass
class OrchestrateCheckpoint:
"""Saved execution state for a job paused at a confirmation gate."""
@@ -65,6 +86,7 @@ class OrchestrateCheckpoint:
respond_with_final: bool = True
# Common
user_role: str = "user"
tool_list: list[str] | None = None
confirm_allow: frozenset = field(default_factory=frozenset)
confirm_deny: frozenset = field(default_factory=frozenset)
rounds_used: int = 0
@@ -88,6 +110,7 @@ async def run(
model_name: str | None = None,
response_role: str = "chat",
user_role: str = "user",
tool_list: list[str] | None = None,
confirm_allow: set[str] | None = None,
confirm_deny: set[str] | None = None,
) -> OrchestratorResult:
@@ -101,6 +124,8 @@ async def run(
respond_with_claude: If False, return Gemini's summary as the response (useful for
background/cron tasks where a polished reply isn't needed)
gemini_api_key: Per-user Gemini API key (falls back to GEMINI_API_KEY in .env)
tool_list: Optional explicit tool allow-list from role config; intersected
with user_role access-level filter (cannot elevate privileges)
confirm_allow: Tools to bypass the confirmation gate for this user
confirm_deny: Tools to always block for this user
@@ -124,7 +149,7 @@ async def run(
contents: list[types.Content] = [
types.Content(role="user", parts=[types.Part(text=task_with_context)])
]
tool_declarations, tool_callables = get_tools_for_role(user_role)
tool_declarations, tool_callables = get_tools_for_role(user_role, tool_list)
tool_call_log: list[dict] = []
gemini_summary, checkpoint = await _run_from_contents(
@@ -141,6 +166,7 @@ async def run(
respond_with_claude=respond_with_claude,
response_role=response_role,
user_role=user_role,
tool_list=tool_list,
confirm_allow=_confirm_allow,
confirm_deny=_confirm_deny,
starting_round=0,
@@ -171,7 +197,7 @@ async def resume(checkpoint: OrchestrateCheckpoint, confirmed: bool) -> Orchestr
"""Continue a job that was paused at a confirmation gate."""
api_key = checkpoint.gemini_api_key or settings.gemini_api_key
client = genai.Client(api_key=api_key)
tool_declarations, tool_callables = get_tools_for_role(checkpoint.user_role)
tool_declarations, tool_callables = get_tools_for_role(checkpoint.user_role, checkpoint.tool_list)
effective_confirm = (CONFIRM_REQUIRED - set(checkpoint.confirm_allow)) | set(checkpoint.confirm_deny)
@@ -215,6 +241,7 @@ async def resume(checkpoint: OrchestrateCheckpoint, confirmed: bool) -> Orchestr
respond_with_claude=checkpoint.respond_with_claude,
response_role=checkpoint.response_role,
user_role=checkpoint.user_role,
tool_list=checkpoint.tool_list,
confirm_allow=checkpoint.confirm_allow,
confirm_deny=checkpoint.confirm_deny,
starting_round=checkpoint.rounds_used,
@@ -259,6 +286,7 @@ async def _run_from_contents(
confirm_deny: frozenset,
starting_round: int = 0,
gemini_api_key: str | None = None,
tool_list: list[str] | None = None,
) -> tuple[str, OrchestrateCheckpoint | None]:
"""
Run the ReAct loop from the current contents state.
@@ -278,6 +306,7 @@ async def _run_from_contents(
system_instruction=_ORCHESTRATOR_SYSTEM,
),
)
_track_gemini_usage(response, model_name)
candidate = response.candidates[0]
parts = candidate.content.parts if candidate.content else []
@@ -341,6 +370,7 @@ async def _run_from_contents(
system_instruction=_ORCHESTRATOR_SYSTEM,
),
)
_track_gemini_usage(conf_response, model_name)
conf_parts = (
conf_response.candidates[0].content.parts
if conf_response.candidates and conf_response.candidates[0].content
@@ -364,6 +394,7 @@ async def _run_from_contents(
respond_with_claude=respond_with_claude,
response_role=response_role,
user_role=user_role,
tool_list=tool_list,
confirm_allow=confirm_allow,
confirm_deny=confirm_deny,
rounds_used=round_num + 2,

115
cortex/push_utils.py Normal file
View File

@@ -0,0 +1,115 @@
"""
Web Push (VAPID) helpers.
Subscriptions are stored per-user at:
home/{user}/push_subscriptions.json → list of {endpoint, keys:{p256dh, auth}}
send_push(username, title, body, url) iterates all stored subscriptions for that
user and fires a push. Stale endpoints (410 Gone) are pruned automatically.
"""
import asyncio
import base64
import json
import logging
from pathlib import Path
from config import settings
logger = logging.getLogger(__name__)
def _subs_path(username: str) -> Path:
return settings.home_root() / username / "push_subscriptions.json"
def load_subscriptions(username: str) -> list[dict]:
path = _subs_path(username)
if not path.exists():
return []
try:
return json.loads(path.read_text())
except Exception:
return []
def _save_subscriptions(username: str, subs: list[dict]) -> None:
path = _subs_path(username)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(subs, indent=2))
def add_subscription(username: str, sub: dict) -> None:
"""Upsert a subscription by endpoint URL."""
subs = load_subscriptions(username)
endpoint = sub.get("endpoint", "")
subs = [s for s in subs if s.get("endpoint") != endpoint]
subs.append(sub)
_save_subscriptions(username, subs)
def remove_subscription(username: str, endpoint: str) -> bool:
subs = load_subscriptions(username)
new_subs = [s for s in subs if s.get("endpoint") != endpoint]
if len(new_subs) == len(subs):
return False
_save_subscriptions(username, new_subs)
return True
def _get_private_key_pem() -> str:
"""Decode the base64-encoded PEM private key from settings."""
raw = settings.vapid_private_key_b64.strip()
if not raw:
raise RuntimeError("VAPID_PRIVATE_KEY_B64 is not set in .env")
return base64.b64decode(raw).decode()
def _send_one(sub: dict, payload: dict) -> bool:
"""Send a push to a single subscription. Returns False if the endpoint is stale (410)."""
from pywebpush import webpush, WebPushException
try:
webpush(
subscription_info=sub,
data=json.dumps(payload),
vapid_private_key=_get_private_key_pem(),
vapid_claims={"sub": settings.vapid_contact},
)
return True
except WebPushException as e:
if e.response is not None and e.response.status_code == 410:
logger.info("push endpoint gone (410), pruning: %s", sub.get("endpoint", "")[:60])
return False
logger.warning("push failed: %s", e)
return True # keep the sub; might be transient
async def send_push(username: str, title: str, body: str, url: str = "") -> dict:
"""
Send a push notification to all subscriptions for username.
Returns {"sent": n, "pruned": m}.
"""
if not settings.vapid_public_key or not settings.vapid_private_key_b64:
return {"error": "VAPID keys not configured"}
subs = load_subscriptions(username)
if not subs:
return {"error": f"No push subscriptions for {username}"}
payload = {"title": title, "body": body, "url": url}
keep = []
sent = 0
pruned = 0
for sub in subs:
alive = await asyncio.to_thread(_send_one, sub, payload)
if alive:
keep.append(sub)
sent += 1
else:
pruned += 1
if pruned:
_save_subscriptions(username, keep)
return {"sent": sent, "pruned": pruned}

View File

@@ -22,5 +22,8 @@ httpx>=0.27.0
# OpenAI-compatible client — tool calling for OpenRouter / LiteLLM / any OAI-compat host
openai>=1.0.0
# Web Push / VAPID — browser push notifications
pywebpush>=2.0.0
# anthropic SDK not needed — using claude CLI subprocess for auth
# anthropic>=0.40.0

122
cortex/routers/audit.py Normal file
View File

@@ -0,0 +1,122 @@
"""
Tool audit log endpoints.
Self-service (any authenticated user, own data):
GET /api/audit/files → list of available date strings (newest first)
GET /api/audit/day?date=YYYY-MM-DD → entries for one day
Admin-only (cross-user aggregation):
GET /api/audit/recent?user=scott&days=7&limit=200
GET /api/audit/stats?user=scott&days=7
"""
import jwt
from collections import Counter
from datetime import date, timedelta
from fastapi import APIRouter, HTTPException, Query, Request
from auth_utils import COOKIE_NAME, decode_token, get_user_role
from config import settings
import tool_audit
from persona import list_users
router = APIRouter(prefix="/api/audit")
def _session_user(request: Request) -> str:
"""Return the authenticated username or raise 401."""
token = request.cookies.get(COOKIE_NAME)
if not token:
raise HTTPException(status_code=401, detail="Not authenticated")
try:
return decode_token(token)
except jwt.InvalidTokenError:
raise HTTPException(status_code=401, detail="Invalid session")
def _require_admin(request: Request) -> str:
username = _session_user(request)
if get_user_role(username) != "admin":
raise HTTPException(status_code=403, detail="Admin access required")
return username
@router.get("/files")
async def audit_files(request: Request) -> dict:
"""List available audit log dates for the current user (newest first)."""
username = _session_user(request)
audit_dir = settings.home_root() / username / "tool_audit"
if not audit_dir.exists():
return {"dates": []}
dates = sorted(
[p.stem for p in audit_dir.glob("*.jsonl") if p.stem],
reverse=True,
)
return {"dates": dates}
@router.get("/day")
async def audit_day(
request: Request,
date: str = Query(..., description="YYYY-MM-DD"),
) -> dict:
"""Return all entries for a specific day (current user only)."""
username = _session_user(request)
try:
from datetime import date as _date
d = _date.fromisoformat(date)
except ValueError:
raise HTTPException(status_code=400, detail="date must be YYYY-MM-DD")
entries = tool_audit.read_day(username, date)
return {"date": date, "entries": entries}
@router.get("/recent")
async def audit_recent(
request: Request,
user: str = Query(None, description="Username to filter (omit for all users)"),
days: int = Query(7, ge=1, le=90),
limit: int = Query(200, ge=1, le=1000),
) -> dict:
_require_admin(request)
if user:
if user not in list_users():
raise HTTPException(status_code=404, detail=f"User not found: {user}")
entries = tool_audit.read_recent(user, days=days, limit=limit)
else:
entries = tool_audit.read_recent_all_users(days=days, limit=limit)
return {"entries": entries, "count": len(entries), "days": days}
@router.get("/stats")
async def audit_stats(
request: Request,
user: str = Query(None),
days: int = Query(7, ge=1, le=90),
) -> dict:
_require_admin(request)
if user:
if user not in list_users():
raise HTTPException(status_code=404, detail=f"User not found: {user}")
entries = tool_audit.read_recent(user, days=days, limit=10000)
else:
entries = tool_audit.read_recent_all_users(days=days, limit=10000)
tool_counts: Counter = Counter()
status_counts: Counter = Counter()
user_counts: Counter = Counter()
for e in entries:
tool_counts[e.get("tool", "?")] += 1
status_counts[e.get("status", "?")] += 1
user_counts[e.get("user", "?")] += 1
return {
"total": len(entries),
"days": days,
"by_tool": dict(tool_counts.most_common()),
"by_status": dict(status_counts),
"by_user": dict(user_counts.most_common()),
}

View File

@@ -8,7 +8,7 @@ from pydantic import BaseModel
from context_loader import load_context
from llm_client import complete
from session_logger import log_turn
from session_store import load as load_session, save as save_session, list_all, generate_session_id, delete as delete_session, rename as rename_session
from session_store import load as load_session, save as save_session, list_all, generate_session_id, delete as delete_session, rename as rename_session, get_name as get_session_name
from config import settings
from persona import set_context, validate as validate_persona
from auth_utils import COOKIE_NAME, decode_token
@@ -264,7 +264,8 @@ async def get_history(
persona: str = Query("inara"),
) -> dict:
_set_ctx(user, persona)
return {"session_id": session_id, "messages": load_session(session_id)}
name = get_session_name(session_id)
return {"session_id": session_id, "name": name, "messages": load_session(session_id)}
@router.get("/sessions")

View File

@@ -5,14 +5,29 @@ Manual memory distillation endpoints.
POST /distill/mid — summarize short → MEMORY_MID.md (LLM)
POST /distill/long — integrate mid → MEMORY_LONG.md (LLM)
POST /distill/all — run all three in sequence
All endpoints require ?user=<username>&persona=<name> query params so distillation
targets the correct persona. Without them, the request is rejected (no silent fallback
to server defaults — that caused wrong-user distillation in a multi-user setup).
"""
from fastapi import APIRouter
from fastapi import APIRouter, HTTPException, Query
from memory_distiller import distill_short, distill_mid, distill_long
from persona import validate as validate_persona, set_context
import scheduler
router = APIRouter(prefix="/distill")
def _resolve(user: str, persona: str) -> tuple[str, str]:
"""Validate and set persona context. Raises 404 if the persona doesn't exist."""
try:
u, p = validate_persona(user, persona)
except Exception:
raise HTTPException(status_code=404, detail=f"Persona not found: {user}/{persona}")
set_context(u, p)
return u, p
@router.get("/status")
async def distill_status() -> dict:
"""Show auto-distillation schedule and next run times."""
@@ -29,29 +44,45 @@ async def distill_status() -> dict:
@router.post("/short")
async def do_distill_short() -> dict:
return {"ok": True, **distill_short()}
async def do_distill_short(
user: str = Query(...),
persona: str = Query(...),
) -> dict:
u, p = _resolve(user, persona)
return {"ok": True, **distill_short(u, p)}
@router.post("/mid")
async def do_distill_mid() -> dict:
result = await distill_mid()
async def do_distill_mid(
user: str = Query(...),
persona: str = Query(...),
) -> dict:
u, p = _resolve(user, persona)
result = await distill_mid(u, p)
return {"ok": "error" not in result, **result}
@router.post("/long")
async def do_distill_long() -> dict:
result = await distill_long()
async def do_distill_long(
user: str = Query(...),
persona: str = Query(...),
) -> dict:
u, p = _resolve(user, persona)
result = await distill_long(u, p)
return {"ok": "error" not in result, **result}
@router.post("/all")
async def do_distill_all() -> dict:
short_result = distill_short()
mid_result = await distill_mid()
async def do_distill_all(
user: str = Query(...),
persona: str = Query(...),
) -> dict:
u, p = _resolve(user, persona)
short_result = distill_short(u, p)
mid_result = await distill_mid(u, p)
if "error" in mid_result:
return {"ok": False, "short": short_result, "mid": mid_result}
long_result = await distill_long()
long_result = await distill_long(u, p)
return {
"ok": "error" not in long_result,
"short": short_result,

View File

@@ -16,10 +16,16 @@ ALLOWED = {
"USER.md",
"PROTOCOLS.md",
"CONTEXT_TIERS.md",
"MEMORY.md", # legacy — kept for reference
"MEMORY.md", # legacy — kept for reference
"MEMORY_LONG.md",
"MEMORY_MID.md",
"MEMORY_SHORT.md",
"MEMORY_LONG.bak1.md",
"MEMORY_LONG.bak2.md",
"MEMORY_MID.bak1.md",
"MEMORY_MID.bak2.md",
"MEMORY_SHORT.bak1.md",
"MEMORY_SHORT.bak2.md",
"HELP.md",
}

View File

@@ -26,6 +26,7 @@ from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse
from auth_utils import COOKIE_NAME, decode_token
from config import settings as app_settings
import model_registry as reg
from tools import TOOL_CATEGORIES
logger = logging.getLogger(__name__)
router = APIRouter()
@@ -285,13 +286,42 @@ def _render(username: str, success: str = "", error: str = "") -> str:
f'data-slot="{slot}" title="{slot_label}">\n{model_opts}\n</select>'
)
role_rows += f'<div class="role-slot"><span class="slot-label">{slot_label}</span>{sel}</div>'
role_rows += '</div></div>'
role_rows += (
f'</div>'
f'<button class="role-cfg-btn" data-role="{role}" title="Configure persona and tools">⚙</button>'
f'</div>'
f'<div class="role-config-panel" id="rcp-{role}">'
f'<div class="rcp-field">'
f'<label class="rcp-label">System prompt addition</label>'
f'<textarea class="rcp-textarea" data-role="{role}" rows="3" '
f'placeholder="Extra instructions injected into the system prompt when this role is active…"></textarea>'
f'</div>'
f'<div class="rcp-field">'
f'<label class="rcp-label">Tool allow-list '
f'<span class="rcp-hint">— all checked means no restriction (use all accessible tools)</span></label>'
f'<div class="rcp-tools" id="rcp-tools-{role}"></div>'
f'</div>'
f'<div class="rcp-actions">'
f'<button class="btn btn-primary btn-sm rcp-save" data-role="{role}">Save</button>'
f'<button class="btn btn-secondary btn-sm rcp-cancel" data-role="{role}">Cancel</button>'
f'</div>'
f'</div>'
)
role_data_js = _json.dumps({
role: {slot: (roles.get(role, {}).get(slot) or "") for slot in reg.PRIORITY_KEYS[:3]}
for role in app_settings.get_defined_roles()
})
role_config_data_js = _json.dumps({
role: {
"system_append": roles.get(role, {}).get("system_append", ""),
"tools": roles.get(role, {}).get("tools") or None,
}
for role in app_settings.get_defined_roles()
})
tool_categories_js = _json.dumps(TOOL_CATEGORIES)
# ── Catalog data + Google accounts for JS ─────────────────────────────────
google_accounts_js = _json.dumps(reg.get_google_accounts(username))
google_catalog_js = _json.dumps(reg.get_catalog("google"))
@@ -305,8 +335,10 @@ def _render(username: str, success: str = "", error: str = "") -> str:
"{{ host_rows }}": host_rows,
"{{ model_rows }}": model_rows,
"{{ host_options }}": host_options,
"{{ role_rows }}": role_rows,
"{{ role_data_js }}": role_data_js,
"{{ role_rows }}": role_rows,
"{{ role_data_js }}": role_data_js,
"{{ role_config_data_js }}": role_config_data_js,
"{{ tool_categories_js }}": tool_categories_js,
"{{ google_accounts_js }}": google_accounts_js,
"{{ google_catalog_js }}": google_catalog_js,
"{{ anthropic_catalog_js }}": anthropic_catalog_js,
@@ -510,6 +542,36 @@ async def set_role(request: Request) -> JSONResponse:
return JSONResponse({"ok": True})
@router.post("/api/models/role-config")
async def set_role_config(request: Request) -> JSONResponse:
"""AJAX: save system_append and tool allow-list for a role.
Body: {"role": "coder", "system_append": "...", "tools": ["web_search", ...] | null}
tools=null clears the allow-list (role uses all accessible tools).
"""
username = _get_user(request)
if not username:
return JSONResponse({"error": "Not authenticated"}, status_code=401)
try:
body = await request.json()
except Exception:
return JSONResponse({"error": "Invalid JSON"}, status_code=400)
role = body.get("role", "").strip()
system_append = body.get("system_append", "")
tools = body.get("tools") # list[str] or None
if not role:
return JSONResponse({"error": "role is required"}, status_code=400)
if tools is not None and not isinstance(tools, list):
return JSONResponse({"error": "tools must be a list or null"}, status_code=400)
reg.set_role_config(username, role, system_append, tools)
logger.info("role config saved: %s %s (tools=%s)", username, role,
len(tools) if tools is not None else "all")
return JSONResponse({"ok": True})
@router.get("/api/local-llm/fetch-models")
async def fetch_models(request: Request, host_id: str = "") -> JSONResponse:
"""Proxy to the host's models endpoint. host_id selects which host."""

View File

@@ -196,11 +196,13 @@ async def _run_job(job_id: str, req: OrchestrateRequest, user: str) -> None:
from session_store import load as load_session, save as save_session, generate_session_id
tier = req.tier or settings.default_tier
role_cfg = model_registry.get_role_config(user, req.chat_role)
system_prompt = load_context(
tier,
include_long=req.include_long,
include_mid=req.include_mid,
include_short=req.include_short,
role_append=role_cfg.get("system_append", ""),
)
session_id = req.session_id or generate_session_id()
@@ -209,6 +211,7 @@ async def _run_job(job_id: str, req: OrchestrateRequest, user: str) -> None:
orch_model = model_registry.get_model_for_role(user, "orchestrator")
user_role = get_user_role(user)
tool_list = role_cfg.get("tools")
policy = get_tool_policy(user)
confirm_allow = set(policy.get("allow", []))
@@ -222,6 +225,7 @@ async def _run_job(job_id: str, req: OrchestrateRequest, user: str) -> None:
model_cfg=orch_model,
respond_with_final=req.respond_with_claude,
user_role=user_role,
tool_list=tool_list,
confirm_allow=confirm_allow,
confirm_deny=confirm_deny,
)
@@ -239,6 +243,7 @@ async def _run_job(job_id: str, req: OrchestrateRequest, user: str) -> None:
model_name=orch_model.get("model_name") if orch_model else None,
response_role=req.chat_role,
user_role=user_role,
tool_list=tool_list,
confirm_allow=confirm_allow,
confirm_deny=confirm_deny,
)

60
cortex/routers/push.py Normal file
View File

@@ -0,0 +1,60 @@
"""
Web Push endpoints.
GET /api/push/vapid-key → public VAPID key for browser PushManager.subscribe()
POST /api/push/subscribe → save a push subscription for the logged-in user
DELETE /api/push/subscribe → remove a subscription by endpoint
"""
import jwt
from fastapi import APIRouter, HTTPException, Request
from pydantic import BaseModel
from auth_utils import COOKIE_NAME, decode_token
from config import settings
import push_utils
router = APIRouter(prefix="/api/push")
def _require_user(request: Request) -> str:
token = request.cookies.get(COOKIE_NAME)
if not token:
raise HTTPException(status_code=401, detail="Not authenticated")
try:
return decode_token(token)
except jwt.InvalidTokenError:
raise HTTPException(status_code=401, detail="Invalid session")
@router.get("/vapid-key")
async def get_vapid_key() -> dict:
"""Return the VAPID public key. Public endpoint — needed before login to subscribe."""
key = settings.vapid_public_key
if not key:
raise HTTPException(status_code=503, detail="Push notifications not configured")
return {"public_key": key}
class SubscribeRequest(BaseModel):
subscription: dict # full PushSubscription JSON from browser
class UnsubscribeRequest(BaseModel):
endpoint: str
@router.post("/subscribe")
async def subscribe(req: SubscribeRequest, request: Request) -> dict:
username = _require_user(request)
sub = req.subscription
if not sub.get("endpoint"):
raise HTTPException(status_code=400, detail="subscription.endpoint is required")
push_utils.add_subscription(username, sub)
return {"ok": True}
@router.delete("/subscribe")
async def unsubscribe(req: UnsubscribeRequest, request: Request) -> dict:
username = _require_user(request)
found = push_utils.remove_subscription(username, req.endpoint)
return {"ok": True, "found": found}

View File

@@ -19,41 +19,54 @@ logger = logging.getLogger(__name__)
_scheduler: AsyncIOScheduler | None = None
def _all_personas() -> list[tuple[str, str]]:
"""Return [(username, persona_name)] for every persona on this instance."""
from persona import list_users, list_user_personas
pairs = []
for u in list_users():
for p in list_user_personas(u):
pairs.append((u, p))
return pairs
async def _run_short() -> None:
from memory_distiller import distill_short
try:
result = distill_short()
logger.info("auto distill short: %d files, %d chars", result["files_included"], result["chars_written"])
except Exception as e:
logger.error("auto distill short failed: %s", e)
for u, p in _all_personas():
try:
result = distill_short(u, p)
logger.info("auto distill short [%s/%s]: %d files, %d chars", u, p, result["files_included"], result["chars_written"])
except Exception as e:
logger.error("auto distill short [%s/%s] failed: %s", u, p, e)
async def _run_mid() -> None:
from memory_distiller import distill_mid
from notification import notify
try:
result = await distill_mid()
if "error" in result:
logger.warning("auto distill mid skipped: %s", result["error"])
else:
logger.info("auto distill mid: %d chars via %s", result["chars_written"], result["backend"])
await notify(result["username"], f"📝 Weekly memory digest complete ({result['chars_written']} chars via {result['backend']}).")
except Exception as e:
logger.error("auto distill mid failed: %s", e)
for u, p in _all_personas():
try:
result = await distill_mid(u, p)
if "error" in result:
logger.warning("auto distill mid [%s/%s] skipped: %s", u, p, result["error"])
else:
logger.info("auto distill mid [%s/%s]: %d chars via %s", u, p, result["chars_written"], result["backend"])
await notify(u, f"📝 Weekly memory digest complete ({result['chars_written']} chars via {result['backend']}).")
except Exception as e:
logger.error("auto distill mid [%s/%s] failed: %s", u, p, e)
async def _run_long() -> None:
from memory_distiller import distill_long
from notification import notify
try:
result = await distill_long()
if "error" in result:
logger.warning("auto distill long skipped: %s", result["error"])
else:
logger.info("auto distill long: %d chars via %s", result["chars_written"], result["backend"])
await notify(result["username"], f"🧠 Monthly long-term memory integration complete ({result['chars_written']} chars via {result['backend']}). Worth a quick review.")
except Exception as e:
logger.error("auto distill long failed: %s", e)
for u, p in _all_personas():
try:
result = await distill_long(u, p)
if "error" in result:
logger.warning("auto distill long [%s/%s] skipped: %s", u, p, result["error"])
else:
logger.info("auto distill long [%s/%s]: %d chars via %s", u, p, result["chars_written"], result["backend"])
await notify(u, f"🧠 Monthly long-term memory integration complete ({result['chars_written']} chars via {result['backend']}). Worth a quick review.")
except Exception as e:
logger.error("auto distill long [%s/%s] failed: %s", u, p, e)
def get_scheduler() -> AsyncIOScheduler | None:

View File

@@ -1,6 +1,5 @@
from datetime import datetime
from config import settings
from persona import persona_path
from persona import persona_path, get_user, get_persona
def log_turn(
@@ -21,11 +20,15 @@ def log_turn(
meta_parts = [p for p in [backend_label, host] if p]
meta = f" · {' / '.join(meta_parts)}" if meta_parts else ""
# Use the actual user/persona names from the current request context
user_label = get_user().title()
persona_label = get_persona().title()
with open(log_file, "a") as f:
if is_new:
f.write(f"# Session Log — {today}\n")
f.write(
f"\n### [{timestamp}] `{session_id}`{meta}\n"
f"**{settings.user_name}:** {user_msg}\n\n"
f"**{settings.agent_name}:** {assistant_msg}\n"
f"**{user_label}:** {user_msg}\n\n"
f"**{persona_label}:** {assistant_msg}\n"
)

View File

@@ -73,6 +73,17 @@ def save(session_id: str, messages: list[dict]) -> None:
path.write_text(json.dumps(data, indent=2))
def get_name(session_id: str) -> str:
"""Return the friendly name for a session, or '' if none set."""
path = _path(session_id)
if not path.exists():
return ""
try:
return json.loads(path.read_text()).get("name", "")
except Exception:
return ""
def rename(session_id: str, name: str) -> bool:
"""Set (or clear) the friendly name on a session. Returns False if not found."""
path = _path(session_id)

View File

@@ -1,6 +1,6 @@
# Tool Reference
> This reference covers all 30 orchestrator tools available when the ⚡ toggle is on.
> This reference covers all 39 orchestrator tools available when the ⚡ toggle is on.
> Tools are invoked automatically by the orchestrator — you don't call them directly.
¹ **Admin only** — requires the `admin` role. Invisible to regular users.

View File

@@ -507,27 +507,61 @@
const displayName = s.name || s.session_id;
sessionNames.set(s.session_id, displayName);
const item = makeItem(
s.session_id === sessionId ? 'active' : '',
displayName,
`${s.message_count} msgs · ${timeAgo(s.updated)}`
);
item.addEventListener('click', () => resumeSession(s.session_id));
const item = document.createElement('div');
item.className = 'session-item' + (s.session_id === sessionId ? ' active' : '');
// Rename button (✎)
const renameBtn = document.createElement('button');
renameBtn.className = 'session-rename-btn';
renameBtn.textContent = '✎';
renameBtn.title = 'Rename session';
renameBtn.addEventListener('click', async (e) => {
// ── Edit button (left) ──────────────────────────────────
const editBtn = document.createElement('button');
editBtn.className = 'session-edit-btn';
editBtn.textContent = '✎';
editBtn.title = 'Rename session';
// ── Body: name (top) + meta (below) ─────────────────────
const bodyEl = document.createElement('div');
bodyEl.className = 'session-body';
const labelEl = document.createElement('span');
labelEl.className = 'session-id';
labelEl.textContent = displayName;
const metaEl = document.createElement('span');
metaEl.className = 'session-meta';
metaEl.textContent = `${s.message_count} msgs · ${timeAgo(s.updated)}`;
bodyEl.append(labelEl, metaEl);
// ── Delete button (far right) ────────────────────────────
const delBtn = document.createElement('button');
delBtn.className = 'session-delete-btn';
delBtn.title = 'Delete session';
delBtn.textContent = '×';
item.append(editBtn, bodyEl, delBtn);
// Click anywhere on the row (not a button) → resume
item.addEventListener('click', (e) => {
if (!e.target.closest('button')) resumeSession(s.session_id);
});
// ── Edit mode ────────────────────────────────────────────
function enterEditMode(e) {
e.stopPropagation();
const labelEl = item.querySelector('.session-id');
const current = s.name || '';
const input = document.createElement('input');
input.className = 'session-rename-input';
input.value = current;
input.value = s.name || '';
input.placeholder = s.session_id;
labelEl.replaceWith(input);
// Swap body + delete for the input
bodyEl.hidden = true;
delBtn.hidden = true;
editBtn.textContent = '✓';
editBtn.title = 'Save name';
editBtn.className = 'session-save-btn';
editBtn.onclick = async (e) => { e.stopPropagation(); await commitRename(); };
editBtn.after(input);
input.focus();
input.select();
@@ -538,28 +572,32 @@
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: newName }),
});
const res = await fetch(`/sessions?${_fileParams}`);
const data = await res.json();
renderPanel(data.sessions);
// Update status bar if this is the active session
if (sessionId === s.session_id) {
if (sessionId === s.session_id)
sessionEl.textContent = `session: ${newName || s.session_id}`;
}
if (newName) showToast('Session renamed', 'success');
const res = await fetch(`/sessions?${_fileParams}`);
renderPanel((await res.json()).sessions);
}
function cancelEdit() {
input.remove();
bodyEl.hidden = false;
delBtn.hidden = false;
editBtn.textContent = '✎';
editBtn.title = 'Rename session';
editBtn.className = 'session-edit-btn';
editBtn.onclick = enterEditMode;
}
input.addEventListener('keydown', (e) => {
if (e.key === 'Enter') { e.preventDefault(); commitRename(); }
if (e.key === 'Escape') { renderPanel(sessions); }
if (e.key === 'Enter') { e.preventDefault(); commitRename(); }
if (e.key === 'Escape') { e.preventDefault(); cancelEdit(); }
});
input.addEventListener('blur', commitRename);
});
item.appendChild(renameBtn);
}
const delBtn = document.createElement('button');
delBtn.className = 'session-delete-btn';
delBtn.textContent = '×';
delBtn.title = 'Delete session';
editBtn.onclick = enterEditMode;
// ── Delete ───────────────────────────────────────────────
delBtn.addEventListener('click', async (e) => {
e.stopPropagation();
await fetch(`/sessions/${s.session_id}?${_fileParams}`, { method: 'DELETE' });
@@ -572,10 +610,8 @@
showToast('Session deleted');
}
const res = await fetch(`/sessions?${_fileParams}`);
const data = await res.json();
renderPanel(data.sessions);
renderPanel((await res.json()).sessions);
});
item.appendChild(delBtn);
sessionsPanel.appendChild(item);
}
@@ -606,9 +642,13 @@
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
// Prefer name from API response, fall back to sessionNames map, then raw ID
const displayName = data.name || sessionNames.get(id) || id;
sessionNames.set(id, displayName);
messagesEl.innerHTML = '';
sessionId = id;
sessionEl.textContent = `session: ${sessionNames.get(id) || id}`;
sessionEl.textContent = `session: ${displayName}`;
currentHistory = [];
for (let i = 0; i < data.messages.length; i++) {
@@ -626,7 +666,7 @@
}
}
if (!silent) addMessage('system', `Resumed session ${id}`);
if (!silent) addMessage('system', `Resumed session: ${displayName}`);
scrollToBottom();
sessionsPanel.classList.remove('open');
sessionsBackdrop.classList.remove('open');
@@ -1335,22 +1375,26 @@
return d.toLocaleDateString(undefined, { month: 'short', day: 'numeric' });
}
function renderFileSidebar(files) {
function _makeFileGroup(label, collapsed = false) {
const groupEl = document.createElement('div');
groupEl.className = 'file-group';
const header = document.createElement('div');
header.className = 'fg-header' + (collapsed ? ' collapsed' : '');
header.textContent = label;
header.addEventListener('click', () => header.classList.toggle('collapsed'));
groupEl.appendChild(header);
const items = document.createElement('div');
items.className = 'fg-items';
groupEl.appendChild(items);
return { groupEl, items };
}
function renderFileSidebar(files, auditDates = []) {
const byName = Object.fromEntries(files.map(f => [f.name, f]));
fileSidebar.innerHTML = '';
for (const group of FILE_GROUPS) {
const groupEl = document.createElement('div');
groupEl.className = 'file-group';
const header = document.createElement('div');
header.className = 'fg-header';
header.textContent = group.label;
header.addEventListener('click', () => header.classList.toggle('collapsed'));
groupEl.appendChild(header);
const items = document.createElement('div');
items.className = 'fg-items';
const { groupEl, items } = _makeFileGroup(group.label);
for (const fname of group.files) {
const f = byName[fname];
@@ -1376,7 +1420,35 @@
items.appendChild(item);
}
groupEl.appendChild(items);
fileSidebar.appendChild(groupEl);
}
// ── Audit Log section (dynamic, date-named files) ──────────
if (auditDates.length > 0) {
const { groupEl, items } = _makeFileGroup('Audit Log', true);
for (const d of auditDates) {
const item = document.createElement('div');
item.className = 'file-item';
item.dataset.name = 'audit:' + d;
if (activeFileName === 'audit:' + d) item.classList.add('active');
const nameEl = document.createElement('div');
nameEl.className = 'fi-name';
nameEl.textContent = d + '.jsonl';
item.appendChild(nameEl);
const metaEl = document.createElement('div');
metaEl.className = 'fi-meta';
const today = new Date().toISOString().slice(0, 10);
const yesterday = new Date(Date.now() - 86400000).toISOString().slice(0, 10);
metaEl.innerHTML = `<span>${d === today ? 'today' : d === yesterday ? 'yesterday' : d}</span>`;
item.appendChild(metaEl);
item.addEventListener('click', () => loadAuditLog(d));
items.appendChild(item);
}
fileSidebar.appendChild(groupEl);
}
}
@@ -1415,6 +1487,10 @@
async function loadFile(name) {
setActiveFile(name);
initMdEditor();
// Restore editor/preview buttons hidden by audit view
fileRawBtn.style.display = '';
filePreviewBtn.style.display = '';
fileSaveBtn.style.display = '';
const res = await fetch(`/files/${encodeURIComponent(name)}?${_fileParams}`);
if (!res.ok) { mdEditor.setValue(`Error loading ${name}`); return; }
const data = await res.json();
@@ -1423,13 +1499,90 @@
setFileMode(fileMode);
}
async function openFileModal() {
const res = await fetch(`/files?${_fileParams}`);
function _auditStatusClass(status) {
if (status === 'ok') return 'at-status ok';
if (status === 'error') return 'at-status error';
if (status === 'denied') return 'at-status denied';
return 'at-status';
}
function _fmtArgs(args) {
if (!args || typeof args !== 'object') return '';
return Object.entries(args)
.map(([k, v]) => {
const s = typeof v === 'string' ? v : JSON.stringify(v);
return `${k}: ${s.length > 60 ? s.slice(0, 60) + '…' : s}`;
})
.join(' · ');
}
async function loadAuditLog(dateStr) {
setActiveFile('audit:' + dateStr);
document.getElementById('file-modal-title').textContent = dateStr + '.jsonl';
// Hide edit controls — audit logs are read-only
fileRawBtn.style.display = 'none';
filePreviewBtn.style.display = 'none';
fileSaveBtn.style.display = 'none';
fileEditorWrap.classList.add('hidden');
filePreview.classList.add('active');
filePreview.style.display = '';
filePreview.innerHTML = '<div class="audit-empty">Loading…</div>';
const res = await fetch(`/api/audit/day?date=${encodeURIComponent(dateStr)}`);
if (!res.ok) {
filePreview.innerHTML = '<div class="audit-empty">Failed to load audit log.</div>';
return;
}
const data = await res.json();
renderFileSidebar(data.files);
const entries = data.entries || [];
if (entries.length === 0) {
filePreview.innerHTML = '<div class="audit-empty">No entries for this date.</div>';
return;
}
const table = document.createElement('table');
table.className = 'audit-table';
table.innerHTML = `<thead><tr>
<th class="at-time">Time</th>
<th class="at-tool">Tool</th>
<th class="at-status">Status</th>
<th class="at-args">Args</th>
<th class="at-result">Result</th>
</tr></thead>`;
const tbody = document.createElement('tbody');
for (const e of entries) {
const time = (e.ts || '').slice(11, 19); // HH:MM:SS
const tr = document.createElement('tr');
tr.innerHTML = `
<td class="at-time">${time}</td>
<td class="at-tool" title="${e.tool || ''}">${e.tool || '?'}</td>
<td class="${_auditStatusClass(e.status)}">${e.status || '?'}</td>
<td class="at-args" title="${(_fmtArgs(e.args) || '').replace(/"/g, '&quot;')}">${_fmtArgs(e.args)}</td>
<td class="at-result" title="${(e.result_snippet || '').replace(/</g, '&lt;').replace(/"/g, '&quot;')}">${
(e.result_snippet || '').replace(/</g, '&lt;').slice(0, 80)
+ (e.result_chars > 80 ? `… <span style="color:var(--muted)">[${e.result_chars} chars]</span>` : '')
}</td>`;
tbody.appendChild(tr);
}
table.appendChild(tbody);
filePreview.innerHTML = '';
filePreview.appendChild(table);
}
async function openFileModal() {
const [filesRes, auditRes] = await Promise.all([
fetch(`/files?${_fileParams}`),
fetch('/api/audit/files'),
]);
const filesData = await filesRes.json();
const auditData = auditRes.ok ? await auditRes.json() : { dates: [] };
renderFileSidebar(filesData.files, auditData.dates);
fileModal.classList.add('open');
// Load first existing file
const first = data.files.find(f => f.exists) || data.files[0];
// Load first existing regular file
const first = filesData.files.find(f => f.exists) || filesData.files[0];
if (first) await loadFile(first.name);
}
@@ -1724,7 +1877,7 @@
async function runDistill(endpoint) {
showDistillStatus('distilling…', false);
try {
const res = await fetch(`/distill/${endpoint}`, { method: 'POST' });
const res = await fetch(`/distill/${endpoint}?${_fileParams}`, { method: 'POST' });
const d = await res.json();
if (!res.ok || d.ok === false) {
const err = d.error || d.mid?.error || d.long?.error || `HTTP ${res.status}`;
@@ -1766,7 +1919,93 @@
if (stored) resumeSession(stored, true).catch(clear_stored_session);
}
// ── Service worker registration ───────────────────────────────
// ── Service worker + Web Push ────────────────────────────────
const pushBtn = document.getElementById('push-btn');
const pushBtnLabel = document.getElementById('push-btn-label');
function _urlBase64ToUint8Array(base64String) {
const padding = '='.repeat((4 - base64String.length % 4) % 4);
const base64 = (base64String + padding).replace(/-/g, '+').replace(/_/g, '/');
const raw = atob(base64);
return Uint8Array.from([...raw].map(c => c.charCodeAt(0)));
}
async function _getPushSubscription() {
if (!('serviceWorker' in navigator) || !('PushManager' in window)) return null;
const reg = await navigator.serviceWorker.ready;
return reg.pushManager.getSubscription();
}
async function _syncPushBtn() {
if (!('PushManager' in window) || !('serviceWorker' in navigator)) return;
pushBtn.style.display = '';
const sub = await _getPushSubscription();
if (sub) {
pushBtnLabel.textContent = 'Notifications on';
pushBtn.classList.add('push-active');
} else {
pushBtnLabel.textContent = 'Enable notifications';
pushBtn.classList.remove('push-active');
}
}
async function _subscribePush() {
try {
const keyRes = await fetch('/api/push/vapid-key');
if (!keyRes.ok) { showToast('Push not configured on server'); return; }
const { public_key } = await keyRes.json();
const perm = await Notification.requestPermission();
if (perm !== 'granted') { showToast('Notification permission denied'); return; }
const reg = await navigator.serviceWorker.ready;
const sub = await reg.pushManager.subscribe({
userVisibleOnly: true,
applicationServerKey: _urlBase64ToUint8Array(public_key),
});
await fetch('/api/push/subscribe', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ subscription: sub.toJSON() }),
});
showToast('Push notifications enabled');
await _syncPushBtn();
} catch (e) {
showToast('Could not enable push: ' + e.message);
}
}
async function _unsubscribePush() {
try {
const sub = await _getPushSubscription();
if (!sub) { await _syncPushBtn(); return; }
await fetch('/api/push/subscribe', {
method: 'DELETE',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ endpoint: sub.endpoint }),
});
await sub.unsubscribe();
showToast('Notifications disabled');
await _syncPushBtn();
} catch (e) {
showToast('Error: ' + e.message);
}
}
if (pushBtn) {
pushBtn.addEventListener('click', async () => {
settings_dd_el.classList.remove('open');
const sub = await _getPushSubscription();
if (sub) await _unsubscribePush();
else await _subscribePush();
});
_syncPushBtn();
}
if ('serviceWorker' in navigator) {
navigator.serviceWorker.register('/sw.js').catch(() => {});
}

View File

@@ -63,6 +63,10 @@
<a href="/settings" class="hdr-dd-item">
<svg data-lucide="user" class="btn-icon"></svg> Account
</a>
<button id="push-btn" class="hdr-dd-item" style="display:none">
<svg data-lucide="bell" class="btn-icon"></svg>
<span id="push-btn-label">Enable notifications</span>
</button>
<div class="hdr-dd-divider"></div>
<form method="POST" action="/logout" style="margin:0">
<button type="submit" class="hdr-dd-item">

View File

@@ -223,7 +223,6 @@
display: flex; align-items: flex-start; gap: 1rem;
padding: 0.6rem 0; border-bottom: 1px solid var(--pg-border-deep);
}
.role-row:last-child { border-bottom: none; }
.role-name { font-size: 0.82rem; font-weight: 600; color: #a78bfa; min-width: 6rem; padding-top: 0.45rem; }
.role-slots { display: flex; flex-wrap: wrap; gap: 0.5rem; flex: 1; }
.role-slot { display: flex; flex-direction: column; gap: 0.2rem; flex: 1; min-width: 8rem; }
@@ -238,6 +237,36 @@
.role-select.saved { border-color: #166534; }
.role-select.saving { border-color: #92400e; }
.role-select.err { border-color: #7f1d1d; }
.role-cfg-btn {
flex-shrink: 0; padding: 0.3rem 0.55rem; font-size: 0.8rem;
background: none; border: 1px solid var(--pg-border); border-radius: 6px;
color: var(--pg-dim); cursor: pointer; transition: color 0.15s, border-color 0.15s;
margin-top: 0.35rem;
}
.role-cfg-btn:hover { color: #a78bfa; border-color: #a78bfa; }
.role-cfg-btn.active { color: #a78bfa; border-color: #a78bfa; background: rgba(167,139,250,0.08); }
/* Role config panel */
.role-config-panel {
display: none; margin: 0 0 0.75rem 7rem;
border: 1px solid var(--pg-border); border-radius: 8px;
background: var(--pg-surface); padding: 1rem;
}
.role-config-panel.open { display: block; }
.rcp-field { margin-bottom: 0.75rem; }
.rcp-label { display: block; font-size: 0.75rem; font-weight: 600; color: var(--pg-muted); margin-bottom: 0.35rem; text-transform: uppercase; letter-spacing: 0.04em; }
.rcp-hint { font-weight: 400; text-transform: none; letter-spacing: 0; color: var(--pg-dimmer); }
.rcp-textarea {
width: 100%; resize: vertical; min-height: 4rem;
background: var(--pg-bg); border: 1px solid var(--pg-border); border-radius: 6px;
color: var(--pg-text); font-family: inherit; font-size: 0.85rem;
padding: 0.5rem 0.6rem; outline: none; transition: border-color 0.15s;
}
.rcp-textarea:focus { border-color: #7c3aed; }
.rcp-tools { display: flex; flex-wrap: wrap; gap: 0.4rem 1rem; }
.rcp-cat { width: 100%; margin: 0.4rem 0 0.1rem; font-size: 0.7rem; font-weight: 600; color: var(--pg-dimmer); text-transform: uppercase; letter-spacing: 0.05em; }
.rcp-check { display: flex; align-items: center; gap: 0.35rem; font-size: 0.8rem; color: var(--pg-bright); cursor: pointer; }
.rcp-check input { accent-color: #a78bfa; cursor: pointer; }
.rcp-actions { display: flex; gap: 0.5rem; padding-top: 0.25rem; }
/* Model select picker */
#model-select-wrap { display: none; margin-bottom: 0.75rem; }
@@ -496,6 +525,8 @@
<script>
// ── Injected data ─────────────────────────────────────────────────────────
const ROLE_DATA = {{ role_data_js }};
const ROLE_CONFIG_DATA = {{ role_config_data_js }};
const TOOL_CATEGORIES = {{ tool_categories_js }};
const GOOGLE_ACCOUNTS = {{ google_accounts_js }};
const GOOGLE_CATALOG = {{ google_catalog_js }};
const ANTHROPIC_CATALOG = {{ anthropic_catalog_js }};
@@ -543,6 +574,112 @@
});
});
// ── Role config panels ────────────────────────────────────────────────────
// All tool names in category order (for checkbox rendering)
const ALL_TOOLS_ORDERED = Object.entries(TOOL_CATEGORIES).flatMap(([,tools]) => tools);
function buildToolChecklist(role, savedTools) {
// savedTools: null = all, array = explicit allow-list
const wrap = document.getElementById(`rcp-tools-${role}`);
if (!wrap) return;
wrap.innerHTML = '';
for (const [cat, tools] of Object.entries(TOOL_CATEGORIES)) {
const catEl = document.createElement('div');
catEl.className = 'rcp-cat';
catEl.style.width = '100%';
catEl.textContent = cat;
wrap.appendChild(catEl);
for (const tool of tools) {
const label = document.createElement('label');
label.className = 'rcp-check';
const cb = document.createElement('input');
cb.type = 'checkbox';
cb.value = tool;
cb.checked = savedTools === null || savedTools.includes(tool);
label.appendChild(cb);
label.appendChild(document.createTextNode(tool));
wrap.appendChild(label);
}
}
}
function openRolePanel(role) {
const panel = document.getElementById(`rcp-${role}`);
const btn = document.querySelector(`.role-cfg-btn[data-role="${role}"]`);
const cfg = ROLE_CONFIG_DATA[role] || {};
if (!panel) return;
// Populate textarea
panel.querySelector('.rcp-textarea').value = cfg.system_append || '';
// Build tool checklist
buildToolChecklist(role, cfg.tools || null);
panel.classList.add('open');
btn && btn.classList.add('active');
}
function closeRolePanel(role) {
const panel = document.getElementById(`rcp-${role}`);
const btn = document.querySelector(`.role-cfg-btn[data-role="${role}"]`);
panel && panel.classList.remove('open');
btn && btn.classList.remove('active');
}
document.querySelectorAll('.role-cfg-btn').forEach(btn => {
btn.addEventListener('click', () => {
const role = btn.dataset.role;
const panel = document.getElementById(`rcp-${role}`);
if (panel.classList.contains('open')) {
closeRolePanel(role);
} else {
// Close any other open panels first
document.querySelectorAll('.role-config-panel.open').forEach(p => {
closeRolePanel(p.id.replace('rcp-', ''));
});
openRolePanel(role);
}
});
});
document.querySelectorAll('.rcp-cancel').forEach(btn => {
btn.addEventListener('click', () => closeRolePanel(btn.dataset.role));
});
document.querySelectorAll('.rcp-save').forEach(btn => {
btn.addEventListener('click', async () => {
const role = btn.dataset.role;
const panel = document.getElementById(`rcp-${role}`);
const ta = panel.querySelector('.rcp-textarea');
const checks = [...panel.querySelectorAll('.rcp-check input[type=checkbox]')];
const allChecked = checks.every(c => c.checked);
const someChecked = checks.some(c => c.checked);
const tools = allChecked ? null : (someChecked ? checks.filter(c => c.checked).map(c => c.value) : []);
btn.disabled = true;
try {
const res = await fetch('/api/models/role-config', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({role, system_append: ta.value, tools}),
});
const data = await res.json();
if (data.ok) {
// Update local state so re-open shows current values
if (!ROLE_CONFIG_DATA[role]) ROLE_CONFIG_DATA[role] = {};
ROLE_CONFIG_DATA[role].system_append = ta.value;
ROLE_CONFIG_DATA[role].tools = tools;
showToast(`${role} config saved`);
closeRolePanel(role);
} else {
showToast(data.error || 'Save failed', true);
}
} catch (e) {
showToast(e.message, true);
} finally {
btn.disabled = false;
}
});
});
// ── Provider tabs ─────────────────────────────────────────────────────────
const providerVal = document.getElementById('add-provider-val');
const pfields = {

View File

@@ -266,6 +266,7 @@
box-sizing: border-box;
}
.hdr-dd-item:hover { background: var(--border); }
.hdr-dd-item.push-active { color: var(--accent); }
.hdr-dd-divider {
border-top: 1px solid var(--border);
@@ -278,8 +279,8 @@
position: absolute;
top: calc(100% + 4px);
right: 12px;
width: min(300px, calc(100vw - 24px));
max-height: 340px;
width: min(420px, calc(100vw - 24px));
max-height: 400px;
overflow-y: auto;
background: var(--surface);
border: 1px solid var(--border);
@@ -291,19 +292,26 @@
#sessions-panel.open { display: block; }
.session-item {
padding: 10px 14px;
padding: 8px 12px;
cursor: pointer;
border-bottom: 1px solid var(--border);
display: flex;
justify-content: space-between;
align-items: center;
gap: 8px;
gap: 6px;
}
.session-item:last-child { border-bottom: none; }
.session-item:hover { background: var(--bg); }
.session-item.new { color: var(--accent); justify-content: center; }
.session-body {
flex: 1;
min-width: 0;
display: flex;
flex-direction: column;
gap: 2px;
}
.session-delete-btn {
background: none;
border: none;
@@ -320,7 +328,7 @@
}
.session-delete-btn:hover { color: #e06c75; }
.session-rename-btn {
.session-edit-btn {
background: none;
border: none;
color: var(--muted);
@@ -330,13 +338,30 @@
cursor: pointer;
border-radius: 3px;
flex-shrink: 0;
opacity: 0.4;
opacity: 0.3;
transition: opacity 0.15s, color 0.15s;
min-width: 24px;
text-align: center;
}
.session-item:hover .session-rename-btn { opacity: 1; }
.session-rename-btn:hover { color: var(--accent); }
.session-item:hover .session-edit-btn { opacity: 0.75; }
.session-edit-btn:hover { color: var(--accent); opacity: 1; }
.session-save-btn {
background: none;
border: none;
color: var(--accent);
font-size: 1rem;
font-weight: bold;
line-height: 1;
padding: 2px 6px;
cursor: pointer;
border-radius: 3px;
flex-shrink: 0;
min-width: 24px;
text-align: center;
transition: opacity 0.15s;
}
.session-save-btn:hover { opacity: 0.75; }
.session-rename-input {
flex: 1;
@@ -363,11 +388,9 @@
}
.session-meta {
font-size: 0.78rem;
font-size: 0.75rem;
color: var(--muted);
white-space: nowrap;
text-align: right;
flex-shrink: 0;
}
/* Messages */
@@ -1209,6 +1232,42 @@
#file-preview.active { display: block; }
#file-editor-wrap.hidden { display: none; }
/* ── Audit log table ────────────────────────────────────────── */
.audit-table {
width: 100%;
border-collapse: collapse;
font-size: 0.78rem;
table-layout: fixed;
}
.audit-table th {
text-align: left;
padding: 5px 8px;
border-bottom: 1px solid var(--border);
color: var(--muted);
font-weight: 600;
white-space: nowrap;
}
.audit-table td {
padding: 5px 8px;
border-bottom: 1px solid color-mix(in srgb, var(--border) 50%, transparent);
vertical-align: top;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.audit-table tr:last-child td { border-bottom: none; }
.audit-table tr:hover td { background: var(--surface); }
/* Column widths */
.at-time { width: 7em; color: var(--muted); white-space: nowrap; }
.at-tool { width: 11em; color: var(--accent); font-weight: 500; }
.at-status { width: 4.5em; font-weight: 600; }
.at-args { width: 30%; color: var(--muted); }
.at-result { color: var(--muted); }
.at-status.ok { color: #4ade80; }
.at-status.error { color: #f87171; }
.at-status.denied { color: #fbbf24; }
.audit-empty { padding: 24px; color: var(--muted); text-align: center; font-size: 0.9rem; }
/* Talk activity badge on Sessions button */
#sessions-btn.talk-badge::after {
content: '●';
@@ -1573,7 +1632,7 @@
top: 0;
right: 0;
bottom: 0;
width: min(300px, 85vw);
width: min(380px, 90vw);
max-height: none;
height: 100%;
border-radius: 0;
@@ -1609,6 +1668,9 @@
min-width: 36px;
min-height: 36px;
}
/* On touch: edit button always fully visible (no hover to reveal it) */
.session-edit-btn { opacity: 0.6; }
}
@media (max-width: 380px) {

View File

@@ -1,4 +1,4 @@
const CACHE = 'cortex-v1';
const CACHE = 'cortex-v2';
const PRECACHE = [
'/static/style.css',
@@ -28,6 +28,37 @@ self.addEventListener('activate', evt => {
);
});
self.addEventListener('push', evt => {
let data = { title: 'Cortex', body: '', url: '/' };
if (evt.data) {
try { data = { ...data, ...evt.data.json() }; } catch (_) {}
}
evt.waitUntil(
self.registration.showNotification(data.title, {
body: data.body,
icon: '/static/icon-192.png',
badge: '/static/icon-192.png',
data: { url: data.url },
})
);
});
self.addEventListener('notificationclick', evt => {
evt.notification.close();
const url = evt.notification.data?.url || '/';
evt.waitUntil(
clients.matchAll({ type: 'window', includeUncontrolled: true }).then(list => {
for (const c of list) {
if (c.url.includes(self.location.origin) && 'focus' in c) {
c.navigate(url);
return c.focus();
}
}
if (clients.openWindow) return clients.openWindow(url);
})
);
});
self.addEventListener('fetch', evt => {
const url = new URL(evt.request.url);

143
cortex/tool_audit.py Normal file
View File

@@ -0,0 +1,143 @@
"""
Tool call audit log.
One JSONL file per user per day:
home/{user}/tool_audit/YYYY-MM-DD.jsonl
Each line is a JSON object:
ts ISO timestamp (seconds)
user username
tool tool name
args call arguments (string values truncated at ARG_MAX chars)
status "ok" | "error" | "denied"
result_chars length of full result string
result_snippet first SNIPPET_MAX chars of result
"""
import asyncio
import json
import logging
from datetime import datetime, date
from pathlib import Path
from config import settings
logger = logging.getLogger(__name__)
_ARG_MAX = 500 # truncate individual arg string values longer than this
_SNIPPET_MAX = 300 # chars of result to keep as snippet
# Per-file write locks — prevents interleaved lines under concurrent tool calls
_locks: dict[str, asyncio.Lock] = {}
def _truncate_args(args: dict) -> dict:
out = {}
for k, v in args.items():
if isinstance(v, str) and len(v) > _ARG_MAX:
out[k] = v[:_ARG_MAX] + f" …[{len(v)} chars total]"
else:
out[k] = v
return out
def _audit_path(user: str, day: date | None = None) -> Path:
d = day or date.today()
audit_dir = settings.home_root() / user / "tool_audit"
audit_dir.mkdir(parents=True, exist_ok=True)
return audit_dir / f"{d.isoformat()}.jsonl"
async def record(
user: str,
tool: str,
args: dict,
status: str, # "ok" | "error" | "denied"
result: str = "",
) -> None:
"""Append one audit entry. Fire with asyncio.create_task — never awaited directly."""
path = _audit_path(user)
key = str(path)
if key not in _locks:
_locks[key] = asyncio.Lock()
entry = {
"ts": datetime.now().isoformat(timespec="seconds"),
"user": user,
"tool": tool,
"args": _truncate_args(args),
"status": status,
"result_chars": len(result),
"result_snippet": result[:_SNIPPET_MAX],
}
async with _locks[key]:
try:
with path.open("a", encoding="utf-8") as f:
f.write(json.dumps(entry) + "\n")
except Exception as e:
logger.warning("audit log write failed for %s: %s", user, e)
def read_recent(user: str, days: int = 7, limit: int = 200) -> list[dict]:
"""Read the most recent `limit` entries across the last `days` days.
Returns entries sorted newest-first (by ts field, file order within a day).
"""
from datetime import timedelta
today = date.today()
entries: list[dict] = []
for offset in range(days):
day = today - timedelta(days=offset)
path = settings.home_root() / user / "tool_audit" / f"{day.isoformat()}.jsonl"
if not path.exists():
continue
try:
lines = path.read_text(encoding="utf-8").splitlines()
except Exception:
continue
day_entries = []
for line in lines:
line = line.strip()
if not line:
continue
try:
day_entries.append(json.loads(line))
except json.JSONDecodeError:
pass
# Newest within the day first
entries.extend(reversed(day_entries))
if len(entries) >= limit:
break
return entries[:limit]
def read_day(user: str, day_str: str) -> list[dict]:
"""Read all entries for a specific date string (YYYY-MM-DD), chronological order."""
path = settings.home_root() / user / "tool_audit" / f"{day_str}.jsonl"
if not path.exists():
return []
entries = []
try:
for line in path.read_text(encoding="utf-8").splitlines():
line = line.strip()
if not line:
continue
try:
entries.append(json.loads(line))
except json.JSONDecodeError:
pass
except Exception:
pass
return entries
def read_recent_all_users(days: int = 7, limit: int = 500) -> list[dict]:
"""Read recent entries across all users, sorted newest-first."""
from persona import list_users
all_entries: list[dict] = []
for user in list_users():
all_entries.extend(read_recent(user, days=days, limit=limit))
all_entries.sort(key=lambda e: e.get("ts", ""), reverse=True)
return all_entries[:limit]

File diff suppressed because it is too large Load Diff

View File

@@ -11,6 +11,7 @@ API: V3 CRUD — POST /v3/crud/journal_entry/search, POST /v3/crud/journal/{id}
import asyncio
import logging
from google.genai import types
from config import settings
logger = logging.getLogger(__name__)
@@ -580,3 +581,167 @@ def _sync_journal_entry_prepend(entry_id: str, content: str, heading: str) -> st
if result != "ok":
return result
return f"Prepended to journal entry `{entry_id}` under heading \"{section_heading}\"."
DECLARATIONS = [
types.FunctionDeclaration(
name="ae_journal_list",
description=(
"List all Aether Journals available for this account. "
"Returns each journal's name and id_random. "
"Call this first when you need to write a new entry or scope a search to a specific journal "
"and don't already know the journal's id."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="ae_journal_search",
description=(
"Search Aether Journal entries. All parameters are optional — combine freely. "
"Use 'query' for fulltext keyword search (supports boolean: +required -excluded \"phrase\"). "
"Use 'tags' to filter by tag substring. Use 'date_from'/'date_to' for date ranges (YYYY-MM-DD). "
"Always search before creating a new entry to avoid duplicates."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"query": types.Schema(type=types.Type.STRING, description="Fulltext keyword search. Supports boolean mode: +required -excluded \"exact phrase\"."),
"journal_id": types.Schema(type=types.Type.STRING, description="Scope results to a specific journal by its id_random. Omit to search all journals."),
"tags": types.Schema(type=types.Type.STRING, description="Filter by tag substring (e.g. 'networking' matches entries tagged 'networking' or 'home-networking')."),
"type_code": types.Schema(type=types.Type.STRING, description="Filter by exact type_code (e.g. 'note', 'meeting', 'log')."),
"topic_code": types.Schema(type=types.Type.STRING, description="Filter by exact topic_code."),
"date_from": types.Schema(type=types.Type.STRING, description="Return entries created on or after this date (YYYY-MM-DD)."),
"date_to": types.Schema(type=types.Type.STRING, description="Return entries created on or before this date (YYYY-MM-DD)."),
"sort_by": types.Schema(type=types.Type.STRING, description="Sort field: 'updated' (default), 'created', 'name', or 'priority'."),
"sort_order": types.Schema(type=types.Type.STRING, description="Sort direction: 'desc' (default, newest first) or 'asc'."),
"status": types.Schema(type=types.Type.INTEGER, description="Filter by exact status code."),
"priority": types.Schema(type=types.Type.INTEGER, description="Filter by exact priority (1=low, 5=high)."),
"max_results": types.Schema(type=types.Type.INTEGER, description="Number of results per page (default 10)."),
"page": types.Schema(type=types.Type.INTEGER, description="Page number for pagination (default 1)."),
},
required=[],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_read",
description=(
"Fetch the full content of a single journal entry by its id_random. "
"Use this when you need to read an entry before editing it, or when search results "
"don't show enough content. Returns title, journal, tags, summary, and full content."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"entry_id": types.Schema(type=types.Type.STRING, description="The id_random of the journal entry to read."),
"max_content_chars": types.Schema(type=types.Type.INTEGER, description="Maximum characters of content to return (default 4000). Increase for long entries."),
},
required=["entry_id"],
),
),
types.FunctionDeclaration(
name="ae_journal_entries_list",
description=(
"List entries in a specific journal, newest first. "
"Use this to browse what's in a journal when you don't have a search keyword, "
"or to find entries by browsing rather than searching. "
"Returns numbered entries with id, title, tags, summary, and date."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"journal_id": types.Schema(type=types.Type.STRING, description="The id_random of the journal to list entries from."),
"max_results": types.Schema(type=types.Type.INTEGER, description="Number of entries to return (default 20, max 50)."),
"page": types.Schema(type=types.Type.INTEGER, description="Page number for pagination (default 1)."),
},
required=["journal_id"],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_create",
description=(
"Create a new entry in an Aether Journal. "
"Use this to save notes, summaries, or any content the user wants to store. "
"Always call ae_journal_search first to check for existing entries on the same topic."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"journal_id": types.Schema(type=types.Type.STRING, description="The id_random of the target journal. Ask the user which journal to write to if not specified."),
"title": types.Schema(type=types.Type.STRING, description="Entry title"),
"content": types.Schema(type=types.Type.STRING, description="Full entry content (markdown supported)"),
"summary": types.Schema(type=types.Type.STRING, description="Optional short summary (1-2 sentences)"),
"tags": types.Schema(type=types.Type.STRING, description="Optional comma-separated tags (e.g. 'wireguard, networking, homelab')"),
},
required=["journal_id", "title", "content"],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_update",
description=(
"Update fields on an existing journal entry. Only the fields you provide are changed — "
"omitted fields are left as-is. Use ae_journal_search to find the entry_id first. "
"To soft-delete, use ae_journal_entry_disable instead."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"entry_id": types.Schema(type=types.Type.STRING, description="Journal entry id_random"),
"title": types.Schema(type=types.Type.STRING, description="New title"),
"content": types.Schema(type=types.Type.STRING, description="Replacement content (full, markdown supported)"),
"summary": types.Schema(type=types.Type.STRING, description="New summary"),
"tags": types.Schema(type=types.Type.STRING, description="Replacement comma-separated tags"),
"enable": types.Schema(type=types.Type.BOOLEAN, description="Set false to hide/disable the entry"),
},
required=["entry_id"],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_disable",
description=(
"Soft-delete a journal entry by setting enable=false. "
"The entry is hidden but not permanently removed. "
"Use ae_journal_search to find the entry_id first."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"entry_id": types.Schema(type=types.Type.STRING, description="Journal entry id_random"),
},
required=["entry_id"],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_append",
description=(
"Append a new section to the bottom of a journal entry's content. "
"Each section gets a UTC timestamp heading unless you provide one. "
"Ideal for timestamped logs, running notes, or data logs."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"entry_id": types.Schema(type=types.Type.STRING, description="Journal entry id_random"),
"content": types.Schema(type=types.Type.STRING, description="The text to append (markdown supported)"),
"heading": types.Schema(type=types.Type.STRING, description="Optional section heading (defaults to current UTC timestamp)"),
},
required=["entry_id", "content"],
),
),
types.FunctionDeclaration(
name="ae_journal_entry_prepend",
description=(
"Prepend a new section to the top of a journal entry's content. "
"Each section gets a UTC timestamp heading unless you provide one. "
"Useful for most-recent-first logs."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"entry_id": types.Schema(type=types.Type.STRING, description="Journal entry id_random"),
"content": types.Schema(type=types.Type.STRING, description="The text to prepend (markdown supported)"),
"heading": types.Schema(type=types.Type.STRING, description="Optional section heading (defaults to current UTC timestamp)"),
},
required=["entry_id", "content"],
),
),
]

View File

@@ -16,6 +16,8 @@ import json
import logging
from pathlib import Path
from google.genai import types
logger = logging.getLogger(__name__)
# Resolved at import time — agents_sync is always at ~/agents_sync on this machine.
@@ -98,3 +100,20 @@ def _read_bucket(bucket_dir: Path) -> list[dict]:
except Exception as e:
logger.warning("Failed to read task file %s: %s", path, e)
return tasks
DECLARATIONS = [
types.FunctionDeclaration(
name="ae_task_list",
description=(
"List tasks from the agents_sync Kanban board (todo and in-progress). "
"Use this when asked about current work, pending tasks, or project status."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"include_done": types.Schema(type=types.Type.BOOLEAN, description="If true, also include completed tasks (default false)"),
},
),
),
]

View File

@@ -17,6 +17,7 @@ import secrets
from datetime import datetime, timezone
from pathlib import Path
from google.genai import types
from persona import persona_path, get_user, get_persona
from cron_runner import load_crons, save_crons, parse_schedule
@@ -194,3 +195,64 @@ async def cron_toggle(cron_id: str) -> str:
async def reminders_clear() -> str:
return await asyncio.to_thread(_reminders_clear)
DECLARATIONS = [
types.FunctionDeclaration(
name="cron_list",
description=(
"List all scheduled cron jobs — their ID, label, schedule, type, and last run time. "
"Use this to see what's scheduled before adding or removing jobs."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="cron_add",
description=(
"Create a new scheduled cron job and register it immediately (no restart needed). "
"Two types: 'remind' writes to the pending reminders queue (Inara sees it automatically "
"in context next session); 'note' appends to the scratchpad. "
"Schedule formats: 'hourly' | 'daily' | 'daily:HH:MM' | 'weekly:DOW' | 'weekly:DOW:HH:MM'. "
"Example: schedule='daily:09:00', type='remind', payload='Check in with Scott.'"
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"label": types.Schema(type=types.Type.STRING, description="Short human-readable name for this job (e.g. 'Morning check-in')"),
"schedule": types.Schema(type=types.Type.STRING, description="When to run. Formats: hourly | daily | daily:HH:MM | weekly:DOW | weekly:DOW:HH:MM (e.g. 'weekly:mon:09:00')"),
"job_type": types.Schema(type=types.Type.STRING, description="'remind' (→ REMINDERS.md, auto-surfaced in context) or 'note' (→ SCRATCH.md)"),
"payload": types.Schema(type=types.Type.STRING, description="The text to write when the job fires"),
},
required=["label", "schedule", "job_type", "payload"],
),
),
types.FunctionDeclaration(
name="cron_remove",
description=(
"Permanently delete a scheduled cron job. Use cron_list first to get the ID. "
"To temporarily disable without deleting, use cron_toggle instead."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"cron_id": types.Schema(type=types.Type.STRING, description="Job ID (e.g. c_abc123) — get from cron_list"),
},
required=["cron_id"],
),
),
types.FunctionDeclaration(
name="cron_toggle",
description=(
"Pause a running cron job, or resume a paused one. "
"The job stays in the list and can be re-enabled later. "
"Use cron_list to see current enabled/paused state."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"cron_id": types.Schema(type=types.Type.STRING, description="Job ID (e.g. c_abc123) — get from cron_list"),
},
required=["cron_id"],
),
),
]

View File

@@ -10,16 +10,27 @@ import asyncio
import logging
from pathlib import Path
from google.genai import types
logger = logging.getLogger(__name__)
# Directories the orchestrator is allowed to read from.
# Paths are resolved (symlinks followed, ~ expanded) at import time.
_ALLOWED_ROOTS: list[Path] = [
Path.home() / "agents_sync",
Path.home() / "OSIT_dev",
Path.home() / "DgrZone_Nextcloud",
Path.home() / "OSIT_Nextcloud",
]
def _build_allowed_roots() -> list[Path]:
roots = [
Path.home() / "agents_sync",
Path.home() / "OSIT_dev",
Path.home() / "DgrZone_Nextcloud",
Path.home() / "OSIT_Nextcloud",
]
try:
from config import settings
roots.append(settings.home_root())
except Exception:
pass
return roots
_ALLOWED_ROOTS: list[Path] = _build_allowed_roots()
# Hard cap on file size to prevent accidental context blowout
_MAX_BYTES = 50_000 # ~50 KB
@@ -212,3 +223,59 @@ def _sync_file_write(path: str, content: str, mode: str) -> str:
except Exception as e:
logger.error("file_write error for %s: %s", resolved, e)
return f"Write error: {e}"
DECLARATIONS = [
types.FunctionDeclaration(
name="file_read",
description=(
"Read a local file and return its contents. "
"Allowed directories: ~/agents_sync/, ~/OSIT_dev/, ~/DgrZone_Nextcloud/, ~/OSIT_Nextcloud/, "
"and the Cortex home/ directory (persona memory, tool audit logs, etc.). "
"Use this to read documentation, notes, CLAUDE.md files, config references, "
"or tool audit logs at home/{user}/tool_audit/YYYY-MM-DD.jsonl. "
"If given a directory path, returns a directory listing instead."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"path": types.Schema(type=types.Type.STRING, description="Absolute or home-relative path to the file (e.g. ~/agents_sync/CLAUDE.md or /home/scott/agents_sync/tasks/01_todo/)"),
"max_lines": types.Schema(type=types.Type.INTEGER, description="Optional line limit (default 500)"),
},
required=["path"],
),
),
types.FunctionDeclaration(
name="file_list",
description=(
"List the files and subdirectories in a directory. "
"Allowed paths: ~/agents_sync/, ~/OSIT_dev/, ~/DgrZone_Nextcloud/, ~/OSIT_Nextcloud/. "
"ADMIN ONLY."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"path": types.Schema(type=types.Type.STRING, description="Absolute or home-relative path to the directory"),
},
required=["path"],
),
),
types.FunctionDeclaration(
name="file_write",
description=(
"Write or append content to a file. "
"Write-allowed paths: ~/agents_sync/ and the Cortex home/ directory. "
"Creates parent directories if needed. "
"ADMIN ONLY. Requires user confirmation before executing."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"path": types.Schema(type=types.Type.STRING, description="Absolute or home-relative path to write to"),
"content": types.Schema(type=types.Type.STRING, description="Content to write"),
"mode": types.Schema(type=types.Type.STRING, description="'overwrite' (default, replaces file) or 'append' (adds to end)"),
},
required=["path", "content"],
),
),
]

View File

@@ -10,6 +10,7 @@ import json
import logging
import re
from google.genai import types
from config import settings
from persona import get_user
@@ -66,6 +67,16 @@ async def email_send(to: str, subject: str, body: str) -> str:
return "Failed to send email — check SMTP configuration in .env."
async def web_push(title: str, body: str, url: str = "") -> str:
"""Send a browser push notification to the current user's registered devices."""
import push_utils
username = get_user()
result = await push_utils.send_push(username, title, body, url)
if "error" in result:
return f"Push failed: {result['error']}"
return f"Push sent to {result['sent']} device(s) for {username} (pruned {result['pruned']} stale)."
async def nc_talk_send(message: str) -> str:
"""Send a message to the user via their configured notification channel.
@@ -80,3 +91,58 @@ async def nc_talk_send(message: str) -> str:
except Exception as e:
logger.warning("nc_talk_send error for %s: %s", username, e)
return f"Failed to send notification: {e}"
DECLARATIONS = [
types.FunctionDeclaration(
name="web_push",
description=(
"Send a browser push notification to the current user. Works even when the "
"Cortex tab is not open. Use for completing long tasks, reminders that fire "
"in the background, or anything the user should see immediately. "
"url is optional — if set, clicking the notification opens that URL."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"title": types.Schema(type=types.Type.STRING, description="Notification title (short)"),
"body": types.Schema(type=types.Type.STRING, description="Notification body text"),
"url": types.Schema(type=types.Type.STRING, description="Optional URL to open on click"),
},
required=["title", "body"],
),
),
types.FunctionDeclaration(
name="email_send",
description=(
"Send an email from the server's configured SMTP account. Use for delivering "
"summaries, reports, reminders, or any content the user wants emailed. "
"body is plain text; newlines are preserved."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"to": types.Schema(type=types.Type.STRING, description="Recipient email address"),
"subject": types.Schema(type=types.Type.STRING, description="Email subject line"),
"body": types.Schema(type=types.Type.STRING, description="Plain-text email body"),
},
required=["to", "subject", "body"],
),
),
types.FunctionDeclaration(
name="nc_talk_send",
description=(
"Send a proactive message to the user via their configured notification channel "
"(Nextcloud Talk by default). Use this to notify the user of completed background "
"tasks, important events, or anything they should know between sessions. "
"Requires notification_channel and notification_room set in channels.json."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"message": types.Schema(type=types.Type.STRING, description="The message to send to the user"),
},
required=["message"],
),
),
]

View File

@@ -16,6 +16,7 @@ import asyncio
from datetime import datetime, timezone
from pathlib import Path
from google.genai import types
from persona import persona_path
@@ -124,3 +125,55 @@ async def reminders_remove(index: int) -> str:
async def reminders_clear() -> str:
return await asyncio.to_thread(_reminders_clear)
DECLARATIONS = [
types.FunctionDeclaration(
name="reminders_add",
description=(
"Add a new reminder to REMINDERS.md. Reminders are automatically surfaced "
"in your context at the start of each session (Tier 2+). "
"Use this when the user asks you to remember something, follow up on something, "
"or surface a note at the next session."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"text": types.Schema(type=types.Type.STRING, description="The reminder text to add"),
"label": types.Schema(type=types.Type.STRING, description="Optional heading for this reminder (e.g. 'Follow up on NC Talk'). Defaults to current timestamp."),
},
required=["text"],
),
),
types.FunctionDeclaration(
name="reminders_list",
description=(
"Read all current pending reminders from REMINDERS.md. "
"Use this to check what reminders are queued before adding duplicates, "
"or to show the user what's pending."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="reminders_remove",
description=(
"Remove a single reminder by its number. "
"Call reminders_list first to get the numbered list, then pass the number of the reminder to remove."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"index": types.Schema(type=types.Type.INTEGER, description="The number of the reminder to remove (1 = first item in reminders_list output)."),
},
required=["index"],
),
),
types.FunctionDeclaration(
name="reminders_clear",
description=(
"Erase all pending reminders from REMINDERS.md. "
"Use this after you have acknowledged and acted on the reminders shown in your context."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
]

View File

@@ -17,6 +17,7 @@ import asyncio
from datetime import datetime, timezone
from pathlib import Path
from google.genai import types
from persona import persona_path
@@ -77,3 +78,51 @@ async def scratch_append(content: str, heading: str | None = None) -> str:
async def scratch_clear() -> str:
return await asyncio.to_thread(_scratch_clear)
DECLARATIONS = [
types.FunctionDeclaration(
name="scratch_read",
description=(
"Read the full contents of the scratchpad. "
"Use this to recall working notes, mid-task context, or anything previously jotted down. "
"The scratchpad is transient — nothing here is distilled or archived."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="scratch_write",
description=(
"Replace the entire scratchpad with new content. "
"Use this to set a clean working note, replacing whatever was there before. "
"For adding without replacing, use scratch_append instead."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"content": types.Schema(type=types.Type.STRING, description="The new scratchpad content (markdown supported)"),
},
required=["content"],
),
),
types.FunctionDeclaration(
name="scratch_append",
description=(
"Add a new section to the bottom of the scratchpad without replacing existing content. "
"Each section gets a timestamp heading unless you supply one."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"content": types.Schema(type=types.Type.STRING, description="The content to append (markdown supported)"),
"heading": types.Schema(type=types.Type.STRING, description="Optional section heading. Defaults to current UTC timestamp."),
},
required=["content"],
),
),
types.FunctionDeclaration(
name="scratch_clear",
description="Erase everything in the scratchpad. Use when the working notes are no longer needed.",
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
]

View File

@@ -11,6 +11,8 @@ import os
import subprocess
from pathlib import Path
from google.genai import types
logger = logging.getLogger(__name__)
# Absolute paths — resolved relative to this file so they work regardless of cwd
@@ -246,3 +248,87 @@ async def cortex_update() -> str:
lines.append("Call `cortex_restart` to apply the update.")
return "\n".join(lines)
DECLARATIONS = [
types.FunctionDeclaration(
name="shell_exec",
description=(
"Execute a shell command on the Cortex host machine and return its output. "
"Use for system diagnostics: disk usage (df -h), process status (ps aux), "
"directory listings (ls), memory (free -h), uptime, network info, log tails, etc. "
"Commands run as the Cortex service user. Timeout enforced (default 30s, max 120s). "
"Avoid destructive commands — prefer read-only system queries."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"command": types.Schema(type=types.Type.STRING, description="Shell command to run (e.g. 'df -h', 'ls ~/agents_sync/', 'journalctl --user -u cortex -n 50')"),
"working_dir": types.Schema(type=types.Type.STRING, description="Optional working directory (e.g. '~/agents_sync/projects'). Defaults to home directory."),
"timeout": types.Schema(type=types.Type.INTEGER, description="Timeout in seconds (default 30, max 120)"),
},
required=["command"],
),
),
types.FunctionDeclaration(
name="claude_allow_dir",
description=(
"Add a directory to Claude Code's auto-allow list so Claude can read or write "
"files there without prompting. Edits ~/.claude/settings.json on the local machine. "
"Use this when Claude is silently hanging or being blocked from accessing a directory. "
"Changes take effect in the next Claude Code session."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"path": types.Schema(type=types.Type.STRING, description="Absolute or home-relative path to the directory (e.g. ~/OSIT_dev/aether_api_fastapi or /home/scott/agents_sync)"),
"mode": types.Schema(type=types.Type.STRING, description="Permission mode: 'r' (read-only), 'w' (write-only), or 'rw' (both). Default: rw"),
},
required=["path"],
),
),
types.FunctionDeclaration(
name="cortex_restart",
description=(
"Restart the Cortex service via systemd. Schedules a restart 5 seconds from now. "
"The current connection will drop — inform the user to refresh the page. "
"Use after config changes, memory edits, or when the service needs a fresh start. "
"ADMIN ONLY."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="cortex_logs",
description=(
"Fetch recent lines from the Cortex systemd service journal. "
"Use for debugging errors, checking startup status, or reviewing recent activity. "
"ADMIN ONLY."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"lines": types.Schema(type=types.Type.INTEGER, description="Number of log lines to return (default 50, max 200)"),
},
),
),
types.FunctionDeclaration(
name="cortex_status",
description=(
"Return Cortex service status: current git branch and commit, how many commits "
"ahead/behind the remote, and the systemctl service state. "
"Use to check what version is running or whether the service is healthy. "
"ADMIN ONLY."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
types.FunctionDeclaration(
name="cortex_update",
description=(
"Pull the latest code from git, run a syntax check on all Python files, and report "
"what changed. Does NOT restart automatically — call cortex_restart separately after "
"reviewing the output. Will report syntax errors if the pull introduces broken code. "
"ADMIN ONLY. Requires confirmation."
),
parameters=types.Schema(type=types.Type.OBJECT, properties={}),
),
]

View File

@@ -20,6 +20,7 @@ import asyncio
from datetime import datetime, timezone
from pathlib import Path
from google.genai import types
from persona import persona_path
@@ -133,3 +134,70 @@ async def task_update(task_id: str, status: str | None = None, title: str | None
async def task_complete(task_id: str) -> str:
return await asyncio.to_thread(_task_complete, task_id)
DECLARATIONS = [
types.FunctionDeclaration(
name="task_list",
description=(
"List personal tasks from Inara's task list. "
"Use this to check what's on the list, review pending work, or find a task ID. "
"Optionally filter by status: 'todo', 'in_progress', or 'done'."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"status": types.Schema(type=types.Type.STRING, description="Filter by status: 'todo', 'in_progress', or 'done'. Omit to list all."),
},
),
),
types.FunctionDeclaration(
name="task_create",
description=(
"Add a new task to Inara's personal task list. "
"Use this when the user asks to remember something, add a to-do, or track a follow-up."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"title": types.Schema(type=types.Type.STRING, description="Short task title"),
"description": types.Schema(type=types.Type.STRING, description="Optional longer description or context"),
"priority": types.Schema(type=types.Type.STRING, description="Priority: 'low', 'normal', or 'high'. Default: normal."),
},
required=["title"],
),
),
types.FunctionDeclaration(
name="task_update",
description=(
"Update an existing task. Use task_list first to get the task ID. "
"Can update status, title, description, or priority. "
"To just mark complete, use task_complete instead."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"task_id": types.Schema(type=types.Type.STRING, description="Task ID (e.g. t_abc123) — get from task_list"),
"status": types.Schema(type=types.Type.STRING, description="New status: 'todo', 'in_progress', or 'done'"),
"title": types.Schema(type=types.Type.STRING, description="Updated title"),
"description": types.Schema(type=types.Type.STRING, description="Updated description"),
"priority": types.Schema(type=types.Type.STRING, description="Updated priority: 'low', 'normal', or 'high'"),
},
required=["task_id"],
),
),
types.FunctionDeclaration(
name="task_complete",
description=(
"Mark a task as done. Use task_list first to get the task ID. "
"Shorthand for task_update with status='done'."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"task_id": types.Schema(type=types.Type.STRING, description="Task ID (e.g. t_abc123) — get from task_list"),
},
required=["task_id"],
),
),
]

View File

@@ -6,6 +6,7 @@ import asyncio
import logging
import httpx
from google.genai import types
from config import settings
@@ -74,3 +75,41 @@ async def http_fetch(
except Exception as e:
logger.warning("http_fetch error for %s: %s", url, e)
return f"Error: {e}"
DECLARATIONS = [
types.FunctionDeclaration(
name="web_search",
description=(
"Search the web for current information. Use this when you need up-to-date "
"facts, news, documentation, or anything not in your training data."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"query": types.Schema(type=types.Type.STRING, description="The search query string"),
"max_results": types.Schema(type=types.Type.INTEGER, description="Number of results to return (default 5, max 10)"),
},
required=["query"],
),
),
types.FunctionDeclaration(
name="http_fetch",
description=(
"Fetch a specific URL and return the response. Unlike web_search, this hits "
"a direct URL — useful for health checks, JSON API endpoints, webhook testing, "
"or reading a specific page when you already know the URL. "
"Response body is capped at 8 KB."
),
parameters=types.Schema(
type=types.Type.OBJECT,
properties={
"url": types.Schema(type=types.Type.STRING, description="Full URL to fetch"),
"method": types.Schema(type=types.Type.STRING, description="HTTP method: GET (default), POST, HEAD"),
"body": types.Schema(type=types.Type.STRING, description="Optional request body (for POST requests)"),
"timeout": types.Schema(type=types.Type.INTEGER, description="Request timeout in seconds (default 15, max 60)"),
},
required=["url"],
),
),
]

75
cortex/usage_tracker.py Normal file
View File

@@ -0,0 +1,75 @@
"""
API usage and token tracking.
Writes daily buckets to home/{username}/usage.json:
{
"2026-05-01": {
"gemini_api/gemini-2.0-flash": {"calls": 3, "prompt_tokens": 8400, "completion_tokens": 520},
"local/llama3.2:latest": {"calls": 2, "prompt_tokens": 1200, "completion_tokens": 310}
}
}
Claude CLI and Gemini CLI backends produce no structured token data and are not tracked.
"""
import asyncio
import json
import logging
from datetime import date
from pathlib import Path
from config import settings
logger = logging.getLogger(__name__)
_LOCK = asyncio.Lock()
def _usage_path(username: str) -> Path:
return settings.home_root() / username / "usage.json"
async def record(
username: str,
backend: str,
model_name: str,
prompt_tokens: int,
completion_tokens: int,
) -> None:
"""Append one call's token counts to the daily usage log for this user.
backend — "gemini_api" | "local"
model_name — the exact model string (e.g. "gemini-2.0-flash", "llama3.2:latest")
"""
path = _usage_path(username)
today = date.today().isoformat()
key = f"{backend}/{model_name}"
async with _LOCK:
try:
data: dict = json.loads(path.read_text()) if path.exists() else {}
except Exception:
data = {}
entry = data.setdefault(today, {}).setdefault(
key, {"calls": 0, "prompt_tokens": 0, "completion_tokens": 0}
)
entry["calls"] += 1
entry["prompt_tokens"] += prompt_tokens
entry["completion_tokens"] += completion_tokens
try:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, indent=2))
except Exception as e:
logger.warning("Failed to write usage data to %s: %s", path, e)
def read_usage(username: str) -> dict:
"""Return the full usage dict for this user. Empty dict if no file yet."""
path = _usage_path(username)
try:
return json.loads(path.read_text()) if path.exists() else {}
except Exception:
return {}

View File

@@ -89,9 +89,9 @@ See `ARCH__Intelligence_Layer.md` for full design.
- [x] Tool: `ae_journal_entry_disable` — soft-delete via enable=false — 2026-04-28
- [x] Tool: `ae_journal_entry_append` — read→append timestamped section→write (running logs) — 2026-04-28
- [x] Tool: `ae_journal_entry_prepend` — read→prepend timestamped section→write (newest-first logs) — 2026-04-28
- [ ] Import script: walk a markdown directory, chunk by H2 section, create entries
- [ ] Target: markdown files from `~/DgrZone_Nextcloud/` and `~/OSIT_Nextcloud/`
- [ ] Tag strategy: source path, date, topic tags from frontmatter or filename
- [x] Import script: walk a markdown directory, chunk by H2 section, create entries — 2026-05-05
- [x] Target: markdown files from `~/DgrZone_Nextcloud/` and `~/OSIT_Nextcloud/` — 2026-05-05
- [x] Tag strategy: source path, topic tags from path components — 2026-05-05
---
@@ -116,8 +116,8 @@ Read before finalising either design.
### [Backend] API usage / cost tracking
Multi-user setup with real Gemini/Claude API costs. Track per-user token consumption
so Scott can see who's spending what.
- [ ] Count input + output tokens per `/chat` and `/orchestrate` call (all backends return usage)
- [ ] Append to `home/{user}/usage.json` — daily buckets, per-model breakdown
- [x] Count input + output tokens — local backend (OpenAI `usage` field) + Gemini API (`usage_metadata`) — 2026-05-05
- [x] Append to `home/{user}/usage.json` — daily buckets, per-model breakdown — 2026-05-05
- [ ] Expose via `/api/usage` endpoint; add a summary row to the Settings page
- [ ] Optional: soft spending limit with a warning toast when exceeded

407
scripts/import_knowledge.py Executable file
View File

@@ -0,0 +1,407 @@
#!/usr/bin/env python3
"""
Knowledge base importer — walks a markdown directory and creates AE journal entries.
Uses a local LLM to generate a 1-2 sentence summary for each chunk.
Tracks progress in a state file so interrupted runs can be resumed.
Usage:
python import_knowledge.py --source ~/DgrZone_Nextcloud --journal <journal_id>
python import_knowledge.py --source ~/OSIT_Nextcloud --journal <journal_id> --dry-run
python import_knowledge.py --source ~/DgrZone_Nextcloud/Notes --journal <journal_id> --limit 5
Reads credentials from cortex/.env (relative to this script's parent directory)
or from environment variables:
AE_API_URL, AE_API_KEY, AE_ACCOUNT_ID
LOCAL_API_URL, LOCAL_API_KEY, LOCAL_MODEL
"""
import argparse
import hashlib
import json
import os
import re
import sys
import time
from datetime import datetime, timezone
from pathlib import Path
# ── Bootstrap: load .env from cortex/.env if not already set ─────────────────
_ENV_PATH = Path(__file__).parent.parent / "cortex" / ".env"
def _load_env(path: Path) -> None:
if not path.exists():
return
for line in path.read_text().splitlines():
line = line.strip()
if not line or line.startswith("#") or "=" not in line:
continue
key, _, val = line.partition("=")
key = key.strip()
if key not in os.environ:
os.environ[key] = val.strip().strip('"').strip("'")
_load_env(_ENV_PATH)
# ── Constants ─────────────────────────────────────────────────────────────────
# Dirs to skip regardless of source
_DEFAULT_EXCLUDE = {
"temp", "Temp", "tmp", "Tmp", "test", "Test",
"Temporary Share", "Test Share", ".obsidian", "media", "Photos",
}
# Max characters per journal entry chunk
_DEFAULT_MAX_CHUNK = 8_000
# Delay between API calls (seconds) to avoid hammering the LLM
_LLM_DELAY = 1.0
_AE_DELAY = 0.3
# ── Path / tag utilities ──────────────────────────────────────────────────────
def _path_tags(source_root: Path, file_path: Path) -> list[str]:
"""Derive tags from path components relative to the source root."""
rel = file_path.relative_to(source_root)
parts = list(rel.parts[:-1]) # exclude filename itself
tags = []
for part in parts:
cleaned = re.sub(r"[^a-zA-Z0-9 ]", " ", part).strip().lower()
words = cleaned.split()
tags.extend(w for w in words if len(w) > 2)
# Add source root name as top-level tag
tags.insert(0, source_root.name.lower().replace("_nextcloud", "").replace("_", ""))
return list(dict.fromkeys(tags)) # deduplicate preserving order
def _file_title(file_path: Path, content: str) -> str:
"""Extract the first H1 heading or fall back to the filename stem."""
m = re.search(r"^# (.+)$", content, re.MULTILINE)
if m:
return m.group(1).strip()
return file_path.stem.replace("_", " ").replace("-", " ")
# ── Chunking ──────────────────────────────────────────────────────────────────
def _chunk_content(
file_path: Path,
content: str,
source_root: Path,
max_size: int,
) -> list[dict]:
"""
Returns a list of chunk dicts:
{
"chunk_key": str, # unique ID for state tracking
"title": str,
"content": str,
"tags": list[str],
"path": str,
}
"""
base_title = _file_title(file_path, content)
base_tags = _path_tags(source_root, file_path)
rel_path = str(file_path.relative_to(source_root))
if len(content) <= max_size:
h = hashlib.sha1(content.encode()).hexdigest()[:12]
return [{
"chunk_key": f"{rel_path}#{h}",
"title": base_title,
"content": content,
"tags": base_tags,
"path": rel_path,
}]
# Split by H2 headings
sections = re.split(r"^(## .+)$", content, flags=re.MULTILINE)
# sections alternates: [preamble, heading, body, heading, body, ...]
chunks = []
preamble = sections[0].strip()
pairs = list(zip(sections[1::2], sections[2::2]))
if not pairs:
# No H2 found — hard split by max_size
for i in range(0, len(content), max_size):
part = content[i:i + max_size]
h = hashlib.sha1(part.encode()).hexdigest()[:12]
chunks.append({
"chunk_key": f"{rel_path}#part{i}_{h}",
"title": f"{base_title} (part {i // max_size + 1})",
"content": part,
"tags": base_tags,
"path": rel_path,
})
return chunks
for heading, body in pairs:
section_title = heading.lstrip("#").strip()
section_content = f"{heading}\n{body}".strip()
# Prepend preamble to first section if it has meaningful content
if preamble and chunks == []:
section_content = f"{preamble}\n\n{section_content}"
# If section itself is too big, hard split it
if len(section_content) > max_size:
for i in range(0, len(section_content), max_size):
part = section_content[i:i + max_size]
h = hashlib.sha1(part.encode()).hexdigest()[:12]
chunks.append({
"chunk_key": f"{rel_path}#{section_title}#part{i}_{h}",
"title": f"{base_title}{section_title} (part {i // max_size + 1})",
"content": part,
"tags": base_tags + [section_title.lower()[:40]],
"path": rel_path,
})
else:
h = hashlib.sha1(section_content.encode()).hexdigest()[:12]
chunks.append({
"chunk_key": f"{rel_path}#{section_title}#{h}",
"title": f"{base_title}{section_title}",
"content": section_content,
"tags": base_tags + [section_title.lower()[:40]],
"path": rel_path,
})
return chunks
# ── LLM summarization ─────────────────────────────────────────────────────────
def _summarize(content: str, llm_url: str, llm_key: str, llm_model: str) -> str:
"""Call the local LLM to generate a 1-2 sentence summary. Returns "" on failure."""
import urllib.request
# Truncate for prompt economy
snippet = content[:3000] if len(content) > 3000 else content
prompt = (
"Summarize the following note in 1-2 sentences. "
"Be specific and factual. Do not start with 'This note' or 'This document'.\n\n"
f"{snippet}"
)
payload = json.dumps({
"model": llm_model,
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 150,
"temperature": 0.3,
}).encode()
headers = {"Content-Type": "application/json"}
if llm_key:
headers["Authorization"] = f"Bearer {llm_key}"
# Try Open WebUI path first, fall back to standard OpenAI path
for path in ("/api/chat/completions", "/chat/completions"):
url = llm_url.rstrip("/") + path
req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
try:
with urllib.request.urlopen(req, timeout=60) as resp:
data = json.loads(resp.read())
return data["choices"][0]["message"]["content"].strip()
except Exception:
continue
return ""
# ── AE API ────────────────────────────────────────────────────────────────────
def _ae_create_entry(
journal_id: str,
title: str,
content: str,
summary: str,
tags: list[str],
ae_url: str,
ae_key: str,
ae_account: str,
) -> str:
"""POST to AE API. Returns the new entry's id_random or raises on error."""
import urllib.request
url = f"{ae_url.rstrip('/')}/v3/crud/journal/{journal_id}/journal_entry/"
payload = json.dumps({
"name": title,
"content": content,
"summary": summary,
"tags": tags,
}).encode()
headers = {
"Content-Type": "application/json",
"x-aether-api-key": ae_key,
"x-account-id": ae_account,
}
req = urllib.request.Request(url, data=payload, headers=headers, method="POST")
with urllib.request.urlopen(req, timeout=30) as resp:
data = json.loads(resp.read())
return (
data.get("data", {}).get("journal_entry_id")
or data.get("data", {}).get("id_random")
or data.get("id_random")
or "?"
)
# ── State file ────────────────────────────────────────────────────────────────
def _load_state(state_file: Path) -> dict:
if state_file.exists():
try:
return json.loads(state_file.read_text())
except Exception:
pass
return {"imported": {}}
def _save_state(state_file: Path, state: dict) -> None:
state_file.write_text(json.dumps(state, indent=2))
# ── File walker ───────────────────────────────────────────────────────────────
def _walk_markdown(source: Path, exclude: set[str]) -> list[Path]:
files = []
for f in sorted(source.rglob("*.md")):
if any(part in exclude for part in f.parts):
continue
if f.stat().st_size < 50: # skip near-empty files
continue
files.append(f)
return files
# ── Main ──────────────────────────────────────────────────────────────────────
def main() -> None:
parser = argparse.ArgumentParser(description="Import markdown notes into AE Journal")
parser.add_argument("--source", required=True, help="Root directory to import from")
parser.add_argument("--journal", required=True, help="Target AE journal id_random")
parser.add_argument("--dry-run", action="store_true", help="Preview without creating entries")
parser.add_argument("--limit", type=int, default=0, help="Stop after N chunks (0 = unlimited)")
parser.add_argument("--max-chunk", type=int, default=_DEFAULT_MAX_CHUNK, help="Max chars per chunk")
parser.add_argument("--exclude", default="", help="Extra dir names to skip (comma-separated)")
parser.add_argument("--state-file", default="import_state.json", help="State tracking file")
parser.add_argument("--no-llm", action="store_true", help="Skip LLM summarization (faster)")
parser.add_argument("--ae-url", default=os.environ.get("AE_API_URL", ""), help="AE API URL")
parser.add_argument("--ae-key", default=os.environ.get("AE_API_KEY", ""), help="AE API key")
parser.add_argument("--ae-account", default=os.environ.get("AE_ACCOUNT_ID", ""), help="AE account ID")
parser.add_argument("--llm-url", default=os.environ.get("LOCAL_API_URL", ""), help="Local LLM API URL")
parser.add_argument("--llm-key", default=os.environ.get("LOCAL_API_KEY", ""), help="Local LLM API key")
parser.add_argument("--llm-model", default=os.environ.get("LOCAL_MODEL", ""), help="Local LLM model name")
args = parser.parse_args()
source = Path(args.source).expanduser().resolve()
if not source.exists():
print(f"ERROR: source directory not found: {source}", file=sys.stderr)
sys.exit(1)
if not args.dry_run:
if not args.ae_url or not args.ae_key or not args.ae_account:
print("ERROR: AE_API_URL, AE_API_KEY, and AE_ACCOUNT_ID are required (or use --dry-run)", file=sys.stderr)
sys.exit(1)
use_llm = not args.no_llm and bool(args.llm_url) and bool(args.llm_model)
if not use_llm and not args.no_llm:
print("NOTE: LLM summarization disabled (LOCAL_API_URL or LOCAL_MODEL not set). Use --no-llm to silence this.")
exclude = _DEFAULT_EXCLUDE | {d.strip() for d in args.exclude.split(",") if d.strip()}
state_file = Path(args.state_file)
state = _load_state(state_file)
print(f"Source: {source}")
print(f"Journal: {args.journal}")
print(f"Dry run: {args.dry_run}")
print(f"LLM: {'enabled (' + args.llm_model + ')' if use_llm else 'disabled'}")
print(f"State file: {state_file} ({len(state['imported'])} already imported)")
print()
files = _walk_markdown(source, exclude)
print(f"Found {len(files)} markdown files")
created = 0
skipped = 0
errors = 0
total_chunks = 0
for file_path in files:
try:
content = file_path.read_text(encoding="utf-8", errors="replace")
except Exception as e:
print(f" SKIP (read error): {file_path.name}{e}")
errors += 1
continue
chunks = _chunk_content(file_path, content, source, args.max_chunk)
total_chunks += len(chunks)
for chunk in chunks:
key = chunk["chunk_key"]
if key in state["imported"]:
skipped += 1
continue
print(f" {'[DRY RUN] ' if args.dry_run else ''}IMPORT: {chunk['title'][:70]}")
summary = ""
if use_llm:
try:
summary = _summarize(chunk["content"], args.llm_url, args.llm_key, args.llm_model)
time.sleep(_LLM_DELAY)
except Exception as e:
print(f" LLM error (continuing without summary): {e}")
if not args.dry_run:
try:
entry_id = _ae_create_entry(
journal_id=args.journal,
title=chunk["title"],
content=chunk["content"],
summary=summary,
tags=chunk["tags"],
ae_url=args.ae_url,
ae_key=args.ae_key,
ae_account=args.ae_account,
)
state["imported"][key] = {
"entry_id": entry_id,
"imported_at": datetime.now(timezone.utc).isoformat(),
"path": chunk["path"],
"title": chunk["title"],
}
_save_state(state_file, state)
time.sleep(_AE_DELAY)
created += 1
except Exception as e:
print(f" AE API error: {e}")
errors += 1
else:
created += 1
if args.limit and created >= args.limit:
print(f"\nReached --limit {args.limit}. Stopping.")
_print_summary(created, skipped, errors, total_chunks, args.dry_run)
return
_print_summary(created, skipped, errors, total_chunks, args.dry_run)
def _print_summary(created: int, skipped: int, errors: int, total: int, dry_run: bool) -> None:
label = "Would create" if dry_run else "Created"
print(f"\n{'=' * 50}")
print(f"{label}: {created} entries")
print(f"Skipped (already imported): {skipped}")
print(f"Errors: {errors}")
print(f"Total chunks processed: {total}")
if __name__ == "__main__":
main()