Operator observability: memory inspector and diagnostics workflow

[aurexav]

Summary

Provide an operator-friendly memory inspection and diagnostics workflow.

Problem

Troubleshooting currently requires deep DB/manual inspection, which slows iteration and quality tuning.

In Scope

• Build read-only inspector for sessions/traces/explain outputs.
• Add status and diagnostics commands/endpoints for common failure classes.
• Improve troubleshooting documentation for ingestion/retrieval loops.

Out of Scope

• Full admin dashboard productization.
• Write-capable production management UI.

Deliverables

• Inspector entrypoint (local-first).
• Diagnostics surfaces.
• Ops runbook updates.

Acceptance Criteria

• Developers can inspect memory decisions end-to-end without direct DB spelunking.
• Common failures are diagnosable with documented commands.
• Inspector works on desktop and mobile local usage.

Dependencies

• Search: graph relation context and explain integration #50
• Quality gate: CI regression checks with elf-eval trace compare #54

Related

• Add a read-only web viewer for sessions and traces #26

Implementation Checklist

• Inspector prototype.
• Diagnostics endpoints/commands.
• Runbook docs.
• Developer workflow validation.

Done When

• Debugging memory quality is fast and repeatable for contributors.

[aurexav]

Proposed execution plan (goal alignment before implementation)

Decision

• Use OTel/Grafana (Tempo) for runtime performance + request tracing.
• Use Postgres as the source of truth for domain explainability (search_traces/search_trace_items/etc.).
• Avoid building a heavy bespoke UI first: start with a DB-backed inspector via Metabase. Add a thin custom viewer only if Metabase + Grafana do not cover the operator workflow.

Phase 1 (MVP: 1-3 days, “operator can debug a trace end-to-end”)

• Add/confirm “entry point” list APIs so operators do not need to paste trace_id/search_id by hand:
  • Recent traces list (filter by time/project/agent/read_profile; paginate)
  • Recent searches list (optional, if separate from traces)
• Ensure drilldown works:
  • Trace -> items -> item explain
  • Trajectory view (stages, per-stage latency, candidate counts, keep counts)
  • Export JSON for trace + item (bounded + full)
• Metabase dashboards/questions:
  • Recent traces
  • Trace detail
  • Item detail
  • Stage latency summary

Phase 2 (enhancements; tracked in #41)

• Candidate-set evolution panels (prefilter exclusions, fusion contributions, rerank deltas)
• Recursive/structured/graph context contribution visibility

Acceptance criteria (MVP)

• Given a single trace_id, an operator can answer in <5 minutes:
  • Which stage dominated latency?
  • How many candidates were produced/kept per stage?
  • Why the top-1 item won (matched_terms/fields + ranking terms)?
  • What relation_context (if enabled) was attached and what evidence_note_ids back it?
  • Export a reproducible trace bundle.

Open questions

• Do we want the inspector to be “Metabase-first” only, or ship a minimal embedded web viewer in elf-api for better drilldown/export UX?
• Default access boundary: loopback-only vs admin-authenticated (even if read-only).