From a384979d5465c135596cd90efd6b0a36c473b91d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 21 Feb 2026 22:58:04 +0000 Subject: [PATCH 1/6] =?UTF-8?q?docs:=20Gap=20analysis=20=E2=80=94=20claude?= =?UTF-8?q?-mem=20vs=20SerialMemory=20feature=20comparison?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Comprehensive analysis of features in claude-mem that could improve SerialMemory's Claude Code integration. Identifies 10 gaps across auto-capture hooks, progressive disclosure, context injection, structured observations, privacy tags, timeline navigation, token tracking, plugin packaging, transcript integration, and web UI. Key P0 recommendations: token-efficient progressive search, automatic session capture via lifecycle hooks, and automatic context injection at session start. https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- ...gap-analysis-claude-mem-vs-serialmemory.md | 346 ++++++++++++++++++ 1 file changed, 346 insertions(+) create mode 100644 docs/gap-analysis-claude-mem-vs-serialmemory.md diff --git a/docs/gap-analysis-claude-mem-vs-serialmemory.md b/docs/gap-analysis-claude-mem-vs-serialmemory.md new file mode 100644 index 0000000..67fac2e --- /dev/null +++ b/docs/gap-analysis-claude-mem-vs-serialmemory.md @@ -0,0 +1,346 @@ +# Gap Analysis: claude-mem vs SerialMemory + +**Date:** 2026-02-21 +**Purpose:** Identify Claude-specific memory features from [claude-mem](https://github.com/thedotmack/claude-mem) that could be added to SerialMemory to improve Claude Code integration. + +--- + +## Executive Summary + +claude-mem is a Claude Code plugin purpose-built for **automatic, invisible session capture** with AI-powered compression and progressive context injection. SerialMemory is a **full knowledge graph** with event sourcing, entity extraction, multi-hop reasoning, and confidence decay — far more powerful as a memory backend, but missing several Claude Code-specific ergonomic features that claude-mem gets right. + +The key insight: **claude-mem excels at the "zero-effort capture" developer experience**, while SerialMemory excels at **deep memory management, reasoning, and integrity**. The features to adopt are primarily in the UX/integration layer, not the storage layer. + +--- + +## Feature Comparison Matrix + +| Feature | claude-mem | SerialMemory | Gap? | +|---------|-----------|--------------|------| +| **Storage Backend** | SQLite + Chroma | PostgreSQL + pgvector | No gap — SM is superior | +| **Semantic Search** | Chroma vector DB | pgvector cosine similarity | No gap | +| **Full-text Search** | SQLite FTS5 | PostgreSQL tsvector | No gap | +| **Hybrid Search** | Yes (Chroma + FTS5) | Yes (vector + text + combined) | No gap | +| **Entity Extraction** | No | Pattern + NER | SM ahead | +| **Knowledge Graph** | No | Full graph w/ relationships | SM ahead | +| **Multi-hop Reasoning** | No | Yes (graph traversal) | SM ahead | +| **Confidence Decay** | No | Exponential half-life | SM ahead | +| **Event Sourcing** | No | 13 event types, append-only | SM ahead | +| **Memory Lifecycle** | No | Update/merge/split/decay/expire/supersede | SM ahead | +| **Memory Integrity** | No | SHA-256 content hashing | SM ahead | +| **Contradiction Detection** | No | Yes | SM ahead | +| **Export Formats** | No | JSON/CSV/GraphML/Markdown | SM ahead | +| **Auto-Capture via Hooks** | Yes — mature, 5 lifecycle events | Basic (JSONL drain) | **GAP** | +| **AI Summarization** | Yes — Claude Agent SDK | Yes — LLM-based | Partial parity | +| **Progressive Disclosure** | 3-layer search → timeline → fetch | No — returns full results | **GAP** | +| **Token Budget Tracking** | Yes — token economics per result | No | **GAP** | +| **Session Context Injection** | Auto-injects via SessionStart hook | Manual (instantiate_context) | **GAP** | +| **Observation Types** | Structured (title/subtitle/facts/narrative/concepts) | Free-text content | **GAP** | +| **File Tracking** | files_read / files_modified per observation | Entity extraction only | **GAP** | +| **Privacy Tags** | `` content exclusion | No | **GAP** | +| **Timeline View** | Chronological with anchoring | No dedicated timeline tool | **GAP** | +| **Discovery Tokens** | Tracks "work tokens" vs "read tokens" | No | **GAP** | +| **Claude Code Plugin** | Native plugin (marketplace) | MCP server (manual config) | **GAP** | +| **Web Viewer UI** | React UI at localhost:37777 | No web UI | **GAP** (lower priority) | +| **Prior Session Messages** | Extracts from Claude transcript files | No | **GAP** | +| **Project Scoping** | Per-project filtering | Workspace scoping | Equivalent | +| **Worker Process** | Background Bun worker on :37777 | In-process (MCP STDIO) | Different arch | +| **Worktree Support** | Multi-project interleaved queries | No | **GAP** (minor) | + +--- + +## Detailed Gap Analysis + +### GAP 1: Automatic Session Capture via Claude Code Hooks (HIGH PRIORITY) + +**What claude-mem does:** Registers 5 lifecycle hooks (SessionStart, UserPromptSubmit, PostToolUse, Stop, SessionEnd) that **automatically** capture tool usage, file edits, and session activity without any manual effort from the user or the AI agent. The hooks run as shell scripts and write observation data to the worker service. + +**What SerialMemory has:** `AutoCaptureTools.cs` reads JSONL files from `~/.cc-serialmemory/sessions/` and a `session-capture.sh` hook exists, but: +- The JSONL capture is minimal (timestamp, tool, file, result) +- No automatic PostToolUse capture of rich observation data +- No automatic SessionStart context injection +- No automatic SessionEnd summarization trigger +- Hook integration is documented but not bundled as a ready-to-install package + +**Recommendation:** Create a complete set of hook scripts that: +1. **SessionStart**: Auto-run `instantiate_context` and inject recent memories +2. **PostToolUse**: Capture structured observations (tool name, files touched, result summary) +3. **Stop**: Auto-drain captures and trigger `summarize_session` +4. **SessionEnd**: Final drain + summary persistence +5. Package as installable hook scripts (not just documentation) + +**Implementation approach:** +- Create `hooks/` directory with shell scripts for each lifecycle event +- Each hook calls the SerialMemory MCP worker/API or writes to the JSONL log +- Add an `install-hooks` script that copies to `~/.claude/settings.json` +- Enhance `AutoCaptureTools` to capture richer structured data (see GAP 5) + +--- + +### GAP 2: Progressive Disclosure / Token-Efficient Search (HIGH PRIORITY) + +**What claude-mem does:** Implements a deliberate 3-layer workflow: +1. **`search`** returns a compact index (~50-100 tokens per result): just IDs, timestamps, type, title +2. **`timeline`** shows chronological context around a specific result +3. **`get_observations`** fetches full details only for selected IDs + +This achieves ~10x token savings by filtering before fetching. The `__IMPORTANT` tool documents this pattern for Claude. + +**What SerialMemory has:** `memory_search` returns full memory content in every result. No way to get a compact index first, then drill down. This means every search consumes significant tokens even when most results are irrelevant. + +**Recommendation:** Add three new tools: +1. **`memory_search_index`** — Returns compact results: `{id, created_at, memory_type, title/first_line, similarity_score, entity_count}` (~50-80 tokens per result) +2. **`memory_timeline`** — Given an anchor memory ID or timestamp, return N memories before and after in chronological order (compact format) +3. **`memory_fetch`** — Batch-fetch full details by memory IDs (like `get_observations`) + +**Keep existing `memory_search`** for backward compatibility — it remains useful when you know you want full content. + +**Implementation approach:** +- `memory_search_index`: Same search logic but project only minimal fields +- `memory_timeline`: New query `SELECT * FROM memories WHERE workspace_id = ? ORDER BY created_at, LIMIT ? OFFSET ?` with anchor-based windowing +- `memory_fetch`: Simple `SELECT * FROM memories WHERE id = ANY(?)` batch fetch + +--- + +### GAP 3: Automatic Context Injection at SessionStart (HIGH PRIORITY) + +**What claude-mem does:** On SessionStart, automatically: +1. Queries recent observations and summaries for the current project +2. Builds a "context" document with sections: header, timeline, summary, "Previously" (prior session messages) +3. Injects this context into the session so Claude starts with full awareness + +The `ContextBuilder` assembles multiple sections, and `ObservationCompiler` merges observations with session summaries chronologically. + +**What SerialMemory has:** `instantiate_context` tool exists and returns recent memories + goals + user persona, but: +- Must be explicitly called by the agent (not automatic) +- Doesn't include prior session transcript context +- No "Previously" section from Claude transcript files +- No configurable token budget for context injection + +**Recommendation:** +1. Enhance `instantiate_context` output to include a structured context document: + - Recent session summary (from last `session_summary` memory) + - Active goals + - User persona excerpt + - Recent key memories (last N, filtered by type) + - Files recently modified (from auto-capture data) +2. Create a SessionStart hook that auto-calls `instantiate_context` and injects the response +3. Add a configurable token budget that truncates context to fit +4. Optionally: parse Claude Code transcript files for "Previously" context + +--- + +### GAP 4: Token Budget Tracking (MEDIUM PRIORITY) + +**What claude-mem does:** Tracks "discovery tokens" (how many tokens of work were done to find/create information) vs "read tokens" (how many tokens are consumed displaying it). Shows savings percentage. This helps users understand the cost of memory operations. + +**What SerialMemory has:** No token tracking whatsoever. + +**Recommendation:** +1. Add `estimated_tokens` to search results (character count / 4 as rough estimate) +2. Track token cost in tool responses: `{results: [...], meta: {result_count, total_tokens, avg_tokens_per_result}}` +3. For the progressive disclosure tools (GAP 2), show savings: "Index: 150 tokens | Full fetch would be: 1,500 tokens | Savings: 90%" + +**Implementation approach:** +- Simple character-count estimation (chars / 4) +- Add `meta` field to all search/fetch tool responses +- No external token counter needed + +--- + +### GAP 5: Structured Observation Format (MEDIUM PRIORITY) + +**What claude-mem does:** Each observation has rich structure: +- `type` (bugfix, feature, decision, etc.) +- `title` and `subtitle` +- `facts` (array of factual points) +- `narrative` (extended description) +- `concepts` (categorization tags) +- `files_read` and `files_modified` (file tracking) +- `prompt_number` (which prompt in the session) +- `discovery_tokens` (token cost) + +**What SerialMemory has:** Memories have `content` (free text), `source`, `memory_type`, and metadata JSON. No structured facts/narrative/concepts fields. File tracking comes only from entity extraction. + +**Recommendation:** Enhance the memory data model with optional structured fields: +1. Add `title` column (or extract from first line of content) +2. Add `facts` JSONB column for structured factual points +3. Add `concepts` JSONB column for categorization tags (distinct from entities) +4. Add `files_read` and `files_modified` JSONB columns +5. Enhance `memory_ingest` to accept these structured fields optionally +6. Use these fields for more precise search filtering + +**Implementation approach:** +- Add columns to `memories` table (nullable, backward compatible) +- Extend `memory_ingest` schema with optional structured params +- Auto-populate from auto-capture data +- Use concepts for lightweight tagging (faster than entity extraction) + +--- + +### GAP 6: Privacy Tags (MEDIUM PRIORITY) + +**What claude-mem does:** Supports `content` tags. Content within these tags is stripped at the hook layer (edge processing) before reaching the database. This gives users control over what gets stored. + +**What SerialMemory has:** No privacy mechanism. Everything passed to `memory_ingest` is stored. + +**Recommendation:** +1. Add `` tag stripping in the `memory_ingest` pipeline +2. Strip before embedding generation and storage +3. Implement at the KnowledgeGraphService level (not just MCP layer) so it works for all ingestion paths +4. Optionally: add a `` tag that stores content but excludes from search results (different from `` which excludes from storage entirely) + +**Implementation approach:** +- Regex strip `.*?` from content before processing +- Add to `KnowledgeGraphService.IngestMemoryAsync()` +- Log stripped content count for debugging (not the content itself) + +--- + +### GAP 7: Timeline / Chronological Navigation (MEDIUM PRIORITY) + +**What claude-mem does:** The `timeline` tool lets you anchor on a specific observation and see N items before and after it chronologically. Supports depth_before/depth_after parameters. This enables "time travel" through session history. + +**What SerialMemory has:** No dedicated timeline navigation. Search returns results by relevance score, not chronology. + +**Recommendation:** Add a `memory_timeline` tool: +- Input: `anchor_id` (memory ID) or `anchor_time` (ISO timestamp), `depth_before`, `depth_after`, optional `project`/`memory_type` filter +- Output: Chronologically ordered memories around the anchor point +- Use compact format (from GAP 2) by default, with option for full content + +**Implementation approach:** +- Query: `SELECT * FROM memories WHERE created_at <= anchor ORDER BY created_at DESC LIMIT depth_before` UNION `SELECT * FROM memories WHERE created_at > anchor ORDER BY created_at ASC LIMIT depth_after` +- Add to core tools alongside `memory_search` + +--- + +### GAP 8: Claude Code Plugin Packaging (LOW PRIORITY — FUTURE) + +**What claude-mem does:** Distributed as a Claude Code plugin via marketplace (`/plugin marketplace add thedotmack/claude-mem`). Includes plugin.json manifest, hooks configuration, skills definition. One-command install. + +**What SerialMemory has:** MCP server requiring manual `claude_desktop_config.json` editing. No plugin packaging. + +**Recommendation:** Investigate Claude Code plugin format when it stabilizes: +1. Create `.claude-plugin/plugin.json` manifest +2. Bundle hooks as plugin hooks (not user-managed) +3. Create a `mem-search` skill equivalent +4. Enable marketplace distribution + +**Note:** This is lower priority because the plugin marketplace is relatively new and the MCP approach works well. Revisit when plugin ecosystem matures. + +--- + +### GAP 9: Prior Session Transcript Integration (LOW PRIORITY) + +**What claude-mem does:** The `ContextBuilder` reads Claude Code transcript files from the filesystem to extract prior assistant messages and inject them as a "Previously" section in context. This provides direct session continuity. + +**What SerialMemory has:** Session continuity through explicit `memory_ingest` + `instantiate_context`, not transcript parsing. + +**Recommendation:** Consider adding optional transcript integration: +1. Read Claude Code session transcripts from `~/.claude/projects/` directory +2. Extract key assistant messages from the most recent session +3. Include as a "Previously" section in `instantiate_context` output +4. Make this opt-in (privacy considerations) + +**Note:** This creates coupling to Claude Code's internal file format, which may change. SerialMemory's approach of explicit memory ingestion is more robust long-term. Only implement if transcript format stabilizes. + +--- + +### GAP 10: Web Viewer UI (LOW PRIORITY) + +**What claude-mem does:** Ships a React-based viewer accessible at `http://localhost:37777` with SSE-based real-time updates. Shows observation timeline, search interface, and session history. + +**What SerialMemory has:** No web UI. API server exists (`SerialMemory.Api`) with SignalR but no frontend. + +**Recommendation:** This is nice-to-have but not critical for Claude Code integration. If pursued: +1. Add a simple web UI to `SerialMemory.Api` +2. Memory timeline visualization +3. Knowledge graph explorer +4. Search interface +5. Session history viewer + +--- + +## Implementation Priority Matrix + +| Priority | Gap | Effort | Impact | +|----------|-----|--------|--------| +| **P0** | GAP 1: Auto-capture hooks | Medium | Removes manual effort entirely | +| **P0** | GAP 2: Progressive disclosure | Medium | ~10x token savings | +| **P0** | GAP 3: Auto context injection | Low | Automatic session awareness | +| **P1** | GAP 5: Structured observations | Medium | Better search/filtering | +| **P1** | GAP 7: Timeline navigation | Low | Chronological exploration | +| **P1** | GAP 4: Token budget tracking | Low | Cost visibility | +| **P2** | GAP 6: Privacy tags | Low | User trust | +| **P2** | GAP 8: Plugin packaging | High | Distribution ease | +| **P3** | GAP 9: Transcript integration | Medium | Direct continuity | +| **P3** | GAP 10: Web viewer | High | Visualization | + +--- + +## What SerialMemory Does Better (No Action Needed) + +These are areas where SerialMemory is ahead and should be preserved: + +1. **Knowledge Graph** — Entity extraction, relationships, multi-hop traversal. claude-mem has nothing comparable. +2. **Event Sourcing** — Full audit trail with 13 event types. Append-only, immutable. +3. **Memory Lifecycle** — Merge, split, decay, reinforce, expire, supersede. claude-mem only has create/read. +4. **Confidence Decay** — Exponential half-life model with reinforcement. claude-mem memories are static. +5. **Contradiction Detection** — Automatic semantic conflict detection. +6. **Memory Integrity** — SHA-256 hash verification on read. +7. **Multi-axis Retrieval** — 6 scoring factors (semantic, recency, confidence, affinity, directive match, contradiction penalty). +8. **Export System** — JSON, CSV, GraphML, Cytoscape, Obsidian Markdown. +9. **Workspace Isolation** — Full RLS-based workspace scoping. +10. **Engineering Reasoning** — Domain-specific analysis (power integrity, signal integrity, dependency analysis). +11. **Goals System** — Persistent goals across sessions. +12. **State Snapshots** — Checkpoint and restore workspace state. + +--- + +## Recommended Implementation Order + +### Phase 1: Token Efficiency (P0) +1. Add `memory_search_index` tool (compact index results) +2. Add `memory_timeline` tool (chronological navigation) +3. Add `memory_fetch` tool (batch fetch by IDs) +4. Add token estimation to all search results + +### Phase 2: Automatic Capture (P0) +1. Create hook scripts: SessionStart, PostToolUse, Stop, SessionEnd +2. Enhance auto-capture to write structured observations +3. Auto-call `instantiate_context` on SessionStart +4. Auto-drain + summarize on SessionEnd +5. Package as installable hook set + +### Phase 3: Data Model Enrichment (P1) +1. Add structured fields to memories (title, facts, concepts, files) +2. Enhance `memory_ingest` to accept structured input +3. Add `` tag stripping +4. Use concepts for lightweight filtering + +### Phase 4: Polish (P2-P3) +1. Plugin packaging investigation +2. Transcript integration (opt-in) +3. Web viewer (if demand warrants) + +--- + +## Architecture Notes + +### Key Difference in Philosophy + +- **claude-mem**: Capture everything automatically, compress with AI, inject transparently. User/agent does nothing. +- **SerialMemory**: Rich memory graph with full lifecycle management. Agent explicitly manages memories. + +**The ideal hybrid**: SerialMemory's powerful backend + claude-mem's zero-effort capture UX. Automatic capture feeds the knowledge graph, while explicit tools allow deep management when needed. + +### Integration Points + +SerialMemory's existing architecture supports all these additions: +- **MCP tools**: Add new tools alongside existing ones +- **KnowledgeGraphService**: Extend with structured ingestion +- **PostgreSQL**: Add columns (backward compatible) +- **Hooks**: Shell scripts + MCP calls (no architecture change) +- **ToolHierarchy**: Add new categories as needed + +No fundamental architecture changes are required. All gaps can be addressed as additive features. From ca1a6845f0a81c98f7d85a31cb75a6a910aab2c0 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 21 Feb 2026 23:23:24 +0000 Subject: [PATCH 2/6] =?UTF-8?q?feat:=20Implement=20P0/P1=20gap=20analysis?= =?UTF-8?q?=20features=20=E2=80=94=20progressive=20disclosure,=20structure?= =?UTF-8?q?d=20observations,=20privacy=20tags?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P0 — Progressive Disclosure (GAP 2): - Add memory_search_index tool: compact index search (~50-80 tokens/result vs ~500+) - Add memory_timeline tool: chronological navigation around anchor point - Add memory_fetch tool: batch fetch full details by IDs - 3-layer workflow saves ~10x tokens: search_index → timeline → fetch - Token estimation on all search results P0 — Automatic Context Injection (GAP 3): - Update SessionStart hooks with progressive disclosure workflow guidance - Hook scripts updated with new tool matchers P0 — Automatic Session Capture (GAP 1): - Hook scripts updated for 15 tool matchers (was 12) - Progressive disclosure tools integrated into PreToolUse/PostToolUse hooks P1 — Structured Observations (GAP 5): - Add title, facts, concepts, files_read, files_modified to Memory model - Auto-generate title from first line when not provided - memory_ingest accepts structured fields for richer data capture - Database migration: ops/migrate_structured_observations.sql P1 — Token Budget Tracking (GAP 4): - EstimatedTokens field on MemorySearchResult - SearchMeta with IndexTokens, FullFetchTokens, TokenSavingsPercent - Compact search results show token savings percentage P2 — Privacy Tags (GAP 6): - ... content automatically stripped before storage - Added to memory_ingest tool description Files changed: - New: ProgressiveDisclosureTools.cs, migrate_structured_observations.sql - Modified: Memory model, IKnowledgeGraphStore, PostgresKnowledgeGraphStore, KnowledgeGraphService, CoreToolHandlers, CoreToolDefinitions, ToolDefinitions, ToolHierarchy, Program.cs, install-hooks.sh https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- .../Interfaces/IKnowledgeGraphStore.cs | 5 + SerialMemory.Core/Models/Memory.cs | 7 + .../Services/KnowledgeGraphService.cs | 255 +++++++++++++++++- .../KnowledgeGraphStoreDecorator.cs | 8 + .../PostgresKnowledgeGraphStore.cs | 115 +++++++- SerialMemory.Mcp/Program.cs | 26 +- SerialMemory.Mcp/Tools/CoreToolDefinitions.cs | 11 +- SerialMemory.Mcp/Tools/CoreToolHandlers.cs | 24 +- .../Tools/ProgressiveDisclosureTools.cs | 198 ++++++++++++++ SerialMemory.Mcp/Tools/ToolDefinitions.cs | 57 ++++ SerialMemory.Mcp/Tools/ToolHierarchy.cs | 7 + ops/install-hooks.sh | 17 +- ops/migrate_structured_observations.sql | 35 +++ 13 files changed, 748 insertions(+), 17 deletions(-) create mode 100644 SerialMemory.Mcp/Tools/ProgressiveDisclosureTools.cs create mode 100644 ops/migrate_structured_observations.sql diff --git a/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs b/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs index 2b53641..b496bc8 100644 --- a/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs +++ b/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs @@ -21,6 +21,11 @@ public interface IMemoryStore Task> GetMemoriesByDateRangeAsync(DateTime fromUtc, DateTime toUtc, int limit = 100, CancellationToken cancellationToken = default); Task> SearchMemoriesByEmbeddingInDateRangeAsync(float[] queryEmbedding, DateTime fromUtc, DateTime toUtc, float threshold = 0.3f, int limit = 50, CancellationToken cancellationToken = default); Task GetMemoryCountAsync(CancellationToken cancellationToken = default); + + // Progressive disclosure methods (P0 - GAP 2) + Task> GetMemoriesByIdsAsync(List ids, CancellationToken cancellationToken = default); + Task> GetMemoriesAroundAnchorAsync(Guid anchorId, int before = 5, int after = 5, string? memoryType = null, CancellationToken cancellationToken = default); + Task> GetMemoriesAroundTimestampAsync(DateTime anchor, int before = 5, int after = 5, string? memoryType = null, CancellationToken cancellationToken = default); } /// diff --git a/SerialMemory.Core/Models/Memory.cs b/SerialMemory.Core/Models/Memory.cs index 7b4032a..f227d18 100644 --- a/SerialMemory.Core/Models/Memory.cs +++ b/SerialMemory.Core/Models/Memory.cs @@ -15,6 +15,13 @@ public class Memory public Dictionary? Metadata { get; set; } public string MemoryType { get; set; } = "knowledge"; // error, decision, pattern, learning, knowledge, session_summary, auto_capture + // Structured observation fields (P1 - GAP 5: claude-mem parity) + public string? Title { get; set; } // Short title/summary (auto-generated from first line if null) + public List? Facts { get; set; } // Structured factual points + public List? Concepts { get; set; } // Categorization tags (distinct from entities) + public List? FilesRead { get; set; } // Files read during this observation + public List? FilesModified { get; set; } // Files modified during this observation + // Search result scores (populated by search queries, not stored in DB) public float Similarity { get; set; } // Cosine similarity from semantic search (0.0 to 1.0) public float Rank { get; set; } // Full-text search rank score diff --git a/SerialMemory.Core/Services/KnowledgeGraphService.cs b/SerialMemory.Core/Services/KnowledgeGraphService.cs index 70b137b..9959203 100644 --- a/SerialMemory.Core/Services/KnowledgeGraphService.cs +++ b/SerialMemory.Core/Services/KnowledgeGraphService.cs @@ -30,8 +30,18 @@ public async Task IngestMemoryAsync( string dedupMode = "warn", float dedupThreshold = 0.85f, string? memoryType = null, + string? title = null, + List? facts = null, + List? concepts = null, + List? filesRead = null, + List? filesModified = null, CancellationToken cancellationToken = default) { + // Strip tags before any processing (P2 - GAP 6) + content = StripPrivateTags(content); + if (string.IsNullOrWhiteSpace(content)) + throw new ArgumentException("Content is empty after removing private tags"); + // Generate embedding var embedding = await _embeddingService.EmbedTextAsync(content, cancellationToken); @@ -108,7 +118,7 @@ public async Task IngestMemoryAsync( ?? (metadata?.TryGetValue("memory_type", out var mtVal) == true ? mtVal?.ToString() : null) ?? "knowledge"; - // Create memory + // Create memory with structured observation fields var memory = new Memory { Content = content, @@ -116,7 +126,12 @@ public async Task IngestMemoryAsync( Source = source, ConversationSessionId = sessionId, Metadata = metadata, - MemoryType = resolvedMemoryType + MemoryType = resolvedMemoryType, + Title = title ?? ExtractTitleFromContent(content), + Facts = facts, + Concepts = concepts, + FilesRead = filesRead, + FilesModified = filesModified }; // Extract entities/relationships in parallel with entity extraction (before UoW) @@ -694,6 +709,181 @@ public async Task GetPreviousDayContextAsync( #endregion + #region Progressive Disclosure (P0 - GAP 2) + + /// + /// Search memories returning only compact index results (ID, timestamp, type, title, similarity, entity count). + /// ~50-80 tokens per result vs ~500+ for full content. Use memory_fetch to get full details. + /// + public async Task SearchMemoriesCompactAsync( + string query, + SearchMode mode = SearchMode.Hybrid, + int limit = 20, + float threshold = 0.5f, + string? memoryType = null, + CancellationToken cancellationToken = default) + { + var results = await SearchMemoriesAsync(query, mode, limit, threshold, true, memoryType, cancellationToken); + + var compactResults = results.Select(r => new CompactMemoryEntry + { + Id = r.Id, + CreatedAt = r.CreatedAt, + MemoryType = r.Source ?? "knowledge", + Title = ExtractTitleFromContent(r.Content), + Similarity = r.Similarity > 0 ? r.Similarity : r.Rank, + EntityCount = r.Entities.Count, + ContentLength = r.Content.Length, + EstimatedTokens = EstimateTokens(r.Content) + }).ToList(); + + var totalFullTokens = compactResults.Sum(r => r.EstimatedTokens); + var indexTokens = compactResults.Sum(r => EstimateTokens($"{r.Id} {r.CreatedAt:O} {r.Title} {r.Similarity:F3}")); + + return new CompactSearchResult + { + Results = compactResults, + TotalResults = compactResults.Count, + Meta = new SearchMeta + { + IndexTokens = indexTokens, + FullFetchTokens = totalFullTokens, + TokenSavingsPercent = totalFullTokens > 0 ? (int)((1.0 - (float)indexTokens / totalFullTokens) * 100) : 0 + } + }; + } + + /// + /// Batch-fetch full memory details by IDs. Step 3 of progressive disclosure. + /// + public async Task> FetchMemoriesByIdsAsync( + List ids, + bool includeEntities = true, + CancellationToken cancellationToken = default) + { + var memories = await _store.GetMemoriesByIdsAsync(ids, cancellationToken); + + var results = memories.Select(m => new MemorySearchResult + { + Id = m.Id, + Content = m.Content, + CreatedAt = m.CreatedAt, + Source = m.Source, + MemoryType = m.MemoryType, + Title = m.Title, + Facts = m.Facts, + Concepts = m.Concepts, + FilesRead = m.FilesRead, + FilesModified = m.FilesModified, + EstimatedTokens = EstimateTokens(m.Content), + Entities = [] + }).ToList(); + + if (includeEntities && results.Count > 0) + { + var memoryIds = results.Select(r => r.Id).ToList(); + var entitiesByMemory = await _store.GetEntitiesForMemoriesAsync(memoryIds, cancellationToken); + foreach (var result in results) + { + if (entitiesByMemory.TryGetValue(result.Id, out var entities)) + { + result.Entities = entities.Select(e => new EntityInfo + { + Id = e.Id, + Name = e.Name, + Type = e.EntityType + }).ToList(); + } + } + } + + return results; + } + + /// + /// Get chronological timeline around an anchor point. + /// + public async Task GetTimelineAsync( + Guid? anchorId = null, + DateTime? anchorTime = null, + int before = 5, + int after = 5, + string? memoryType = null, + CancellationToken cancellationToken = default) + { + List memories; + + if (anchorId.HasValue) + { + memories = await _store.GetMemoriesAroundAnchorAsync(anchorId.Value, before, after, memoryType, cancellationToken); + } + else if (anchorTime.HasValue) + { + memories = await _store.GetMemoriesAroundTimestampAsync(anchorTime.Value, before, after, memoryType, cancellationToken); + } + else + { + memories = await _store.GetRecentMemoriesAsync(before + after, cancellationToken); + } + + return new TimelineResult + { + AnchorId = anchorId, + AnchorTime = anchorTime ?? memories.FirstOrDefault()?.CreatedAt, + Entries = memories.Select(m => new CompactMemoryEntry + { + Id = m.Id, + CreatedAt = m.CreatedAt, + MemoryType = m.MemoryType, + Title = m.Title ?? ExtractTitleFromContent(m.Content), + Similarity = 0, + EntityCount = 0, + ContentLength = m.Content.Length, + EstimatedTokens = EstimateTokens(m.Content) + }).ToList(), + TotalEntries = memories.Count + }; + } + + #endregion + + #region Privacy Tag Support (P2 - GAP 6) + + /// + /// Strip <private>...</private> tags from content before storage. + /// + public static string StripPrivateTags(string content) + { + return System.Text.RegularExpressions.Regex.Replace( + content, + @".*?", + "", + System.Text.RegularExpressions.RegexOptions.Singleline | System.Text.RegularExpressions.RegexOptions.IgnoreCase) + .Trim(); + } + + #endregion + + #region Helpers + + private static string ExtractTitleFromContent(string content) + { + var newlinePos = content.IndexOf('\n'); + if (newlinePos > 0 && newlinePos <= 120) + return content[..newlinePos].Trim(); + return content.Length > 100 ? content[..100].Trim() + "..." : content.Trim(); + } + + /// + /// Estimate token count from character count (rough: chars / 4). + /// + public static int EstimateTokens(string text) + { + return (text.Length + 3) / 4; + } + + #endregion + #region User Persona Operations /// @@ -1020,6 +1210,17 @@ public class MemorySearchResult public float Similarity { get; set; } public float Rank { get; set; } public List Entities { get; set; } = []; + + // Structured observation fields (P1 - GAP 5) + public string? MemoryType { get; set; } + public string? Title { get; set; } + public List? Facts { get; set; } + public List? Concepts { get; set; } + public List? FilesRead { get; set; } + public List? FilesModified { get; set; } + + // Token estimation (P1 - GAP 4) + public int EstimatedTokens { get; set; } } public class MultiHopSearchResult @@ -1098,3 +1299,53 @@ public class PreviousDayContext } #endregion + +#region Progressive Disclosure Result Types (P0 - GAP 2) + +/// +/// Compact memory entry for token-efficient search results (~50-80 tokens each). +/// +public class CompactMemoryEntry +{ + public Guid Id { get; set; } + public DateTime CreatedAt { get; set; } + public string MemoryType { get; set; } = "knowledge"; + public string Title { get; set; } = ""; + public float Similarity { get; set; } + public int EntityCount { get; set; } + public int ContentLength { get; set; } + public int EstimatedTokens { get; set; } +} + +/// +/// Result from compact search index. Includes token savings metadata. +/// +public class CompactSearchResult +{ + public List Results { get; set; } = []; + public int TotalResults { get; set; } + public SearchMeta Meta { get; set; } = new(); +} + +/// +/// Token usage metadata for search results. +/// +public class SearchMeta +{ + public int IndexTokens { get; set; } + public int FullFetchTokens { get; set; } + public int TokenSavingsPercent { get; set; } +} + +/// +/// Timeline navigation result showing memories around an anchor point. +/// +public class TimelineResult +{ + public Guid? AnchorId { get; set; } + public DateTime? AnchorTime { get; set; } + public List Entries { get; set; } = []; + public int TotalEntries { get; set; } +} + +#endregion diff --git a/SerialMemory.Infrastructure/KnowledgeGraphStoreDecorator.cs b/SerialMemory.Infrastructure/KnowledgeGraphStoreDecorator.cs index dec74e7..f97a76c 100644 --- a/SerialMemory.Infrastructure/KnowledgeGraphStoreDecorator.cs +++ b/SerialMemory.Infrastructure/KnowledgeGraphStoreDecorator.cs @@ -41,6 +41,14 @@ public virtual Task> SearchMemoriesByEmbeddingInDateRangeAsync(floa public virtual Task GetMemoryCountAsync(CancellationToken cancellationToken = default) => Inner.GetMemoryCountAsync(cancellationToken); + // Progressive disclosure methods + public virtual Task> GetMemoriesByIdsAsync(List ids, CancellationToken cancellationToken = default) + => Inner.GetMemoriesByIdsAsync(ids, cancellationToken); + public virtual Task> GetMemoriesAroundAnchorAsync(Guid anchorId, int before = 5, int after = 5, string? memoryType = null, CancellationToken cancellationToken = default) + => Inner.GetMemoriesAroundAnchorAsync(anchorId, before, after, memoryType, cancellationToken); + public virtual Task> GetMemoriesAroundTimestampAsync(DateTime anchor, int before = 5, int after = 5, string? memoryType = null, CancellationToken cancellationToken = default) + => Inner.GetMemoriesAroundTimestampAsync(anchor, before, after, memoryType, cancellationToken); + // Entity operations public virtual Task CreateEntityAsync(Entity entity, CancellationToken cancellationToken = default) => Inner.CreateEntityAsync(entity, cancellationToken); diff --git a/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs b/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs index c2bd3cb..1ed496f 100644 --- a/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs +++ b/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs @@ -150,8 +150,8 @@ public async Task CreateMemoryAsync(Memory memory, CancellationToken cance const string sql = """ - INSERT INTO memories (id, tenant_id, workspace_id, content, embedding, source, conversation_session_id, metadata, memory_type) - VALUES (@Id, @TenantId, @WorkspaceId, @Content, @Embedding, @Source, @SessionId, @Metadata::jsonb, @MemoryType) + INSERT INTO memories (id, tenant_id, workspace_id, content, embedding, source, conversation_session_id, metadata, memory_type, title, facts, concepts, files_read, files_modified) + VALUES (@Id, @TenantId, @WorkspaceId, @Content, @Embedding, @Source, @SessionId, @Metadata::jsonb, @MemoryType, @Title, @Facts::jsonb, @Concepts::jsonb, @FilesRead::jsonb, @FilesModified::jsonb) """; await using var _connLease = await OpenConnectionAsync(cancellationToken); @@ -169,6 +169,11 @@ INSERT INTO memories (id, tenant_id, workspace_id, content, embedding, source, c cmd.Parameters.Add(new NpgsqlParameter("@SessionId", NpgsqlTypes.NpgsqlDbType.Uuid) { Value = (object?)memory.ConversationSessionId ?? DBNull.Value }); cmd.Parameters.Add(new NpgsqlParameter("@Metadata", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.Metadata != null ? System.Text.Json.JsonSerializer.Serialize(memory.Metadata) : DBNull.Value }); cmd.Parameters.Add(new NpgsqlParameter("@MemoryType", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.MemoryType ?? "knowledge" }); + cmd.Parameters.Add(new NpgsqlParameter("@Title", NpgsqlTypes.NpgsqlDbType.Text) { Value = (object?)memory.Title ?? DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter("@Facts", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.Facts != null ? System.Text.Json.JsonSerializer.Serialize(memory.Facts) : DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter("@Concepts", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.Concepts != null ? System.Text.Json.JsonSerializer.Serialize(memory.Concepts) : DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter("@FilesRead", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.FilesRead != null ? System.Text.Json.JsonSerializer.Serialize(memory.FilesRead) : DBNull.Value }); + cmd.Parameters.Add(new NpgsqlParameter("@FilesModified", NpgsqlTypes.NpgsqlDbType.Text) { Value = memory.FilesModified != null ? System.Text.Json.JsonSerializer.Serialize(memory.FilesModified) : DBNull.Value }); try { @@ -875,10 +880,22 @@ private static Memory MapToMemory(MemoryRow row) : null, Similarity = row.similarity, Rank = row.rank, - MemoryType = row.memory_type ?? "knowledge" + MemoryType = row.memory_type ?? "knowledge", + Title = row.title, + Facts = DeserializeJsonList(row.facts), + Concepts = DeserializeJsonList(row.concepts), + FilesRead = DeserializeJsonList(row.files_read), + FilesModified = DeserializeJsonList(row.files_modified) }; } + private static List? DeserializeJsonList(string? json) + { + if (string.IsNullOrEmpty(json)) return null; + try { return System.Text.Json.JsonSerializer.Deserialize>(json); } + catch { return null; } + } + /// /// Typed DTO for Dapper memory row mapping. Properties use snake_case to match DB columns. /// Optional columns (similarity, rank) default to 0f when not present in query results. @@ -895,6 +912,12 @@ private sealed record MemoryRow public string? memory_type { get; init; } = "knowledge"; public float similarity { get; init; } public float rank { get; init; } + // Structured observation fields + public string? title { get; init; } + public string? facts { get; init; } + public string? concepts { get; init; } + public string? files_read { get; init; } + public string? files_modified { get; init; } } private static Entity MapToEntity(dynamic row) @@ -1422,6 +1445,92 @@ LIMIT @Limit return results.Select(MapToMemory).ToList(); } + // --- Progressive Disclosure Methods (P0 - GAP 2) --- + + public async Task> GetMemoriesByIdsAsync(List ids, CancellationToken cancellationToken = default) + { + if (ids.Count == 0) return []; + + const string sql = """ + SELECT id, content, created_at, updated_at, source, + conversation_session_id, metadata::text, memory_type, + title, facts::text, concepts::text, files_read::text, files_modified::text + FROM memories + WHERE id = ANY(@Ids) + ORDER BY created_at DESC + """; + + await using var _connLease = await OpenConnectionAsync(cancellationToken); + var conn = _connLease.Connection; + + var results = await conn.QueryAsync(new CommandDefinition( + sql, + new { Ids = ids.ToArray() }, + cancellationToken: cancellationToken)); + + return results.Select(MapToMemory).ToList(); + } + + public async Task> GetMemoriesAroundAnchorAsync( + Guid anchorId, int before = 5, int after = 5, string? memoryType = null, + CancellationToken cancellationToken = default) + { + // First get the anchor memory's timestamp + const string anchorSql = "SELECT created_at FROM memories WHERE id = @Id"; + + await using var _connLease = await OpenConnectionAsync(cancellationToken); + var conn = _connLease.Connection; + + var anchorTime = await conn.QuerySingleOrDefaultAsync(new CommandDefinition( + anchorSql, new { Id = anchorId }, cancellationToken: cancellationToken)); + + if (anchorTime == null) return []; + + return await GetMemoriesAroundTimestampInternalAsync(conn, anchorTime.Value, before, after, memoryType, cancellationToken); + } + + public async Task> GetMemoriesAroundTimestampAsync( + DateTime anchor, int before = 5, int after = 5, string? memoryType = null, + CancellationToken cancellationToken = default) + { + await using var _connLease = await OpenConnectionAsync(cancellationToken); + var conn = _connLease.Connection; + return await GetMemoriesAroundTimestampInternalAsync(conn, anchor, before, after, memoryType, cancellationToken); + } + + private static async Task> GetMemoriesAroundTimestampInternalAsync( + NpgsqlConnection conn, DateTime anchor, int before, int after, string? memoryType, + CancellationToken cancellationToken) + { + var typeFilter = memoryType != null ? " AND memory_type = @MemoryType" : ""; + + var sql = $""" + (SELECT id, content, created_at, updated_at, source, + conversation_session_id, metadata::text, memory_type, + title, facts::text, concepts::text, files_read::text, files_modified::text + FROM memories + WHERE created_at <= @Anchor{typeFilter} + ORDER BY created_at DESC + LIMIT @Before) + UNION ALL + (SELECT id, content, created_at, updated_at, source, + conversation_session_id, metadata::text, memory_type, + title, facts::text, concepts::text, files_read::text, files_modified::text + FROM memories + WHERE created_at > @Anchor{typeFilter} + ORDER BY created_at ASC + LIMIT @After) + ORDER BY created_at ASC + """; + + var results = await conn.QueryAsync(new CommandDefinition( + sql, + new { Anchor = anchor, Before = before, After = after, MemoryType = memoryType }, + cancellationToken: cancellationToken)); + + return results.Select(MapToMemory).ToList(); + } + #endregion #region Workspace Operations diff --git a/SerialMemory.Mcp/Program.cs b/SerialMemory.Mcp/Program.cs index 7c09c39..21b9353 100644 --- a/SerialMemory.Mcp/Program.cs +++ b/SerialMemory.Mcp/Program.cs @@ -293,6 +293,9 @@ static void RegisterToolCategory( ["goal_complete"] = args => goalHandlers.HandleGoalComplete(args), }); +// Initialize progressive disclosure tools (P0 - GAP 2) +var progressiveDisclosureTools = new ProgressiveDisclosureTools(kgService, logger); + // Initialize auto-capture tools var autoCaptureTools = new AutoCaptureTools(kgService, logger); @@ -329,6 +332,13 @@ static void RegisterToolCategory( // Wire up session handlers now that autoCaptureTools and summarizationTools are available var sessionToolHandlers = new SessionToolHandlers(kgService, sessionState, autoCaptureTools, summarizationTools, logger); +RegisterToolCategory(gateway, "disclosure", ToolDefinitions.GetProgressiveDisclosureTools(), new() +{ + ["memory_search_index"] = args => progressiveDisclosureTools.HandleMemorySearchIndex(args), + ["memory_timeline"] = args => progressiveDisclosureTools.HandleMemoryTimeline(args), + ["memory_fetch"] = args => progressiveDisclosureTools.HandleMemoryFetch(args), +}); + RegisterToolCategory(gateway, "capture", ToolDefinitions.GetCaptureTools(), new() { ["drain_session_captures"] = args => autoCaptureTools.HandleDrainSessionCaptures(args), @@ -441,11 +451,14 @@ object HandleToolsList() var lazyToolNames = new HashSet { "memory_search", "memory_ingest", "memory_multi_hop_search", "memory_about_user", - "initialise_conversation_session", "end_conversation_session" + "initialise_conversation_session", "end_conversation_session", + "memory_search_index", "memory_timeline", "memory_fetch" }; var lazyTools = coreTools.Where(t => lazyToolNames.Contains(((dynamic)t).name)).ToArray(); + var pdTools = ToolDefinitions.GetProgressiveDisclosureTools() + .Where(t => lazyToolNames.Contains(((dynamic)t).name)).ToArray(); var gatewayTools = ToolDefinitions.GetGatewayTools(); - return new { tools = lazyTools.Concat(gatewayTools).ToArray() }; + return new { tools = lazyTools.Concat(pdTools).Concat(gatewayTools).ToArray() }; } return new @@ -610,6 +623,11 @@ async Task HandleToolsCall(JsonNode? @params) "end_conversation_session" => await sessionToolHandlers.HandleEndSession(), "memory_multi_hop_search" => await coreToolHandlers.HandleMultiHopSearch(arguments), + // Progressive disclosure tools (P0 - GAP 2) + "memory_search_index" => await progressiveDisclosureTools.HandleMemorySearchIndex(arguments), + "memory_timeline" => await progressiveDisclosureTools.HandleMemoryTimeline(arguments), + "memory_fetch" => await progressiveDisclosureTools.HandleMemoryFetch(arguments), + // Gateway meta-tools "get_tools" => gateway.HandleGetTools(arguments), "use_tool" => await HandleUseToolViaGateway(arguments), @@ -815,6 +833,10 @@ async Task HandleExecuteTool(JsonNode? arguments) "snapshot_create" => snapshotTools.HandleSnapshotCreate(arguments), "snapshot_list" => snapshotTools.HandleSnapshotList(arguments), "snapshot_load" => snapshotTools.HandleSnapshotLoad(arguments), + // Progressive disclosure tools + "memory_search_index" => progressiveDisclosureTools.HandleMemorySearchIndex(arguments), + "memory_timeline" => progressiveDisclosureTools.HandleMemoryTimeline(arguments), + "memory_fetch" => progressiveDisclosureTools.HandleMemoryFetch(arguments), // Auto-capture tools "drain_session_captures" => autoCaptureTools.HandleDrainSessionCaptures(arguments), "capture_status" => autoCaptureTools.HandleCaptureStatus(arguments), diff --git a/SerialMemory.Mcp/Tools/CoreToolDefinitions.cs b/SerialMemory.Mcp/Tools/CoreToolDefinitions.cs index 3a4dcf7..a28ef3a 100644 --- a/SerialMemory.Mcp/Tools/CoreToolDefinitions.cs +++ b/SerialMemory.Mcp/Tools/CoreToolDefinitions.cs @@ -32,19 +32,24 @@ public static object[] GetCoreTools() => new { name = "memory_ingest", - description = "Add a new memory (episode) to the knowledge graph. Automatically extracts entities, relationships, and generates embeddings.", + description = "Add a new memory (episode) to the knowledge graph. Automatically extracts entities, relationships, and generates embeddings. Content inside ... tags is automatically stripped before storage.", inputSchema = new { type = "object", properties = new { - content = new { type = "string", description = "Memory content to store" }, + content = new { type = "string", description = "Memory content to store. Content inside ... tags is automatically stripped." }, source = new { type = "string", description = "Source of the memory (e.g., 'claude-desktop', 'cursor')" }, metadata = new { type = "object", description = "Additional metadata (tags, importance, etc.)" }, extract_entities = new { type = "boolean", @default = true, description = "Whether to extract entities and relationships" }, dedup_mode = new { type = "string", @enum = new[] { "warn", "skip", "append", "off" }, @default = "warn", description = "Dedup mode: warn (create+report), skip (reject if dup), append (merge into existing), off (no check)" }, dedup_threshold = new { type = "number", @default = 0.85, description = "Similarity threshold for duplicate detection (0.0-1.0)" }, - memory_type = new { type = "string", @enum = new[] { "error", "decision", "pattern", "learning", "knowledge", "session_summary", "auto_capture" }, @default = "knowledge", description = "Memory type for categorization and filtered retrieval" } + memory_type = new { type = "string", @enum = new[] { "error", "decision", "pattern", "learning", "knowledge", "session_summary", "auto_capture" }, @default = "knowledge", description = "Memory type for categorization and filtered retrieval" }, + title = new { type = "string", description = "Short title/summary (auto-generated from first line if omitted)" }, + facts = new { type = "array", items = new { type = "string" }, description = "Structured factual points extracted from the observation" }, + concepts = new { type = "array", items = new { type = "string" }, description = "Categorization tags: how-it-works, why-it-exists, what-changed, problem-solution, gotcha, pattern, trade-off" }, + files_read = new { type = "array", items = new { type = "string" }, description = "Files read during this observation" }, + files_modified = new { type = "array", items = new { type = "string" }, description = "Files modified during this observation" } }, required = new[] { "content" } } diff --git a/SerialMemory.Mcp/Tools/CoreToolHandlers.cs b/SerialMemory.Mcp/Tools/CoreToolHandlers.cs index b6b19cb..cea2453 100644 --- a/SerialMemory.Mcp/Tools/CoreToolHandlers.cs +++ b/SerialMemory.Mcp/Tools/CoreToolHandlers.cs @@ -74,6 +74,13 @@ public async Task HandleMemoryIngest(JsonNode? arguments) metadata = JsonSerializer.Deserialize>(metadataNode.ToJsonString()); } + // Structured observation fields (P1 - GAP 5) + var title = arguments?["title"]?.GetValue()?.Trim(); + var facts = ParseStringArray(arguments?["facts"]); + var concepts = ParseStringArray(arguments?["concepts"]); + var filesRead = ParseStringArray(arguments?["files_read"]); + var filesModified = ParseStringArray(arguments?["files_modified"]); + var result = await kgService.IngestMemoryAsync( content, source, @@ -82,7 +89,12 @@ public async Task HandleMemoryIngest(JsonNode? arguments) extractEntities, dedupMode, dedupThreshold, - memoryTypeParam); + memoryTypeParam, + title, + facts, + concepts, + filesRead, + filesModified); // Set dedup metadata for usage tracking toolMetadataContext.Value = new Dictionary @@ -120,6 +132,16 @@ public async Task HandleMemoryIngest(JsonNode? arguments) return CreateTextResponse(text); } + private static List? ParseStringArray(JsonNode? node) + { + if (node is not JsonArray arr || arr.Count == 0) return null; + return arr + .Select(n => n?.GetValue()?.Trim()) + .Where(s => !string.IsNullOrEmpty(s)) + .Cast() + .ToList(); + } + public async Task HandleMultiHopSearch(JsonNode? arguments) { var query = arguments?["query"]?.GetValue()?.Trim(); diff --git a/SerialMemory.Mcp/Tools/ProgressiveDisclosureTools.cs b/SerialMemory.Mcp/Tools/ProgressiveDisclosureTools.cs new file mode 100644 index 0000000..99eaa34 --- /dev/null +++ b/SerialMemory.Mcp/Tools/ProgressiveDisclosureTools.cs @@ -0,0 +1,198 @@ +using System.Text.Json; +using System.Text.Json.Nodes; +using Microsoft.Extensions.Logging; +using SerialMemory.Core.Services; +using static SerialMemory.Mcp.McpResponseHelpers; + +namespace SerialMemory.Mcp.Tools; + +/// +/// Handlers for progressive disclosure tools (P0 - GAP 2): +/// memory_search_index, memory_timeline, memory_fetch. +/// +/// Implements a 3-layer search workflow for ~10x token savings: +/// 1. memory_search_index → compact index (~50-80 tokens/result) +/// 2. memory_timeline → chronological context around anchor +/// 3. memory_fetch → full details for selected IDs (~500+ tokens/result) +/// +internal sealed class ProgressiveDisclosureTools( + KnowledgeGraphService kgService, + ILogger logger) +{ + /// + /// Step 1: Search returning compact index results. + /// Returns IDs, timestamps, types, titles, similarity scores (~50-80 tokens each). + /// Use memory_fetch to get full content for selected results. + /// + public async Task HandleMemorySearchIndex(JsonNode? arguments) + { + var query = arguments?["query"]?.GetValue()?.Trim(); + if (string.IsNullOrEmpty(query)) + throw new ArgumentException("Query is required and cannot be empty"); + if (query.Length > 10000) + throw new ArgumentException("Query exceeds maximum length of 10000 characters"); + + var modeStr = arguments?["mode"]?.GetValue() ?? "hybrid"; + var limit = Math.Clamp(arguments?["limit"]?.GetValue() ?? 20, 1, 100); + var threshold = Math.Clamp(arguments?["threshold"]?.GetValue() ?? 0.5f, 0f, 1f); + var memoryType = arguments?["memory_type"]?.GetValue()?.Trim()?.ToLowerInvariant(); + + var mode = modeStr.ToLowerInvariant() switch + { + "semantic" => SearchMode.Semantic, + "text" => SearchMode.Text, + _ => SearchMode.Hybrid + }; + + var result = await kgService.SearchMemoriesCompactAsync(query, mode, limit, threshold, memoryType); + + if (result.TotalResults == 0) + { + return CreateTextResponse("No memories found matching the query."); + } + + var text = $"## Search Index — {result.TotalResults} results\n" + + $"**Token savings:** {result.Meta.TokenSavingsPercent}% " + + $"(index: ~{result.Meta.IndexTokens} tokens | full: ~{result.Meta.FullFetchTokens} tokens)\n\n" + + $"| # | ID | Date | Type | Title | Score | Entities | Tokens |\n" + + $"|---|-----|------|------|-------|-------|----------|--------|\n" + + string.Join("\n", result.Results.Select((r, i) => + $"| {i + 1} | `{r.Id.ToString()[..8]}` | {r.CreatedAt:yyyy-MM-dd HH:mm} | {r.MemoryType} | {Truncate(r.Title, 50)} | {r.Similarity:F2} | {r.EntityCount} | ~{r.EstimatedTokens} |")) + + "\n\nUse `memory_fetch` with selected IDs to get full content. " + + "Use `memory_timeline` with an anchor ID for chronological context."; + + return CreateTextResponse(text); + } + + /// + /// Step 2: Timeline navigation around an anchor point. + /// Shows N memories before and after a specific memory or timestamp. + /// + public async Task HandleMemoryTimeline(JsonNode? arguments) + { + var anchorIdStr = arguments?["anchor_id"]?.GetValue()?.Trim(); + var anchorTimeStr = arguments?["anchor_time"]?.GetValue()?.Trim(); + var before = Math.Clamp(arguments?["depth_before"]?.GetValue() ?? 5, 0, 50); + var after = Math.Clamp(arguments?["depth_after"]?.GetValue() ?? 5, 0, 50); + var memoryType = arguments?["memory_type"]?.GetValue()?.Trim()?.ToLowerInvariant(); + + Guid? anchorId = null; + DateTime? anchorTime = null; + + if (!string.IsNullOrEmpty(anchorIdStr) && Guid.TryParse(anchorIdStr, out var parsedId)) + { + anchorId = parsedId; + } + else if (!string.IsNullOrEmpty(anchorTimeStr) && DateTime.TryParse(anchorTimeStr, out var parsedTime)) + { + anchorTime = parsedTime.ToUniversalTime(); + } + + var result = await kgService.GetTimelineAsync(anchorId, anchorTime, before, after, memoryType); + + if (result.TotalEntries == 0) + { + return CreateTextResponse("No memories found around the anchor point."); + } + + var anchorLabel = anchorId.HasValue + ? $"Memory `{anchorId.Value.ToString()[..8]}`" + : anchorTime.HasValue + ? $"{anchorTime.Value:yyyy-MM-dd HH:mm}" + : "most recent"; + + var text = $"## Timeline — {result.TotalEntries} entries around {anchorLabel}\n\n"; + + foreach (var entry in result.Entries) + { + var isAnchor = anchorId.HasValue && entry.Id == anchorId.Value; + var marker = isAnchor ? " **← anchor**" : ""; + text += $"- `{entry.Id.ToString()[..8]}` [{entry.CreatedAt:yyyy-MM-dd HH:mm}] ({entry.MemoryType}) {Truncate(entry.Title, 60)} (~{entry.EstimatedTokens} tokens){marker}\n"; + } + + text += "\nUse `memory_fetch` with IDs to get full content."; + + return CreateTextResponse(text); + } + + /// + /// Step 3: Batch fetch full memory details by IDs. + /// Returns complete content, entities, structured fields. + /// + public async Task HandleMemoryFetch(JsonNode? arguments) + { + var idsNode = arguments?["ids"]; + if (idsNode == null || idsNode is not JsonArray idsArray) + throw new ArgumentException("ids is required (array of memory UUIDs)"); + + var ids = new List(); + foreach (var node in idsArray) + { + var idStr = node?.GetValue()?.Trim(); + if (!string.IsNullOrEmpty(idStr) && Guid.TryParse(idStr, out var id)) + ids.Add(id); + } + + if (ids.Count == 0) + throw new ArgumentException("At least one valid memory ID is required"); + + if (ids.Count > 50) + throw new ArgumentException("Maximum 50 memories can be fetched at once"); + + var includeEntities = arguments?["include_entities"]?.GetValue() ?? true; + var results = await kgService.FetchMemoriesByIdsAsync(ids, includeEntities); + + if (results.Count == 0) + { + return CreateTextResponse("No memories found for the given IDs."); + } + + var totalTokens = results.Sum(r => r.EstimatedTokens); + + var text = $"## Fetched {results.Count} memories (~{totalTokens} tokens)\n\n" + + string.Join("\n---\n\n", results.Select((r, i) => + { + var sb = new System.Text.StringBuilder(); + sb.AppendLine($"### Memory {i + 1} (ID: {r.Id})"); + sb.AppendLine($"**Created:** {r.CreatedAt:O} | **Type:** {r.MemoryType ?? "knowledge"} | **Source:** {r.Source ?? "unknown"}"); + + if (!string.IsNullOrEmpty(r.Title)) + sb.AppendLine($"**Title:** {r.Title}"); + + sb.AppendLine(); + sb.AppendLine(r.Content); + + if (r.Facts is { Count: > 0 }) + { + sb.AppendLine(); + sb.AppendLine("**Facts:**"); + foreach (var fact in r.Facts) + sb.AppendLine($"- {fact}"); + } + + if (r.Concepts is { Count: > 0 }) + sb.AppendLine($"**Concepts:** {string.Join(", ", r.Concepts)}"); + + if (r.FilesRead is { Count: > 0 }) + sb.AppendLine($"**Files Read:** {string.Join(", ", r.FilesRead)}"); + + if (r.FilesModified is { Count: > 0 }) + sb.AppendLine($"**Files Modified:** {string.Join(", ", r.FilesModified)}"); + + if (r.Entities.Count > 0) + sb.AppendLine($"**Entities:** {string.Join(", ", r.Entities.Select(e => $"{e.Name} ({e.Type})"))}"); + + sb.AppendLine($"**Estimated Tokens:** ~{r.EstimatedTokens}"); + + return sb.ToString(); + })); + + return CreateTextResponse(text); + } + + private static string Truncate(string text, int maxLen) + { + if (string.IsNullOrEmpty(text)) return ""; + return text.Length <= maxLen ? text : text[..(maxLen - 3)] + "..."; + } +} diff --git a/SerialMemory.Mcp/Tools/ToolDefinitions.cs b/SerialMemory.Mcp/Tools/ToolDefinitions.cs index 0ce70fe..994f625 100644 --- a/SerialMemory.Mcp/Tools/ToolDefinitions.cs +++ b/SerialMemory.Mcp/Tools/ToolDefinitions.cs @@ -433,6 +433,63 @@ public static object[] GetGoalTools() => } ]; + /// + /// Progressive disclosure tools (P0 - GAP 2): token-efficient 3-layer search workflow. + /// + public static object[] GetProgressiveDisclosureTools() => + [ + new + { + name = "memory_search_index", + description = "Token-efficient search: returns compact index (~50-80 tokens/result) with IDs, timestamps, titles, scores. Use memory_fetch to get full content for selected results. ~10x fewer tokens than memory_search.", + inputSchema = new + { + type = "object", + properties = new + { + query = new { type = "string", description = "Search query (natural language)" }, + mode = new { type = "string", @enum = new[] { "semantic", "text", "hybrid" }, @default = "hybrid", description = "Search mode" }, + limit = new { type = "integer", @default = 20, description = "Maximum results to return" }, + threshold = new { type = "number", @default = 0.5, description = "Minimum similarity threshold (0.0-1.0)" }, + memory_type = new { type = "string", @enum = new[] { "error", "decision", "pattern", "learning", "knowledge", "session_summary", "auto_capture" }, description = "Filter by memory type" } + }, + required = new[] { "query" } + } + }, + new + { + name = "memory_timeline", + description = "Get chronological context around a memory or timestamp. Shows N entries before and after the anchor point. Useful for understanding what happened around a specific event.", + inputSchema = new + { + type = "object", + properties = new + { + anchor_id = new { type = "string", description = "UUID of memory to center timeline on" }, + anchor_time = new { type = "string", description = "ISO timestamp to center timeline on (alternative to anchor_id)" }, + depth_before = new { type = "integer", @default = 5, description = "Number of entries before anchor" }, + depth_after = new { type = "integer", @default = 5, description = "Number of entries after anchor" }, + memory_type = new { type = "string", description = "Filter by memory type" } + } + } + }, + new + { + name = "memory_fetch", + description = "Batch fetch full memory details by IDs. Returns complete content, entities, structured fields (facts, concepts, files). Use after memory_search_index to get details for selected results.", + inputSchema = new + { + type = "object", + properties = new + { + ids = new { type = "array", items = new { type = "string" }, description = "Array of memory UUIDs to fetch (max 50)" }, + include_entities = new { type = "boolean", @default = true, description = "Include linked entities" } + }, + required = new[] { "ids" } + } + } + ]; + /// /// Shared context schema fragment for per-call context envelope. /// Added as optional property to tool schemas. diff --git a/SerialMemory.Mcp/Tools/ToolHierarchy.cs b/SerialMemory.Mcp/Tools/ToolHierarchy.cs index f139f51..dbceb0f 100644 --- a/SerialMemory.Mcp/Tools/ToolHierarchy.cs +++ b/SerialMemory.Mcp/Tools/ToolHierarchy.cs @@ -21,6 +21,7 @@ private static readonly (string Key, CategoryInfo Info)[] OrderedCategories = ("session", new("Session Management", "Create/end sessions, instantiate context")), ("admin", new("Administration", "Persona, integrations, import, crawl, statistics, model info, reembed")), ("workspace", new("Workspace & Snapshots", "Create/switch workspaces, create/load state snapshots")), + ("disclosure", new("Progressive Disclosure", "Token-efficient 3-layer search: index → timeline → fetch")), ("capture", new("Auto-Capture", "Drain session captures, check capture buffer status")), ("summarization", new("Summarization", "AI-powered session and context summarization")) ]; @@ -99,6 +100,11 @@ private static readonly (string Key, CategoryInfo Info)[] OrderedCategories = ["workspace.snapshot_list"] = "snapshot_list", ["workspace.snapshot_load"] = "snapshot_load", + // Progressive Disclosure + ["disclosure.memory_search_index"] = "memory_search_index", + ["disclosure.memory_timeline"] = "memory_timeline", + ["disclosure.memory_fetch"] = "memory_fetch", + // Auto-Capture ["capture.drain_session_captures"] = "drain_session_captures", ["capture.capture_status"] = "capture_status", @@ -127,6 +133,7 @@ public static object[] GetToolsForCategory(string category, object[]? coreTools "set_user_persona", "get_integrations", "import_from_core", "crawl_relationships", "get_graph_statistics", "get_model_info", "reembed_memories"), "workspace" => ToolDefinitions.GetWorkspaceTools(), + "disclosure" => ToolDefinitions.GetProgressiveDisclosureTools(), "capture" => ToolDefinitions.GetCaptureTools(), "summarization" => ToolDefinitions.GetSummarizationTools(), _ => [] diff --git a/ops/install-hooks.sh b/ops/install-hooks.sh index 459f01f..ba1b9ba 100755 --- a/ops/install-hooks.sh +++ b/ops/install-hooks.sh @@ -35,7 +35,7 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true "hooks": [ { "type": "command", - "command": "echo && echo 'POST-COMPACTION CONTEXT RELOAD' && echo && echo 'REQUIRED: Call these MCP tools to restore context:' && echo ' 1. mcp__serialmemory-memory__instantiate_context with project_or_subject for the current project' && echo ' 2. mcp__serialmemory-memory__memory_search for specific topics if needed' && echo && echo 'This will reload pre-compaction findings from SerialMemory.' && echo" + "command": "echo && echo 'POST-COMPACTION CONTEXT RELOAD' && echo && echo 'REQUIRED: Call these MCP tools to restore context:' && echo ' 1. mcp__serialmemory-memory__instantiate_context with project_or_subject for the current project' && echo ' 2. mcp__serialmemory-memory__memory_search_index for token-efficient search (then memory_fetch for details)' && echo && echo 'Progressive disclosure workflow: search_index → timeline → fetch (saves ~10x tokens)' && echo" } ] }, @@ -44,7 +44,7 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true "hooks": [ { "type": "command", - "command": "echo 'CLAUDE: Call mcp__serialmemory-memory__initialise_conversation_session and mcp__serialmemory-memory__instantiate_context for project context'" + "command": "echo 'CLAUDE: Call mcp__serialmemory-memory__initialise_conversation_session and mcp__serialmemory-memory__instantiate_context for project context. For token-efficient search use memory_search_index → memory_timeline → memory_fetch workflow.'" } ] } @@ -90,6 +90,9 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true { "matcher": "mcp__serialmemory-memory__initialise_conversation_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Starting session...'" }] }, { "matcher": "mcp__serialmemory-memory__end_conversation_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Ending session...'" }] }, { "matcher": "mcp__serialmemory-memory__instantiate_context", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Loading context...'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_search_index", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Compact search...'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_timeline", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Loading timeline...'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_fetch", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Fetching memories...'" }] }, { "matcher": "mcp__serialmemory-memory__get_tools_in_category", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Browsing tools...'" }] }, { "matcher": "mcp__serialmemory-memory__execute_tool", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Executing tool...'" }] }, { "matcher": "mcp__serialmemory-memory__memory_lineage", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tracing memory lineage...'" }] }, @@ -106,6 +109,9 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true { "matcher": "mcp__serialmemory-memory__initialise_conversation_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Session started'" }] }, { "matcher": "mcp__serialmemory-memory__end_conversation_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Session ended'" }] }, { "matcher": "mcp__serialmemory-memory__instantiate_context", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Context loaded'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_search_index", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Index search complete'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_timeline", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Timeline loaded'" }] }, + { "matcher": "mcp__serialmemory-memory__memory_fetch", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Memories fetched'" }] }, { "matcher": "mcp__serialmemory-memory__get_tools_in_category", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tools listed'" }] }, { "matcher": "mcp__serialmemory-memory__execute_tool", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tool executed'" }] }, { "matcher": "mcp__serialmemory-memory__memory_lineage", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Lineage traced'" }] }, @@ -201,10 +207,9 @@ echo " Stop - Response complete indicator" echo " SubagentStop - Subagent completion indicator" echo "" echo "MCP tool prefix: mcp__serialmemory-memory__" -echo "Tool coverage: 12 matchers (8 core + execute_tool + get_tools_in_category + memory_lineage + memory_trace)" -echo "Gateway tools (42+) are covered by the execute_tool matcher." -echo "New Phase 1-3 tools (drain_session_captures, capture_status, summarize_session, summarize_context)" -echo "are accessible via execute_tool gateway." +echo "Tool coverage: 15 matchers (8 core + 3 progressive disclosure + execute_tool + get_tools_in_category + memory_lineage + memory_trace)" +echo "Progressive disclosure tools: memory_search_index, memory_timeline, memory_fetch (saves ~10x tokens)" +echo "Gateway tools (45+) are covered by the execute_tool matcher." echo "" echo "Note: This only installs SerialMemory hooks." echo " Your existing non-hook settings are preserved." diff --git a/ops/migrate_structured_observations.sql b/ops/migrate_structured_observations.sql new file mode 100644 index 0000000..8fa3281 --- /dev/null +++ b/ops/migrate_structured_observations.sql @@ -0,0 +1,35 @@ +-- Migration: Add structured observation fields to memories table +-- Supports: title, facts, concepts, files_read, files_modified +-- These fields enable progressive disclosure (compact search) and structured filtering. +-- All fields are nullable for backward compatibility. + +-- Phase 1: Add new columns +ALTER TABLE memories ADD COLUMN IF NOT EXISTS title TEXT; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS facts JSONB; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS concepts JSONB; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS files_read JSONB; +ALTER TABLE memories ADD COLUMN IF NOT EXISTS files_modified JSONB; + +-- Phase 2: Index for concept-based filtering +CREATE INDEX IF NOT EXISTS idx_memories_concepts ON memories USING GIN (concepts); + +-- Phase 3: Index for file-based filtering +CREATE INDEX IF NOT EXISTS idx_memories_files_modified ON memories USING GIN (files_modified); + +-- Phase 4: Generate titles for existing memories that lack one (first line or first 100 chars) +-- This is idempotent and non-destructive +UPDATE memories +SET title = CASE + WHEN position(E'\n' in content) > 0 AND position(E'\n' in content) <= 120 + THEN left(content, position(E'\n' in content) - 1) + ELSE left(content, 100) +END +WHERE title IS NULL; + +-- Record migration +CREATE TABLE IF NOT EXISTS schema_migrations ( + name TEXT PRIMARY KEY, + applied_at TIMESTAMPTZ DEFAULT NOW() +); +INSERT INTO schema_migrations (name) VALUES ('migrate_structured_observations') +ON CONFLICT (name) DO NOTHING; From fccb1ba13fa229b4516ac6370cb2c0c31a41403f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 21 Feb 2026 23:47:04 +0000 Subject: [PATCH 3/6] feat: Auto-updating memory list with scroll and highlight animations Memories now auto-refresh every 10s (when showing recent, not search results) and new items get a cyan glow animation. Auto-scrolls to top unless the user has scrolled down manually. Shows a "+N new" badge when fresh memories arrive. https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- SerialMemory.Web/src/App.tsx | 3 +- .../src/components/Sidebar/MemoryList.tsx | 78 ++++++++++++++++++- SerialMemory.Web/src/index.css | 18 +++++ 3 files changed, 95 insertions(+), 4 deletions(-) diff --git a/SerialMemory.Web/src/App.tsx b/SerialMemory.Web/src/App.tsx index 338a5ee..740d1b8 100644 --- a/SerialMemory.Web/src/App.tsx +++ b/SerialMemory.Web/src/App.tsx @@ -23,7 +23,7 @@ function AppContent() { hops: 2, }); - // Fetch memories for sidebar + // Fetch memories for sidebar (auto-refresh every 10s when showing recent) const { data: memories, isLoading: memoriesLoading } = useQuery({ queryKey: ['memories', searchQuery, searchMode], queryFn: async () => { @@ -32,6 +32,7 @@ function AppContent() { } return fetchRecentMemories(20); }, + refetchInterval: searchQuery ? false : 10000, }); // Filter nodes based on active filters diff --git a/SerialMemory.Web/src/components/Sidebar/MemoryList.tsx b/SerialMemory.Web/src/components/Sidebar/MemoryList.tsx index edc6649..cdb3d41 100644 --- a/SerialMemory.Web/src/components/Sidebar/MemoryList.tsx +++ b/SerialMemory.Web/src/components/Sidebar/MemoryList.tsx @@ -1,3 +1,4 @@ +import { useEffect, useRef, useState } from 'react'; import { Clock } from 'lucide-react'; import type { SearchMemory } from '../../types/graph'; import { getEntityColor } from '../../types/graph'; @@ -28,6 +29,60 @@ function truncate(text: string, maxLength: number): string { } export function MemoryList({ memories, isLoading, title = 'Recent Memories' }: MemoryListProps) { + const listRef = useRef(null); + const [knownIds, setKnownIds] = useState>(new Set()); + const [newIds, setNewIds] = useState>(new Set()); + const [userScrolled, setUserScrolled] = useState(false); + const isInitialLoad = useRef(true); + + // Track which memories are new + useEffect(() => { + if (!memories.length) return; + + const currentIds = new Set(memories.map(m => m.id)); + + if (isInitialLoad.current) { + // First load — mark everything as known, nothing is "new" + setKnownIds(currentIds); + isInitialLoad.current = false; + return; + } + + const freshIds = new Set(); + for (const id of currentIds) { + if (!knownIds.has(id)) { + freshIds.add(id); + } + } + + if (freshIds.size > 0) { + setNewIds(freshIds); + setKnownIds(currentIds); + + // Auto-scroll to top if user hasn't scrolled away + if (!userScrolled && listRef.current) { + listRef.current.scrollTo({ top: 0, behavior: 'smooth' }); + } + + // Clear "new" highlight after animation completes + const timer = setTimeout(() => setNewIds(new Set()), 2000); + return () => clearTimeout(timer); + } + }, [memories]); + + // Detect manual scrolling + useEffect(() => { + const el = listRef.current; + if (!el) return; + + const handleScroll = () => { + setUserScrolled(el.scrollTop > 20); + }; + + el.addEventListener('scroll', handleScroll, { passive: true }); + return () => el.removeEventListener('scroll', handleScroll); + }, []); + if (isLoading) { return (
@@ -55,10 +110,27 @@ export function MemoryList({ memories, isLoading, title = 'Recent Memories' }: M return (
-

{title}

-
+
+

{title}

+ {newIds.size > 0 && ( + + +{newIds.size} new + + )} +
+
{memories.map(memory => ( -
+

{truncate(memory.content, 100)}

diff --git a/SerialMemory.Web/src/index.css b/SerialMemory.Web/src/index.css index 467148b..c16362b 100644 --- a/SerialMemory.Web/src/index.css +++ b/SerialMemory.Web/src/index.css @@ -166,6 +166,24 @@ body { transform: translateX(2px); } +/* New memory highlight animation */ +@keyframes memory-flash { + 0% { + background: rgba(34, 211, 238, 0.15); + border-color: rgba(34, 211, 238, 0.4); + box-shadow: 0 0 12px rgba(34, 211, 238, 0.2); + } + 100% { + background: rgba(255, 255, 255, 0.02); + border-color: rgba(59, 130, 246, 0.1); + box-shadow: none; + } +} + +.memory-item-new { + animation: memory-flash 2s ease-out forwards; +} + /* Search mode tabs */ .search-tabs { display: flex; From 8c52225cd52d30c89d6bf09b964c0eaa9961e29a Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 21 Feb 2026 23:55:00 +0000 Subject: [PATCH 4/6] feat: Expose new MCP tools as HTTP API endpoints Adds 10 new REST endpoints matching recently added MCP tools: Progressive Disclosure: - GET /api/memories/search-index (compact token-efficient search) - GET /api/memories/timeline (chronological navigation around anchor) - POST /api/memories/fetch (batch fetch full details by IDs) Goal Management: - POST /api/goals (set/update a goal) - GET /api/goals (list active goals) - POST /api/goals/{key}/complete (mark goal complete) Summarization (requires LLM): - POST /api/summarize/session (summarize session memories) - POST /api/summarize/context (summarize recent context) Auto-Capture: - GET /api/captures/status (check capture buffer status) - POST /api/captures/drain (drain JSONL entries into memories) https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- SerialMemory.Api/Program.cs | 427 ++++++++++++++++++++++++++++++++++++ 1 file changed, 427 insertions(+) diff --git a/SerialMemory.Api/Program.cs b/SerialMemory.Api/Program.cs index da2af75..ad07a01 100644 --- a/SerialMemory.Api/Program.cs +++ b/SerialMemory.Api/Program.cs @@ -1346,6 +1346,391 @@ await kgService.SetUserPersonaAttributeAsync( return Results.Ok(result); }); +// ============================================ +// PROGRESSIVE DISCLOSURE API ENDPOINTS +// ============================================ + +// Token-efficient compact search index (~50-80 tokens/result vs 500+) +app.MapGet("/api/memories/search-index", async ( + string query, + string? mode, + int? limit, + float? threshold, + string? memory_type, + KnowledgeGraphService kgService) => +{ + try + { + var searchMode = mode?.ToLower() switch + { + "semantic" => SearchMode.Semantic, + "text" => SearchMode.Text, + _ => SearchMode.Hybrid + }; + + var result = await kgService.SearchMemoriesCompactAsync( + query, + searchMode, + limit ?? 20, + threshold ?? 0.5f, + memory_type); + + return Results.Ok(result); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// Chronological navigation around an anchor point +app.MapGet("/api/memories/timeline", async ( + Guid? anchor_id, + string? anchor_time, + int? depth_before, + int? depth_after, + string? memory_type, + KnowledgeGraphService kgService) => +{ + try + { + DateTime? anchorTime = null; + if (!string.IsNullOrEmpty(anchor_time) && DateTime.TryParse(anchor_time, out var parsed)) + anchorTime = parsed.ToUniversalTime(); + + var result = await kgService.GetTimelineAsync( + anchor_id, + anchorTime, + depth_before ?? 5, + depth_after ?? 5, + memory_type); + + return Results.Ok(result); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// Batch fetch full memory details by IDs +app.MapPost("/api/memories/fetch", async (MemoryFetchRequest request, KnowledgeGraphService kgService) => +{ + try + { + if (request.Ids == null || request.Ids.Count == 0) + return Results.BadRequest(new { error = "At least one memory ID is required" }); + + if (request.Ids.Count > 50) + return Results.BadRequest(new { error = "Maximum 50 memories can be fetched at once" }); + + var results = await kgService.FetchMemoriesByIdsAsync( + request.Ids, + request.IncludeEntities ?? true); + + return Results.Ok(results); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// ============================================ +// GOAL MANAGEMENT API ENDPOINTS +// ============================================ + +// Set or update a goal +app.MapPost("/api/goals", async (GoalSetRequest request, KnowledgeGraphService kgService) => +{ + try + { + if (string.IsNullOrWhiteSpace(request.Key)) + return Results.BadRequest(new { error = "key is required" }); + if (string.IsNullOrWhiteSpace(request.Description)) + return Results.BadRequest(new { error = "description is required" }); + + var priority = Math.Clamp(request.Priority ?? 1.0f, 0.1f, 1f); + await kgService.SetGoalAsync(request.Key, request.Description, priority, request.UserId ?? "default_user"); + return Results.Ok(new { key = request.Key, priority, status = "set" }); + } + catch (Npgsql.PostgresException ex) when (ex.SqlState is "42P01" or "42703") + { + return Results.Problem(detail: "User persona schema not available. Run workspace scoping migration.", statusCode: 503); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// List active goals +app.MapGet("/api/goals", async (string? userId, KnowledgeGraphService kgService) => +{ + try + { + var goals = await kgService.GetActiveGoalsAsync(userId ?? "default_user"); + var mapped = goals.Select(g => new + { + key = g.AttributeKey, + description = g.AttributeValue, + priority = g.Confidence, + priorityLabel = g.Confidence >= 0.8f ? "HIGH" : g.Confidence >= 0.5f ? "MEDIUM" : "LOW", + updatedAt = g.UpdatedAt + }); + return Results.Ok(mapped); + } + catch (Npgsql.PostgresException ex) when (ex.SqlState is "42P01" or "42703") + { + return Results.Ok(Array.Empty()); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// Complete a goal +app.MapPost("/api/goals/{key}/complete", async (string key, string? userId, KnowledgeGraphService kgService) => +{ + try + { + await kgService.CompleteGoalAsync(key, userId ?? "default_user"); + return Results.Ok(new { key, status = "completed" }); + } + catch (Npgsql.PostgresException ex) when (ex.SqlState is "42P01" or "42703") + { + return Results.Problem(detail: "User persona schema not available. Run workspace scoping migration.", statusCode: 503); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// ============================================ +// SUMMARIZATION API ENDPOINTS +// ============================================ + +// Summarize all memories from a session using LLM +app.MapPost("/api/summarize/session", async (SessionSummarizeRequest request, KnowledgeGraphService kgService, ILlmService llmService) => +{ + try + { + if (!Guid.TryParse(request.SessionId, out var sessionId)) + return Results.BadRequest(new { error = "Valid session_id is required" }); + + var maxMemories = Math.Clamp(request.MaxMemories ?? 100, 1, 500); + var memories = await kgService.GetMemoriesBySessionAsync(sessionId, maxMemories); + + if (!(request.IncludeAutoCaptures ?? true)) + memories = memories.Where(m => m.MemoryType != "auto_capture").ToList(); + + if (memories.Count == 0) + return Results.Ok(new { summary = "", memoryCount = 0, message = $"No memories found for session {sessionId}" }); + + var content = BuildMemoryContentForSummary(memories); + var summary = await llmService.ChatAsync(content, + "Summarize the following session memories into a concise, actionable summary. Focus on key decisions, problems solved, and next steps. Format as structured markdown under 500 words.", + temperature: 0.3f, maxTokens: 1000); + + if (request.StoreSummary ?? true) + { + await kgService.IngestMemoryAsync( + content: summary, + source: "summarization", + sessionId: sessionId, + metadata: new Dictionary + { + ["memory_type"] = "session_summary", + ["summarized_memory_count"] = memories.Count, + ["llm_provider"] = llmService.ProviderName, + ["llm_model"] = llmService.ModelName + }, + extractEntities: true, + memoryType: "session_summary"); + } + + return Results.Ok(new + { + summary, + memoryCount = memories.Count, + stored = request.StoreSummary ?? true, + llmProvider = llmService.ProviderName, + llmModel = llmService.ModelName + }); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// Summarize recent context (for PreCompact or manual use) +app.MapPost("/api/summarize/context", async (ContextSummarizeRequest? request, KnowledgeGraphService kgService, ILlmService llmService) => +{ + try + { + var hoursBack = Math.Clamp(request?.HoursBack ?? 4, 1, 72); + var maxMemories = Math.Clamp(request?.MaxMemories ?? 50, 1, 200); + var fromUtc = DateTime.UtcNow.AddHours(-hoursBack); + + var memories = await kgService.GetMemoriesByDateRangeAsync(fromUtc, DateTime.UtcNow, maxMemories); + + if (memories.Count == 0) + return Results.Ok(new { summary = "", memoryCount = 0, message = $"No memories found in the last {hoursBack} hours" }); + + var content = BuildMemoryContentForSummary(memories); + var summary = await llmService.ChatAsync(content, + "Summarize the following recent memories into a concise context briefing. Focus on current work state, decisions, and blockers. Format as structured markdown under 300 words.", + temperature: 0.3f, maxTokens: 600); + + if (request?.StoreSummary ?? true) + { + await kgService.IngestMemoryAsync( + content: summary, + source: "summarization", + metadata: new Dictionary + { + ["memory_type"] = "session_summary", + ["hours_back"] = hoursBack, + ["summarized_memory_count"] = memories.Count, + ["llm_provider"] = llmService.ProviderName, + ["llm_model"] = llmService.ModelName + }, + extractEntities: true, + memoryType: "session_summary"); + } + + return Results.Ok(new + { + summary, + hoursBack, + memoryCount = memories.Count, + stored = request?.StoreSummary ?? true, + llmProvider = llmService.ProviderName, + llmModel = llmService.ModelName + }); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// ============================================ +// AUTO-CAPTURE API ENDPOINTS +// ============================================ + +var captureSessionDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".cc-serialmemory", "sessions"); + +// Check capture buffer status +app.MapGet("/api/captures/status", () => +{ + if (!Directory.Exists(captureSessionDir)) + return Results.Ok(new { files = 0, totalEntries = 0, message = "No capture directory found" }); + + var files = Directory.GetFiles(captureSessionDir, "*.jsonl") + .Where(f => !f.EndsWith(".drained")) + .OrderByDescending(File.GetLastWriteTimeUtc) + .Select(f => + { + var lineCount = File.ReadLines(f).Count(l => !string.IsNullOrWhiteSpace(l)); + return new + { + name = Path.GetFileName(f), + entries = lineCount, + lastModified = File.GetLastWriteTimeUtc(f) + }; + }) + .ToList(); + + return Results.Ok(new + { + files = files.Count, + totalEntries = files.Sum(f => f.entries), + captures = files + }); +}); + +// Drain captured JSONL session entries into memories +app.MapPost("/api/captures/drain", async ( + string? session_id, + int? max_entries, + bool? dry_run, + KnowledgeGraphService kgService) => +{ + try + { + if (!Directory.Exists(captureSessionDir)) + return Results.Ok(new { entriesProcessed = 0, memoriesCreated = 0, message = "No capture directory found" }); + + // Find session log file + string? logFile = null; + if (!string.IsNullOrEmpty(session_id)) + { + var specific = Path.Combine(captureSessionDir, $"{session_id}.jsonl"); + if (File.Exists(specific)) logFile = specific; + } + logFile ??= Directory.GetFiles(captureSessionDir, "*.jsonl") + .Where(f => !f.EndsWith(".drained")) + .OrderByDescending(File.GetLastWriteTimeUtc) + .FirstOrDefault(); + + if (logFile == null) + return Results.Ok(new { entriesProcessed = 0, memoriesCreated = 0, message = "No active capture file found" }); + + var maxCount = Math.Clamp(max_entries ?? 500, 1, 5000); + var lines = File.ReadLines(logFile) + .Where(l => !string.IsNullOrWhiteSpace(l)) + .Take(maxCount) + .ToList(); + + if (dry_run == true) + return Results.Ok(new { entriesFound = lines.Count, dryRun = true, file = Path.GetFileName(logFile) }); + + var memoriesCreated = 0; + var errors = 0; + + // Batch lines and ingest as memories + foreach (var chunk in lines.Chunk(10)) + { + try + { + var content = string.Join("\n", chunk); + await kgService.IngestMemoryAsync( + content: content, + source: "auto-capture", + metadata: new Dictionary + { + ["memory_type"] = "auto_capture", + ["entry_count"] = chunk.Length + }, + extractEntities: true, + dedupMode: "off", + memoryType: "auto_capture"); + memoriesCreated++; + } + catch { errors++; } + } + + // Rename to .drained + try { File.Move(logFile, logFile + ".drained", overwrite: true); } catch { } + + return Results.Ok(new + { + entriesProcessed = lines.Count, + memoriesCreated, + errors, + file = Path.GetFileName(logFile) + }); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + // ============================================ // USAGE & BILLING API ENDPOINTS // ============================================ @@ -6480,6 +6865,30 @@ internal record SelfHealingTriggerRequest( internal record HealingDismissRequest( string? Reason = null); +// Progressive disclosure DTOs +internal record MemoryFetchRequest( + List Ids, + bool? IncludeEntities = true); + +// Goal management DTOs +internal record GoalSetRequest( + string Key, + string Description, + float? Priority = 1.0f, + string? UserId = null); + +// Summarization DTOs +internal record SessionSummarizeRequest( + string SessionId, + bool? IncludeAutoCaptures = true, + int? MaxMemories = 100, + bool? StoreSummary = true); + +internal record ContextSummarizeRequest( + int? HoursBack = 4, + int? MaxMemories = 50, + bool? StoreSummary = true); + // Mutation state holder for tracking pause/resume state (thread-safe) internal class MutationStateHolder { @@ -6522,3 +6931,21 @@ public Task DeleteAsync(string key, CancellationToken ct = default) public Task> ListKeysAsync(CancellationToken ct = default) => Task.FromResult>(_store.Keys); } + +// Helper for summarization endpoints +static string BuildMemoryContentForSummary(List memories) +{ + var sb = new StringBuilder(); + sb.AppendLine("# Session Memories"); + sb.AppendLine(); + foreach (var memory in memories.OrderBy(m => m.CreatedAt)) + { + sb.AppendLine($"## [{memory.CreatedAt:yyyy-MM-dd HH:mm}] ({memory.MemoryType ?? "knowledge"})"); + if (!string.IsNullOrEmpty(memory.Source)) + sb.AppendLine($"Source: {memory.Source}"); + sb.AppendLine(); + sb.AppendLine(memory.Content.Length > 500 ? memory.Content[..500] + "..." : memory.Content); + sb.AppendLine(); + } + return sb.ToString(); +} From a1b817bd965baa44a05dff98ed49807db2918b9f Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 22 Feb 2026 00:05:32 +0000 Subject: [PATCH 5/6] feat: Server-side capture storage via PostgreSQL (replaces local filesystem) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captures are now stored in a `session_captures` DB table instead of local JSONL files. All operations go through HTTP — no filesystem dependency on the server. New/updated endpoints: - POST /api/captures — batch-store capture entries - POST /api/captures/entry — store a single capture entry - GET /api/captures/status — buffer status from DB - POST /api/captures/drain — drain staged captures into memories Stack: - ops/migrate_session_captures.sql — table with RLS + indexes - ICaptureStore interface + PostgresKnowledgeGraphStore impl - KnowledgeGraphService.DrainCapturesAsync with tool-type grouping - SessionCapture model https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- SerialMemory.Api/Program.cs | 168 +++++++-------- .../Interfaces/IKnowledgeGraphStore.cs | 36 +++- SerialMemory.Core/Models/SessionCapture.cs | 19 ++ .../Services/KnowledgeGraphService.cs | 197 ++++++++++++++++++ .../PostgresKnowledgeGraphStore.cs | 165 +++++++++++++++ ops/migrate_session_captures.sql | 35 ++++ 6 files changed, 532 insertions(+), 88 deletions(-) create mode 100644 SerialMemory.Core/Models/SessionCapture.cs create mode 100644 ops/migrate_session_captures.sql diff --git a/SerialMemory.Api/Program.cs b/SerialMemory.Api/Program.cs index ad07a01..20b729e 100644 --- a/SerialMemory.Api/Program.cs +++ b/SerialMemory.Api/Program.cs @@ -1619,41 +1619,78 @@ await kgService.IngestMemoryAsync( }); // ============================================ -// AUTO-CAPTURE API ENDPOINTS +// AUTO-CAPTURE API ENDPOINTS (server-side DB storage) // ============================================ -var captureSessionDir = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile), ".cc-serialmemory", "sessions"); - -// Check capture buffer status -app.MapGet("/api/captures/status", () => +// POST capture entries to the server (replaces local JSONL file writes) +app.MapPost("/api/captures", async (CaptureIngestRequest request, KnowledgeGraphService kgService) => { - if (!Directory.Exists(captureSessionDir)) - return Results.Ok(new { files = 0, totalEntries = 0, message = "No capture directory found" }); + try + { + if (request.Entries == null || request.Entries.Count == 0) + return Results.BadRequest(new { error = "At least one entry is required" }); + + if (request.Entries.Count > 1000) + return Results.BadRequest(new { error = "Maximum 1000 entries per request" }); - var files = Directory.GetFiles(captureSessionDir, "*.jsonl") - .Where(f => !f.EndsWith(".drained")) - .OrderByDescending(File.GetLastWriteTimeUtc) - .Select(f => + var captures = request.Entries.Select(e => new SessionCapture { - var lineCount = File.ReadLines(f).Count(l => !string.IsNullOrWhiteSpace(l)); - return new - { - name = Path.GetFileName(f), - entries = lineCount, - lastModified = File.GetLastWriteTimeUtc(f) - }; - }) - .ToList(); + SessionId = request.SessionId ?? e.SessionId, + Ts = e.Ts ?? DateTime.UtcNow, + Tool = e.Tool, + File = e.File, + Result = e.Result, + RawJson = e.RawJson + }).ToList(); - return Results.Ok(new + var count = await kgService.StoreCapturesBatchAsync(captures); + return Results.Ok(new { stored = count, sessionId = request.SessionId }); + } + catch (Exception ex) { - files = files.Count, - totalEntries = files.Sum(f => f.entries), - captures = files - }); + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// POST a single capture entry (convenience endpoint) +app.MapPost("/api/captures/entry", async (CaptureEntryRequest entry, KnowledgeGraphService kgService) => +{ + try + { + var capture = new SessionCapture + { + SessionId = entry.SessionId, + Ts = entry.Ts ?? DateTime.UtcNow, + Tool = entry.Tool, + File = entry.File, + Result = entry.Result, + RawJson = entry.RawJson + }; + + var id = await kgService.StoreCaptureAsync(capture); + return Results.Ok(new { id, stored = true }); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } +}); + +// Check capture buffer status (reads from DB) +app.MapGet("/api/captures/status", async (KnowledgeGraphService kgService) => +{ + try + { + var status = await kgService.GetCaptureStatusAsync(); + return Results.Ok(status); + } + catch (Exception ex) + { + return Results.Problem(detail: ex.Message, statusCode: 500); + } }); -// Drain captured JSONL session entries into memories +// Drain staged captures into memories (reads from DB, marks drained) app.MapPost("/api/captures/drain", async ( string? session_id, int? max_entries, @@ -1662,68 +1699,12 @@ await kgService.IngestMemoryAsync( { try { - if (!Directory.Exists(captureSessionDir)) - return Results.Ok(new { entriesProcessed = 0, memoriesCreated = 0, message = "No capture directory found" }); - - // Find session log file - string? logFile = null; - if (!string.IsNullOrEmpty(session_id)) - { - var specific = Path.Combine(captureSessionDir, $"{session_id}.jsonl"); - if (File.Exists(specific)) logFile = specific; - } - logFile ??= Directory.GetFiles(captureSessionDir, "*.jsonl") - .Where(f => !f.EndsWith(".drained")) - .OrderByDescending(File.GetLastWriteTimeUtc) - .FirstOrDefault(); - - if (logFile == null) - return Results.Ok(new { entriesProcessed = 0, memoriesCreated = 0, message = "No active capture file found" }); - - var maxCount = Math.Clamp(max_entries ?? 500, 1, 5000); - var lines = File.ReadLines(logFile) - .Where(l => !string.IsNullOrWhiteSpace(l)) - .Take(maxCount) - .ToList(); - - if (dry_run == true) - return Results.Ok(new { entriesFound = lines.Count, dryRun = true, file = Path.GetFileName(logFile) }); + var result = await kgService.DrainCapturesAsync( + sessionId: session_id, + maxEntries: max_entries ?? 500, + dryRun: dry_run ?? false); - var memoriesCreated = 0; - var errors = 0; - - // Batch lines and ingest as memories - foreach (var chunk in lines.Chunk(10)) - { - try - { - var content = string.Join("\n", chunk); - await kgService.IngestMemoryAsync( - content: content, - source: "auto-capture", - metadata: new Dictionary - { - ["memory_type"] = "auto_capture", - ["entry_count"] = chunk.Length - }, - extractEntities: true, - dedupMode: "off", - memoryType: "auto_capture"); - memoriesCreated++; - } - catch { errors++; } - } - - // Rename to .drained - try { File.Move(logFile, logFile + ".drained", overwrite: true); } catch { } - - return Results.Ok(new - { - entriesProcessed = lines.Count, - memoriesCreated, - errors, - file = Path.GetFileName(logFile) - }); + return Results.Ok(result); } catch (Exception ex) { @@ -6889,6 +6870,19 @@ internal record ContextSummarizeRequest( int? MaxMemories = 50, bool? StoreSummary = true); +// Capture DTOs (server-side DB storage) +internal record CaptureIngestRequest( + string? SessionId, + List Entries); + +internal record CaptureEntryRequest( + string? SessionId = null, + DateTime? Ts = null, + string? Tool = null, + string? File = null, + string? Result = null, + Dictionary? RawJson = null); + // Mutation state holder for tracking pause/resume state (thread-safe) internal class MutationStateHolder { diff --git a/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs b/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs index b496bc8..a6c758a 100644 --- a/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs +++ b/SerialMemory.Core/Interfaces/IKnowledgeGraphStore.cs @@ -101,11 +101,45 @@ public interface IWorkspaceStore Task GetSnapshotByNameAsync(string workspaceId, string snapshotName, CancellationToken ct = default); } +/// +/// Server-side session capture staging operations. +/// Captures are received over HTTP, stored in DB, then drained into memories. +/// +public interface ICaptureStore +{ + Task InsertCaptureAsync(SessionCapture capture, CancellationToken ct = default); + Task InsertCapturesBatchAsync(List captures, CancellationToken ct = default); + Task> GetUndrainedCapturesAsync(string? sessionId = null, int limit = 5000, CancellationToken ct = default); + Task MarkCapturesDrainedAsync(List captureIds, CancellationToken ct = default); + Task GetCaptureStatusAsync(CancellationToken ct = default); +} + +/// +/// Capture buffer status summary. +/// +public class CaptureStatusResult +{ + public int TotalUndrained { get; set; } + public int TotalDrained { get; set; } + public List Sessions { get; set; } = []; +} + +/// +/// Per-session capture summary. +/// +public class CaptureSessionSummary +{ + public string? SessionId { get; set; } + public int EntryCount { get; set; } + public DateTime? FirstTs { get; set; } + public DateTime? LastTs { get; set; } +} + /// /// Composite repository interface for knowledge graph operations. /// Inherits from focused sub-interfaces for backward compatibility. /// -public interface IKnowledgeGraphStore : IMemoryStore, IEntityStore, IRelationshipStore, IUserProfileStore, ISessionStore, IStatisticsStore, IWorkspaceStore +public interface IKnowledgeGraphStore : IMemoryStore, IEntityStore, IRelationshipStore, IUserProfileStore, ISessionStore, IStatisticsStore, IWorkspaceStore, ICaptureStore { Task BeginUnitOfWorkAsync(CancellationToken cancellationToken = default); } diff --git a/SerialMemory.Core/Models/SessionCapture.cs b/SerialMemory.Core/Models/SessionCapture.cs new file mode 100644 index 0000000..05a52f1 --- /dev/null +++ b/SerialMemory.Core/Models/SessionCapture.cs @@ -0,0 +1,19 @@ +namespace SerialMemory.Core.Models; + +/// +/// A staged capture entry received over HTTP. +/// Stored server-side in session_captures table, drained into memories. +/// +public class SessionCapture +{ + public Guid Id { get; set; } + public string? SessionId { get; set; } + public DateTime Ts { get; set; } = DateTime.UtcNow; + public string? Tool { get; set; } + public string? File { get; set; } + public string? Result { get; set; } + public Dictionary? RawJson { get; set; } + public bool Drained { get; set; } + public DateTime? DrainedAt { get; set; } + public DateTime CreatedAt { get; set; } = DateTime.UtcNow; +} diff --git a/SerialMemory.Core/Services/KnowledgeGraphService.cs b/SerialMemory.Core/Services/KnowledgeGraphService.cs index 9959203..b8ea0fe 100644 --- a/SerialMemory.Core/Services/KnowledgeGraphService.cs +++ b/SerialMemory.Core/Services/KnowledgeGraphService.cs @@ -982,6 +982,180 @@ public async Task CompleteGoalAsync( #endregion + #region Capture Operations + + /// + /// Store a capture entry server-side. + /// + public async Task StoreCaptureAsync( + SessionCapture capture, + CancellationToken cancellationToken = default) + { + return await _store.InsertCaptureAsync(capture, cancellationToken); + } + + /// + /// Store a batch of capture entries server-side. + /// + public async Task StoreCapturesBatchAsync( + List captures, + CancellationToken cancellationToken = default) + { + return await _store.InsertCapturesBatchAsync(captures, cancellationToken); + } + + /// + /// Get capture buffer status. + /// + public async Task GetCaptureStatusAsync( + CancellationToken cancellationToken = default) + { + return await _store.GetCaptureStatusAsync(cancellationToken); + } + + /// + /// Drain undrained captures into memories. Groups consecutive entries by tool type. + /// Returns number of memories created. + /// + public async Task DrainCapturesAsync( + string? sessionId = null, + int maxEntries = 500, + bool dryRun = false, + CancellationToken cancellationToken = default) + { + var captures = await _store.GetUndrainedCapturesAsync(sessionId, maxEntries, cancellationToken); + + if (captures.Count == 0) + return new CaptureDrainResult { Message = "No undrained captures found" }; + + if (dryRun) + return new CaptureDrainResult + { + EntriesFound = captures.Count, + DryRun = true, + Preview = GroupCapturesForPreview(captures) + }; + + var result = new CaptureDrainResult(); + var drainedIds = new List(); + + // Group consecutive entries by tool type + var batches = GroupCaptureIntoBatches(captures); + + foreach (var batch in batches) + { + try + { + var content = FormatCaptureContent(batch); + if (string.IsNullOrWhiteSpace(content)) continue; + + var metadata = new Dictionary + { + ["memory_type"] = "auto_capture", + ["trigger"] = "http_drain", + ["entry_count"] = batch.Entries.Count, + ["tool_type"] = batch.ToolType + }; + + if (batch.FirstTs.HasValue) metadata["first_ts"] = batch.FirstTs.Value.ToString("O"); + if (batch.LastTs.HasValue) metadata["last_ts"] = batch.LastTs.Value.ToString("O"); + + await IngestMemoryAsync( + content: content, + source: "auto-capture", + sessionId: batch.Entries.FirstOrDefault()?.SessionId is string sid && Guid.TryParse(sid, out var parsedSid) ? parsedSid : null, + metadata: metadata, + extractEntities: true, + dedupMode: "off", + memoryType: "auto_capture", + cancellationToken: cancellationToken); + + result.MemoriesCreated++; + drainedIds.AddRange(batch.Entries.Select(e => e.Id)); + } + catch + { + result.Errors++; + drainedIds.AddRange(batch.Entries.Select(e => e.Id)); + } + } + + // Mark all as drained + if (drainedIds.Count > 0) + await _store.MarkCapturesDrainedAsync(drainedIds, cancellationToken); + + result.EntriesProcessed = drainedIds.Count; + return result; + } + + private static List GroupCaptureIntoBatches(List captures) + { + var batches = new List(); + CaptureBatch? current = null; + + foreach (var capture in captures) + { + var toolType = NormalizeCaptureToolType(capture.Tool); + if (current == null || current.ToolType != toolType) + { + current = new CaptureBatch { ToolType = toolType }; + batches.Add(current); + } + current.Entries.Add(capture); + current.FirstTs ??= capture.Ts; + current.LastTs = capture.Ts; + } + + return batches; + } + + private static string NormalizeCaptureToolType(string? tool) + { + if (string.IsNullOrEmpty(tool)) return "unknown"; + var lower = tool.ToLowerInvariant(); + if (lower.Contains("write") || lower.Contains("edit")) return "file_edit"; + if (lower.Contains("bash")) return "bash_command"; + if (lower.Contains("read") || lower.Contains("glob") || lower.Contains("grep")) return "file_read"; + return lower; + } + + private static string FormatCaptureContent(CaptureBatch batch) + { + var files = batch.Entries + .Where(e => !string.IsNullOrEmpty(e.File)) + .Select(e => e.File!) + .Distinct() + .ToList(); + + return batch.ToolType switch + { + "file_edit" => files.Count > 0 + ? $"Files modified: {string.Join(", ", files)}" + : $"File edit operations ({batch.Entries.Count} actions)", + "bash_command" => $"Commands executed ({batch.Entries.Count} actions)" + + (files.Count > 0 ? $" — files: {string.Join(", ", files.Take(5))}" : ""), + "file_read" => files.Count > 0 + ? $"Files read: {string.Join(", ", files.Take(10))}" + : $"File read operations ({batch.Entries.Count} actions)", + _ => $"Activity: {batch.ToolType} ({batch.Entries.Count} actions)" + + (files.Count > 0 ? $" — {string.Join(", ", files.Take(5))}" : "") + }; + } + + private static List GroupCapturesForPreview(List captures) + { + var batches = GroupCaptureIntoBatches(captures); + return batches.Select(b => (object)new + { + toolType = b.ToolType, + entryCount = b.Entries.Count, + firstTs = b.FirstTs, + lastTs = b.LastTs + }).ToList(); + } + + #endregion + #region Session Operations /// @@ -1349,3 +1523,26 @@ public class TimelineResult } #endregion + +#region Capture Result Types + +public class CaptureDrainResult +{ + public int EntriesProcessed { get; set; } + public int EntriesFound { get; set; } + public int MemoriesCreated { get; set; } + public int Errors { get; set; } + public bool DryRun { get; set; } + public string? Message { get; set; } + public List? Preview { get; set; } +} + +internal class CaptureBatch +{ + public required string ToolType { get; init; } + public List Entries { get; } = []; + public DateTime? FirstTs { get; set; } + public DateTime? LastTs { get; set; } +} + +#endregion diff --git a/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs b/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs index 1ed496f..b5697c6 100644 --- a/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs +++ b/SerialMemory.Infrastructure/PostgresKnowledgeGraphStore.cs @@ -1742,4 +1742,169 @@ FROM workspace_snapshots } #endregion + + #region ICaptureStore + + public async Task InsertCaptureAsync(SessionCapture capture, CancellationToken ct = default) + { + const string sql = """ + INSERT INTO session_captures (id, session_id, ts, tool, file, result, raw_json) + VALUES (@Id, @SessionId, @Ts, @Tool, @File, @Result, @RawJson::jsonb) + RETURNING id + """; + + var id = capture.Id == Guid.Empty ? Guid.CreateVersion7() : capture.Id; + + await using var _connLease = await OpenConnectionAsync(ct); + var conn = _connLease.Connection; + + await conn.ExecuteAsync(new CommandDefinition(sql, new + { + Id = id, + capture.SessionId, + capture.Ts, + capture.Tool, + capture.File, + capture.Result, + RawJson = capture.RawJson != null + ? System.Text.Json.JsonSerializer.Serialize(capture.RawJson) + : null + }, cancellationToken: ct)); + + return id; + } + + public async Task InsertCapturesBatchAsync(List captures, CancellationToken ct = default) + { + if (captures.Count == 0) return 0; + + await using var _connLease = await OpenConnectionAsync(ct); + var conn = _connLease.Connection; + + var count = 0; + foreach (var capture in captures) + { + var id = capture.Id == Guid.Empty ? Guid.CreateVersion7() : capture.Id; + + await conn.ExecuteAsync(new CommandDefinition( + """ + INSERT INTO session_captures (id, session_id, ts, tool, file, result, raw_json) + VALUES (@Id, @SessionId, @Ts, @Tool, @File, @Result, @RawJson::jsonb) + """, + new + { + Id = id, + capture.SessionId, + capture.Ts, + capture.Tool, + capture.File, + capture.Result, + RawJson = capture.RawJson != null + ? System.Text.Json.JsonSerializer.Serialize(capture.RawJson) + : null + }, + cancellationToken: ct)); + count++; + } + + return count; + } + + public async Task> GetUndrainedCapturesAsync(string? sessionId = null, int limit = 5000, CancellationToken ct = default) + { + var sql = sessionId != null + ? """ + SELECT id, session_id, ts, tool, file, result, created_at + FROM session_captures + WHERE drained = FALSE AND session_id = @SessionId + ORDER BY ts ASC + LIMIT @Limit + """ + : """ + SELECT id, session_id, ts, tool, file, result, created_at + FROM session_captures + WHERE drained = FALSE + ORDER BY ts ASC + LIMIT @Limit + """; + + await using var _connLease = await OpenConnectionAsync(ct); + var conn = _connLease.Connection; + + var rows = await conn.QueryAsync(new CommandDefinition( + sql, new { SessionId = sessionId, Limit = limit }, cancellationToken: ct)); + + return rows.Select(r => new SessionCapture + { + Id = r.id, + SessionId = (string?)r.session_id, + Ts = r.ts, + Tool = (string?)r.tool, + File = (string?)r.file, + Result = (string?)r.result, + CreatedAt = r.created_at + }).ToList(); + } + + public async Task MarkCapturesDrainedAsync(List captureIds, CancellationToken ct = default) + { + if (captureIds.Count == 0) return 0; + + const string sql = """ + UPDATE session_captures + SET drained = TRUE, drained_at = NOW() + WHERE id = ANY(@Ids) + """; + + await using var _connLease = await OpenConnectionAsync(ct); + var conn = _connLease.Connection; + + return await conn.ExecuteAsync(new CommandDefinition( + sql, new { Ids = captureIds.ToArray() }, cancellationToken: ct)); + } + + public async Task GetCaptureStatusAsync(CancellationToken ct = default) + { + const string sql = """ + SELECT + COUNT(*) FILTER (WHERE drained = FALSE) AS undrained, + COUNT(*) FILTER (WHERE drained = TRUE) AS drained + FROM session_captures + """; + + const string sessionSql = """ + SELECT + session_id, + COUNT(*) AS entry_count, + MIN(ts) AS first_ts, + MAX(ts) AS last_ts + FROM session_captures + WHERE drained = FALSE + GROUP BY session_id + ORDER BY MAX(ts) DESC + LIMIT 20 + """; + + await using var _connLease = await OpenConnectionAsync(ct); + var conn = _connLease.Connection; + + var counts = await conn.QuerySingleAsync(new CommandDefinition(sql, cancellationToken: ct)); + + var sessions = await conn.QueryAsync(new CommandDefinition(sessionSql, cancellationToken: ct)); + + return new CaptureStatusResult + { + TotalUndrained = (int)(long)counts.undrained, + TotalDrained = (int)(long)counts.drained, + Sessions = sessions.Select(s => new CaptureSessionSummary + { + SessionId = (string?)s.session_id, + EntryCount = (int)(long)s.entry_count, + FirstTs = s.first_ts, + LastTs = s.last_ts + }).ToList() + }; + } + + #endregion } diff --git a/ops/migrate_session_captures.sql b/ops/migrate_session_captures.sql new file mode 100644 index 0000000..706699b --- /dev/null +++ b/ops/migrate_session_captures.sql @@ -0,0 +1,35 @@ +-- Migration: Server-side session capture staging table +-- Stores JSONL-style capture entries received over HTTP. +-- Captures are staged here, then drained into memories. +-- Separate from the memories table to keep capture data isolated. + +CREATE TABLE IF NOT EXISTS session_captures ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + tenant_id TEXT NOT NULL DEFAULT current_setting('app.tenant_id', true), + session_id TEXT, + ts TIMESTAMPTZ NOT NULL DEFAULT NOW(), + tool TEXT, + file TEXT, + result TEXT, + raw_json JSONB, + drained BOOLEAN NOT NULL DEFAULT FALSE, + drained_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +-- RLS policy for tenant isolation +ALTER TABLE session_captures ENABLE ROW LEVEL SECURITY; + +CREATE POLICY session_captures_tenant_policy ON session_captures + USING (tenant_id = current_setting('app.tenant_id', true)) + WITH CHECK (tenant_id = current_setting('app.tenant_id', true)); + +-- Indexes for common queries +CREATE INDEX IF NOT EXISTS idx_session_captures_drained ON session_captures (drained) WHERE drained = FALSE; +CREATE INDEX IF NOT EXISTS idx_session_captures_session ON session_captures (session_id) WHERE session_id IS NOT NULL; +CREATE INDEX IF NOT EXISTS idx_session_captures_ts ON session_captures (ts DESC); +CREATE INDEX IF NOT EXISTS idx_session_captures_tenant ON session_captures (tenant_id); + +-- Record migration +INSERT INTO schema_migrations (name) VALUES ('migrate_session_captures') +ON CONFLICT (name) DO NOTHING; From f66aba7aa8e139ae8d5194a7832e6058e8054998 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 22 Feb 2026 00:11:53 +0000 Subject: [PATCH 6/6] =?UTF-8?q?feat:=20Wire=20everything=20through=20HTTP?= =?UTF-8?q?=20=E2=80=94=20hooks,=20captures,=20MCP=20submodule?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MCP submodule updated with 16 new tool routes + schemas for progressive disclosure, goals, server-side captures, and summarization. ops/session-capture.sh — now POSTs to /api/captures/entry over HTTP instead of writing local JSONL files. Uses curl with 2s timeout. Reads SERIALMEMORY_ENDPOINT and SERIALMEMORY_API_KEY from env. ops/session-summarize.sh — now reads capture status from /api/captures/status over HTTP instead of local filesystem. ops/install-hooks.sh — added PreToolUse/PostToolUse matchers for drain_session_captures, capture_status, goal_set/list/complete, summarize_session, summarize_context (23 total matchers). https://claude.ai/code/session_01M5GUmiM8WehUBUvKWiWgcJ --- SerialMemory-MCP | 2 +- ops/install-hooks.sh | 21 +++++- ops/session-capture.sh | 46 +++++++++---- ops/session-summarize.sh | 137 +++++++++++++-------------------------- 4 files changed, 99 insertions(+), 107 deletions(-) diff --git a/SerialMemory-MCP b/SerialMemory-MCP index f1083b8..21b2256 160000 --- a/SerialMemory-MCP +++ b/SerialMemory-MCP @@ -1 +1 @@ -Subproject commit f1083b8651227bfa8a56453da03ffd0f5fe782a5 +Subproject commit 21b2256e900055601aea59150f6315b9805a0e79 diff --git a/ops/install-hooks.sh b/ops/install-hooks.sh index ba1b9ba..93da258 100755 --- a/ops/install-hooks.sh +++ b/ops/install-hooks.sh @@ -97,6 +97,13 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true { "matcher": "mcp__serialmemory-memory__execute_tool", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Executing tool...'" }] }, { "matcher": "mcp__serialmemory-memory__memory_lineage", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tracing memory lineage...'" }] }, { "matcher": "mcp__serialmemory-memory__memory_trace", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tracing memory...'" }] }, + { "matcher": "mcp__serialmemory-memory__drain_session_captures", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Draining captures...'" }] }, + { "matcher": "mcp__serialmemory-memory__capture_status", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Checking captures...'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_set", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Setting goal...'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_list", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Listing goals...'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_complete", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Completing goal...'" }] }, + { "matcher": "mcp__serialmemory-memory__summarize_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Summarizing session...'" }] }, + { "matcher": "mcp__serialmemory-memory__summarize_context", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Summarizing context...'" }] }, { "matcher": "Write", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Writing file...'" }] }, { "matcher": "Edit", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Editing file...'" }] }, { "matcher": "Bash", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Running command...'" }] } @@ -116,6 +123,13 @@ read -r -d '' HOOKS_JSON << 'HOOKS_EOF' || true { "matcher": "mcp__serialmemory-memory__execute_tool", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Tool executed'" }] }, { "matcher": "mcp__serialmemory-memory__memory_lineage", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Lineage traced'" }] }, { "matcher": "mcp__serialmemory-memory__memory_trace", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Memory traced'" }] }, + { "matcher": "mcp__serialmemory-memory__drain_session_captures", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Captures drained'" }] }, + { "matcher": "mcp__serialmemory-memory__capture_status", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Capture status loaded'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_set", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Goal set'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_list", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Goals loaded'" }] }, + { "matcher": "mcp__serialmemory-memory__goal_complete", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Goal completed'" }] }, + { "matcher": "mcp__serialmemory-memory__summarize_session", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Session summarized'" }] }, + { "matcher": "mcp__serialmemory-memory__summarize_context", "hooks": [{ "type": "command", "command": "cat > /dev/null; echo 'Context summarized'" }] }, { "matcher": "Write", "hooks": [{ "type": "command", "command": "bash ~/Projects/SerialMemoryServer/ops/session-capture.sh" }] }, { "matcher": "Edit", "hooks": [{ "type": "command", "command": "bash ~/Projects/SerialMemoryServer/ops/session-capture.sh" }] }, { "matcher": "Bash", "hooks": [{ "type": "command", "command": "bash ~/Projects/SerialMemoryServer/ops/session-capture.sh" }] } @@ -207,9 +221,10 @@ echo " Stop - Response complete indicator" echo " SubagentStop - Subagent completion indicator" echo "" echo "MCP tool prefix: mcp__serialmemory-memory__" -echo "Tool coverage: 15 matchers (8 core + 3 progressive disclosure + execute_tool + get_tools_in_category + memory_lineage + memory_trace)" -echo "Progressive disclosure tools: memory_search_index, memory_timeline, memory_fetch (saves ~10x tokens)" -echo "Gateway tools (45+) are covered by the execute_tool matcher." +echo "Tool coverage: 23 matchers (8 core + 3 disclosure + 3 goals + 2 captures + 2 summarization + 5 meta/observability)" +echo "Progressive disclosure: memory_search_index, memory_timeline, memory_fetch (saves ~10x tokens)" +echo "Captures: POST to HTTP API (no local filesystem) — drain_session_captures, capture_status" +echo "Gateway tools (50+) are covered by the execute_tool matcher." echo "" echo "Note: This only installs SerialMemory hooks." echo " Your existing non-hook settings are preserved." diff --git a/ops/session-capture.sh b/ops/session-capture.sh index 9558e4a..bd5c965 100755 --- a/ops/session-capture.sh +++ b/ops/session-capture.sh @@ -1,17 +1,16 @@ #!/usr/bin/env bash -# session-capture.sh - Capture tool activity from Claude Code PostToolUse hooks +# session-capture.sh - POST tool activity to SerialMemory API over HTTP # Called from PostToolUse hooks for Write/Edit/Bash -# Reads tool result JSON from stdin, appends JSONL entry to session log -# Designed for <50ms execution time - no MCP calls, just file appends +# Reads tool result JSON from stdin, POSTs to /api/captures/entry +# Designed for <100ms execution time — single HTTP POST, no MCP calls set -euo pipefail -# Session log directory -LOG_DIR="$HOME/.cc-serialmemory/sessions" -mkdir -p "$LOG_DIR" +# SerialMemory API endpoint (set via env or default to localhost) +API_URL="${SERIALMEMORY_ENDPOINT:-http://localhost:5000}" +API_KEY="${SERIALMEMORY_API_KEY:-}" # Session ID from Claude Code env var, fallback to date-based SESSION_ID="${CLAUDE_SESSION_ID:-$(date +%Y%m%d)}" -LOG_FILE="$LOG_DIR/${SESSION_ID}.jsonl" # Read tool result from stdin (Claude Code passes JSON) INPUT=$(cat 2>/dev/null || echo '{}') @@ -42,12 +41,35 @@ fi # Timestamp in ISO 8601 TS=$(date -u +"%Y-%m-%dT%H:%M:%SZ") -# Append JSONL entry (atomic write via echo) +# Build JSON payload if command -v jq &>/dev/null; then - jq -nc --arg ts "$TS" --arg tool "$TOOL_NAME" --arg file "$FILE_PATH" --arg result "$RESULT" \ - '{ts: $ts, tool: $tool, file: $file, result: $result}' >> "$LOG_FILE" + PAYLOAD=$(jq -nc \ + --arg sid "$SESSION_ID" \ + --arg ts "$TS" \ + --arg tool "$TOOL_NAME" \ + --arg file "$FILE_PATH" \ + --arg result "$RESULT" \ + '{session_id: $sid, ts: $ts, tool: $tool, file: $file, result: $result}') else - echo "{\"ts\":\"$TS\",\"tool\":\"$TOOL_NAME\",\"file\":\"$FILE_PATH\",\"result\":\"\"}" >> "$LOG_FILE" + PAYLOAD="{\"session_id\":\"$SESSION_ID\",\"ts\":\"$TS\",\"tool\":\"$TOOL_NAME\",\"file\":\"$FILE_PATH\",\"result\":\"\"}" fi -echo "Activity logged" +# POST to API (fire-and-forget with timeout) +AUTH_HEADER="" +if [ -n "$API_KEY" ]; then + AUTH_HEADER="-H \"Authorization: Bearer $API_KEY\"" +fi + +if command -v curl &>/dev/null; then + curl -s -m 2 -X POST "${API_URL}/api/captures/entry" \ + -H "Content-Type: application/json" \ + ${API_KEY:+-H "Authorization: Bearer $API_KEY"} \ + -d "$PAYLOAD" > /dev/null 2>&1 || true +elif command -v wget &>/dev/null; then + echo "$PAYLOAD" | wget -q -O /dev/null --timeout=2 \ + --header="Content-Type: application/json" \ + ${API_KEY:+--header="Authorization: Bearer $API_KEY"} \ + --post-data="$PAYLOAD" "${API_URL}/api/captures/entry" 2>/dev/null || true +fi + +echo "Activity captured" diff --git a/ops/session-summarize.sh b/ops/session-summarize.sh index 0b5fa00..1ea1351 100755 --- a/ops/session-summarize.sh +++ b/ops/session-summarize.sh @@ -1,125 +1,80 @@ #!/usr/bin/env bash -# session-summarize.sh - Summarize session activity for PreCompact/SessionEnd hooks -# Reads session capture JSONL log and outputs structured summary text +# session-summarize.sh - Summarize session activity via SerialMemory HTTP API +# Reads capture status from API and outputs structured summary text # Usage: bash ops/session-summarize.sh set -euo pipefail TRIGGER="${1:-session_end}" -LOG_DIR="$HOME/.cc-serialmemory/sessions" -SESSION_ID="${CLAUDE_SESSION_ID:-$(date +%Y%m%d)}" -LOG_FILE="$LOG_DIR/${SESSION_ID}.jsonl" -# Fallback message if no session log exists -if [ ! -f "$LOG_FILE" ] || [ ! -s "$LOG_FILE" ]; then - if [ "$TRIGGER" = "precompact" ]; then - echo "" - echo "CONTEXT COMPACTION IMMINENT" - echo "" - echo "Auto-capture drain and AI summarization will run at session end." - echo "For PreCompact, call mcp__serialmemory-memory__execute_tool with:" - echo " tool_path: 'summarization.summarize_context'" - echo " arguments: {\"hours_back\": 4, \"store_summary\": true}" - echo "" - echo "Also ingest any critical context not yet saved via mcp__serialmemory-memory__memory_ingest." - echo "After compaction, call mcp__serialmemory-memory__instantiate_context to restore context." - else - echo "" - echo "Session Ending — auto-drain and AI summarization will run automatically via end_conversation_session." - echo "Call mcp__serialmemory-memory__end_conversation_session to trigger auto-capture drain + summarization." - echo "Or manually ingest key insights via mcp__serialmemory-memory__memory_ingest." - fi - exit 0 -fi - -# Extract structured summary using jq or python3 -if command -v jq &>/dev/null; then - TOTAL=$(wc -l < "$LOG_FILE" | tr -d ' ') - FIRST_TS=$(head -1 "$LOG_FILE" | jq -r '.ts // "unknown"' 2>/dev/null || echo "unknown") - LAST_TS=$(tail -1 "$LOG_FILE" | jq -r '.ts // "unknown"' 2>/dev/null || echo "unknown") +# SerialMemory API endpoint +API_URL="${SERIALMEMORY_ENDPOINT:-http://localhost:5000}" +API_KEY="${SERIALMEMORY_API_KEY:-}" - FILES_CREATED=$(jq -r 'select(.tool == "Write") | .file' "$LOG_FILE" 2>/dev/null | sort -u | head -20 || echo "") - FILES_MODIFIED=$(jq -r 'select(.tool == "Edit") | .file' "$LOG_FILE" 2>/dev/null | sort -u | head -20 || echo "") - ERRORS=$(jq -r 'select(.result | test("error|fail|Error|Fail"; "i")) | "\(.tool): \(.result)"' "$LOG_FILE" 2>/dev/null | head -10 || echo "") -elif command -v python3 &>/dev/null; then - read -r TOTAL FIRST_TS LAST_TS <<< "$(python3 -c " -import json, sys -lines = open('$LOG_FILE').readlines() -entries = [json.loads(l) for l in lines if l.strip()] -print(len(entries), entries[0].get('ts','?') if entries else '?', entries[-1].get('ts','?') if entries else '?') -" 2>/dev/null || echo "0 unknown unknown")" - - FILES_CREATED=$(python3 -c " -import json -for l in open('$LOG_FILE'): - d = json.loads(l) - if d.get('tool') == 'Write' and d.get('file'): - print(d['file']) -" 2>/dev/null | sort -u | head -20 || echo "") +# Build auth header +AUTH_OPTS="" +if [ -n "$API_KEY" ]; then + AUTH_OPTS="-H \"Authorization: Bearer $API_KEY\"" +fi - FILES_MODIFIED=$(python3 -c " -import json -for l in open('$LOG_FILE'): - d = json.loads(l) - if d.get('tool') == 'Edit' and d.get('file'): - print(d['file']) -" 2>/dev/null | sort -u | head -20 || echo "") +# Fetch capture status from API +STATUS="" +if command -v curl &>/dev/null; then + STATUS=$(curl -s -m 5 "${API_URL}/api/captures/status" \ + -H "Accept: application/json" \ + ${API_KEY:+-H "Authorization: Bearer $API_KEY"} 2>/dev/null || echo "") +elif command -v wget &>/dev/null; then + STATUS=$(wget -q -O - --timeout=5 \ + --header="Accept: application/json" \ + ${API_KEY:+--header="Authorization: Bearer $API_KEY"} \ + "${API_URL}/api/captures/status" 2>/dev/null || echo "") +fi - ERRORS=$(python3 -c " -import json, re -for l in open('$LOG_FILE'): - d = json.loads(l) - r = d.get('result','') - if re.search(r'error|fail', r, re.I): - print(f\"{d.get('tool','?')}: {r[:200]}\") -" 2>/dev/null | head -10 || echo "") -else - TOTAL=$(wc -l < "$LOG_FILE" | tr -d ' ') - FIRST_TS="unknown" - LAST_TS="unknown" - FILES_CREATED="" - FILES_MODIFIED="" - ERRORS="" +# Parse capture status +TOTAL_UNDRAINED=0 +SESSION_COUNT=0 +if [ -n "$STATUS" ] && command -v jq &>/dev/null; then + TOTAL_UNDRAINED=$(echo "$STATUS" | jq -r '.totalUndrained // 0' 2>/dev/null || echo "0") + SESSION_COUNT=$(echo "$STATUS" | jq -r '.sessions | length // 0' 2>/dev/null || echo "0") fi -# Format output echo "" echo "AUTO-SESSION SUMMARY ($TRIGGER)" echo "================================" -echo "Session span: $FIRST_TS to $LAST_TS" -echo "Total tool invocations: $TOTAL" -if [ -n "$FILES_CREATED" ]; then +if [ "$TOTAL_UNDRAINED" -gt 0 ]; then + echo "Undrained captures: $TOTAL_UNDRAINED entries across $SESSION_COUNT session(s)" echo "" - echo "Files created:" - echo "$FILES_CREATED" | while read -r f; do [ -n "$f" ] && echo " - $f"; done -fi -if [ -n "$FILES_MODIFIED" ]; then - echo "" - echo "Files modified:" - echo "$FILES_MODIFIED" | while read -r f; do [ -n "$f" ] && echo " - $f"; done -fi - -if [ -n "$ERRORS" ]; then - echo "" - echo "Errors encountered:" - echo "$ERRORS" | while read -r e; do [ -n "$e" ] && echo " - $e"; done + if command -v jq &>/dev/null && [ -n "$STATUS" ]; then + echo "Sessions with pending captures:" + echo "$STATUS" | jq -r '.sessions[]? | " - \(.sessionId // "unknown"): \(.entryCount) entries (\(.firstTs // "?") to \(.lastTs // "?"))"' 2>/dev/null || true + fi +else + echo "No pending captures in buffer." fi echo "" if [ "$TRIGGER" = "precompact" ]; then + echo "CONTEXT COMPACTION IMMINENT" + echo "" + if [ "$TOTAL_UNDRAINED" -gt 0 ]; then + echo "ACTION: Call mcp__serialmemory-memory__drain_session_captures to flush $TOTAL_UNDRAINED entries." + fi echo "ACTION: Call mcp__serialmemory-memory__execute_tool with:" echo " tool_path: 'summarization.summarize_context'" echo " arguments: {\"hours_back\": 4, \"store_summary\": true}" echo "" - echo "Also ingest any critical findings not captured above via mcp__serialmemory-memory__memory_ingest." + echo "Also ingest any critical context not yet saved via mcp__serialmemory-memory__memory_ingest." echo "After compaction, call mcp__serialmemory-memory__instantiate_context to restore context." else echo "ACTION: Call mcp__serialmemory-memory__end_conversation_session to trigger:" - echo " 1. Auto-capture drain (ingests JSONL session logs as memories)" + if [ "$TOTAL_UNDRAINED" -gt 0 ]; then + echo " 1. Drain $TOTAL_UNDRAINED capture entries into memories" + fi echo " 2. AI summarization (LLM summary stored as session_summary)" echo " 3. Session close" echo "" + echo "Or call mcp__serialmemory-memory__drain_session_captures first, then end session." echo "Or manually save additional insights via mcp__serialmemory-memory__memory_ingest." fi echo ""