Structured completion pipeline with semantic metadata

[curtisman]

Summary

Replaces the ad-hoc, client-side token-boundary heuristics for command completion with a structured, metadata-driven pipeline. The backend now reports what it matched (startIndex), how completions relate to the prefix (separatorMode), whether the completion list is exhaustive (closedSet), and when to advance to the next level (commitMode). The shell and CLI consume this metadata through a formal state machine instead of string-splitting heuristics.

69 files changed, ~8,000 lines added, ~700 removed across 5 packages. 14 new test files totaling 5,637 lines of unit test coverage.

TODO: DFA/NFA grammar matcher work is not done for this architectural change. The current NFA/DFA paths in the grammar store pass through completions but do not yet produce closedSet, separatorMode, or matchedPrefixLength metadata — only the simple matchGrammarCompletion path is fully updated.

---

End-to-End Architecture

The completion feature spans 5 layers. A user keystroke flows top-to-bottom; metadata flows bottom-to-top.

┌─────────────────────────────────────────────────────────┐
│  Shell (renderer) / CLI                                 │
│  PartialCompletionSession  ←  state machine             │
│  PartialCompletion         ←  DOM adapter               │
│  SearchMenuBase            ←  trie-backed filter        │
├─────────────────────────────────────────────────────────┤
│  Dispatcher                                             │
│  getCommandCompletion()    ←  orchestration entry point │
│  resolveCompletionTarget() ←  decision engine           │
│  getCommandParameterCompletion() ← parameter pipeline   │
├─────────────────────────────────────────────────────────┤
│  Agent SDK                                              │
│  AppAgentCommandInterface.getCommandCompletion()        │
│  CompletionGroups (return type with metadata)           │
├─────────────────────────────────────────────────────────┤
│  Cache (Construction + Grammar Store)                   │
│  constructionCache.completion() → mergeCompletionResults│
│  grammarStore.completion()  → DFA > NFA > simple        │
├─────────────────────────────────────────────────────────┤
│  Grammar Matcher (actionGrammar)                        │
│  matchGrammarCompletion()  ←  longest-match + word-by-  │
│  tryPartialStringMatch()      word progression          │
└─────────────────────────────────────────────────────────┘

---

Component Responsibilities

1. Grammar Matcher (packages/actionGrammar/src/grammarMatcher.ts)

Responsibility: Given a partial input string and compiled grammar rules, compute the set of valid next-tokens using longest-match semantics.

Behavior:

• Seeds a work-list with one MatchState per top-level grammar rule
• Greedily advances each state through the prefix, tracking maxPrefixLength — the furthest point any rule consumed
• When maxPrefixLength advances, all shorter-prefix completions are discarded (only the longest match contributes)
• Categorizes outcomes into:
  • Category 1 — Exact match (entire rule matched, prefix fully consumed)
  • Category 2 — Partial match, clean boundary (prefix consumed, rule has remaining parts → offer next part)
  • Category 3 — Partial match, dirty (trailing non-separator text or pending wildcard)
• Word-by-word progression via tryPartialStringMatch(): multi-word string parts (e.g. ["play", "shuffle"]) are offered one word at a time instead of all-at-once
• Computes separatorMode per completion candidate using character-boundary analysis — determines whether space, punctuation, or nothing is needed between prefix and completion
• Sets closedSet = false when property/wildcard (entity) completions are emitted (external values can't be enumerated)
• Returns { completions, properties, matchedPrefixLength, separatorMode, closedSet }

2. Cache Layer (packages/cache/src/constructions/constructionCache.ts, grammarStore.ts)

Responsibility: Merge completion results from construction-based matching and grammar-based matching, keeping only the longest-matched-prefix candidates.

Behavior:

• Construction cache calls match(prefix, options, partial=true), tracks maxPrefixLength, eagerly discards shorter-prefix results. Sets closedSet = false when property completions exist.
• Grammar store tries three strategies in priority order: DFA → NFA → simple matchGrammarCompletion. AND-merges closedSet across grammar results (only closed if ALL are closed).
• mergeCompletionResults() combines construction + grammar results: keeps the longer prefix, AND-merges closedSet, OR-merges separatorMode (strongest wins).
• Top-level cache.completion(): if _useNFAGrammar, grammar-only; otherwise merge both sources.

3. Agent SDK (packages/agentSdk/src/command.ts, helpers/commandHelpers.ts)

Responsibility: Define the completion contract between dispatcher and agents. Agents return domain-specific completions with rich metadata.

New types:

• SeparatorMode — "space" | "spacePunctuation" | "optional" | "none" — what separator is required between prefix and completion text
• CommitMode — "explicit" | "eager" — when uniquely-satisfied completion triggers next-level re-fetch
• CompletionGroups — wrapper around CompletionGroup[] adding matchedPrefixLength, separatorMode, closedSet, commitMode
• AppAgentCommandInterface.getCommandCompletion() now returns CompletionGroups (was CompletionGroup[])

Helper: mergeSeparatorMode(a, b) — priority-based merge where the strongest separator requirement wins (space > spacePunctuation > optional > none)

Type cleanup: Internal parsing fields (tokens, lastCompletableParam, lastParamImplicitQuotes, nextArgs) moved from SDK's ParsedCommandParams to dispatcher-internal ParseParamsResult, keeping the public SDK surface clean.

4. Dispatcher (packages/dispatcher/dispatcher/src/command/completion.ts, parameters.ts)

Responsibility: Orchestrate command resolution, parameter parsing, agent invocation, and built-in completions into a single CommandCompletionResult.

Key functions and behavior:

• getCommandCompletion(input, context) — Top-level entry point. Never fails (always returns a result). Three-way branch:

  1. Uncommitted command — last command token matched but no trailing space → offer sibling subcommands (separatorMode="none")
  2. Resolved descriptor → completeDescriptor() for parameter + subcommand completions
  3. Unresolved table → offer subcommand names as trie filter candidates
  • Special: startIndex === 0 → prepend @ as Command Prefix with commitMode="eager"

• resolveCompletionTarget(params, flags, input, hasTrailingSpace) — Pure decision engine:

  • Spec case 1 — Partial parse (remainderLength > 0): offer what follows longest valid prefix
  • Spec case 3a — Full parse, last token uncommitted: either editing free-form value (3a-i, string type only) or uncommitted flag name (3a-ii)
  • Spec case 3b — Full parse, last token committed: complete next parameter/flag
  • Returns CompletionTarget with completionNames, startIndex, isPartialValue, includeFlags, booleanFlagName

• getCommandParameterCompletion() — Full parameter pipeline:

  • Parses parameters partially → ParseParamsResult
  • Resolves completion target
  • Materializes built-in completions (boolean flag values, flag names via collectFlags())
  • Invokes agent getCommandCompletion() if available → agent can override startIndex via matchedPrefixLength
  • Computes final closedSet via computeClosedSet() heuristic (agent authoritative when invoked; free-form → open; no remaining args → closed)

• CommandCompletionResult returned to shell/CLI:

  { startIndex, completions[], separatorMode, closedSet, commitMode }
  

5. Shell — PartialCompletionSession (packages/shell/src/renderer/src/partialCompletionSession.ts)

Responsibility: State machine managing the lifecycle of a completion interaction. Decides when to re-fetch from the backend vs. filter locally.

States: IDLE → PENDING → ACTIVE (and back to IDLE on reset)

Key behavior — reuseSession(input) decision tree:

Code	Trigger	Category	Action
A1	No active session (IDLE)	Invalidation	Re-fetch
A2	Input moved past anchor	Invalidation	Re-fetch
A3	Non-separator typed when separator required	Invalidation	Re-fetch
B4	Unique match + eager commit mode	Navigation	Re-fetch (next level)
B5	Committed past boundary (separator after exact match)	Navigation	Re-fetch (next level)
C6	No menu matches + open set (closedSet=false)	Discovery	Re-fetch
—	Menu has matches	—	Reuse (filter locally)
—	No matches + closed set	—	Reuse (menu hidden, no re-fetch)

Key methods:

• update(input, getPosition) — Main entry called by DOM adapter on every keystroke
• startNewSession(input, getPosition) — Issues backend request; on result, sets anchor, populates menu, computes completionPrefix, shows menu
• getCompletionPrefix(input) — Returns typed text after anchor, stripping leading separators per separatorMode
• hide() — Cancels in-flight fetch but preserves session state (reusable on re-focus)
• resetToIdle() — Full state wipe

Helper functions:

• requiresSeparator(mode) — Boolean: does this mode need a separator?
• separatorRegex(mode) — Regex for the separator character class
• stripLeadingSeparator(rawPrefix, mode) — Removes leading separator from prefix
• toMenuItems(groups) — Converts CompletionGroup[] → flat SearchMenuItem[] preserving group order

6. Shell — PartialCompletion (packages/shell/src/renderer/src/partial.ts)

Responsibility: DOM adapter bridging the text editor and the session state machine. Handles all browser-side concerns.

Behavior:

• Wraps SearchMenu → ISearchMenu and Dispatcher → ICompletionDispatcher adapters
• update(contentChanged) — checks cursor position, normalizes text, delegates to session.update()
• handleSelect(item) — DOM manipulation: computes replacement range from completion prefix, deletes old text, inserts selected text, repositions cursor, triggers fresh completion
• getSearchMenuPosition(prefix) — calculates pixel coordinates for menu positioning using DOM Range API

7. Shell — SearchMenuBase (packages/shell/src/renderer/src/searchMenuBase.ts)

Responsibility: Abstract base class providing trie-backed prefix filtering, extracted from SearchMenu to enable testability.

Behavior:

• setChoices(choices) — Populates TST (ternary search tree) from items, deduplicates by normalized text
• updatePrefix(prefix, position) — Queries trie, returns true if unique/exact match; calls onShow()/onHide() template methods
• hasExactMatch(text) — Tests exact trie membership
• normalizeMatchText(text) — NFD normalization, diacritics removal, case folding

---

Previous Architecture vs. New Architecture

Aspect	Previous (origin/main)	New (this branch)
Token-boundary logic	Client-side heuristic: shell and CLI manually split input on spaces, only sent "complete tokens" to backend. Mid-word input was locally filtered or held until next space.	Backend-driven: full input sent to dispatcher. Grammar matcher reports matchedPrefixLength → startIndex, eliminating client-side splitting.
Re-fetch decision	Simple string-prefix check (input.startsWith(current)) + noCompletion flag. Re-fetched when trie emptied regardless of reason.	Formal state machine with 6 categorized triggers (invalidation A1–A3, navigation B4–B5, discovery C6). closedSet suppresses re-fetch when trie empties on finite sets.
Separator handling	Hardcoded: @ commands used space-based boundaries; grammar completions used lastIndexOf(" ") to find boundaries. CJK/non-space scripts unsupported.	Metadata-driven: separatorMode flows from grammar through cache/agent/dispatcher to shell. Four modes support Latin (space/spacePunctuation), CJK (optional), and [spacing=none] grammars (none).
Commit behavior	Implicit: unique match always triggered re-fetch (eager). No way for agents to suppress it.	Explicit commitMode field: "eager" (re-fetch on unique match, used for @ prefix) vs. "explicit" (user must type delimiter, default for parameter completions).
Closed-set signaling	None. When trie emptied, shell always re-fetched backend hoping for new results, even for finite enum parameters.	closedSet boolean flows through entire pipeline. When true, shell skips re-fetch on trie exhaustion. Grammar entities force closedSet=false; pure keyword alternatives remain true.
Completion result format	CompletionGroup[] flat array returned by agents. No wrapper metadata. Completion fields (tokens, lastCompletableParam, etc.) mixed into SDK's ParsedCommandParams.	CompletionGroups wrapper with { groups, matchedPrefixLength, separatorMode, closedSet, commitMode }. Internal parsing fields moved to dispatcher-only ParseParamsResult.
Multi-word completions	Grammar offered entire multi-word strings at once (e.g. "shuffle mode" as one completion).	tryPartialStringMatch() offers one word at a time, guiding users through multi-word phrases progressively.
Session state	Three loose fields in PartialCompletion: current (anchor string), noCompletion (boolean), completionP (pending promise). State transitions implicit in method flow.	Dedicated PartialCompletionSession class with explicit IDLE/PENDING/ACTIVE states, typed anchor/separator/commit metadata, and documented decision tree.
Testability	SearchMenu was concrete and DOM-coupled. No unit tests for completion session logic.	SearchMenuBase extracted as abstract base. TestSearchMenu uses real trie logic. 6 test files cover session state transitions, commit modes, separator modes, result processing, error handling, and public API.
Test coverage	No completion-specific tests.	5,637 lines across 14 new test files covering grammar matching, cache merging, dispatcher orchestration, and shell session behavior.

---

Packages Modified

Package	Changes
actionGrammar	matchGrammarCompletion rewritten: longest-match prioritization, tryPartialStringMatch for word-by-word completion, closedSet/separatorMode/matchedPrefixLength output. 4 new test files (1,364 lines).
agentSdk	New types: SeparatorMode, CommitMode, CompletionGroups. Agent return type changed from CompletionGroup[] → CompletionGroups. mergeSeparatorMode() helper. Internal parsing fields removed from ParsedCommandParams.
cache	constructionCache.completion() returns closedSet/separatorMode/matchedPrefixLength. mergeCompletionResults() implements AND-merge for closedSet, longest-prefix wins. 2 new test files (1,144 lines).
dispatcher	getCommandCompletion() returns full CommandCompletionResult with closedSet/separatorMode/commitMode. resolveCompletionTarget() extracted as pure decision engine. computeClosedSet() heuristic. parseParams extended with ParseParamsResult. 2 new test files (1,660 lines).
shell	PartialCompletionSession extracted (440 lines). PartialCompletion simplified from 416 → 280 lines. SearchMenuBase extracted (91 lines). 6 new test files (1,469 lines). Jest config added.
cli	Token-boundary heuristics removed. Full input sent to backend. separatorMode consumed to insert separator in readline display.
agents	Callsites updated for CompletionGroups return type (browser, taskflow).