Add embedding pipeline and rebuildable secondary indexes

[voidrot]

Summary

Add the pipeline that derives embeddings and secondary search indexes from the live markdown corpus.

Why this work is needed

Hybrid retrieval requires derived indexes, but those indexes must remain reconstructable from markdown and frontmatter rather than becoming a new source of truth.

Scope

• Derive embeddings from live markdown.
• Build and rebuild secondary indexes from the live corpus.
• Define incremental and full rebuild behavior.
• Add fallback expectations when indexes are unavailable.

Out of scope

• Replacing markdown storage.
• UI for search management.
• Final MCP integration work.

Acceptance criteria

• Embeddings and indexes can be rebuilt from live markdown.
• Index build behavior is documented and testable.
• Fallback behavior to the deterministic retrieval baseline is explicit.

Dependencies

• Depends on Epic: Hybrid retrieval implementation over reconstructable indexes #60.
• Depends on Define machine auth model for agents and MCP clients #69.