perf: optimize BM25 scoring strategy

[dgageot]

Summary

Optimizes the BM25 keyword-based retrieval strategy in pkg/rag/strategy/bm25.go.

Changes

• Move replacer and stopwords into struct fields — built once in the constructor instead of allocating a new strings.NewReplacer (16 pairs) and a new map[string]bool (20 entries) on every tokenize() call.
• Pre-tokenize documents and pre-compute document frequencies — the old code tokenized each document twice per scoring call and scanned the entire corpus with strings.Contains per query term per document (O(T×N²×L)). Now Query() tokenizes all documents once and pre-computes DF in a single pass (O(N×L + T×N)).
• Replace O(n²) selection sort with slices.SortFunc — idiomatic O(n log n).
• Guard against division by zero — docLength / avgDocLength produced +Inf/NaN when avgDocLength was 0. Now defaults to a length ratio of 1.0.
• Fix DF consistency — old code used substring matching (strings.Contains) for document frequency but exact token matching for term frequency, inflating DF and suppressing IDF scores. Now both use exact token matching, which is the correct BM25 formulation.

[docker-agent]

Review Summary

Assessment: 🟢 APPROVE

This is a well-executed performance optimization with multiple improvements to the BM25 scoring implementation:

✅ Memory optimization: Moving replacer and stopwords to struct fields eliminates repeated allocations on every tokenization call

✅ Algorithmic improvement: Pre-tokenizing documents and pre-computing document frequencies reduces complexity from O(T×N²×L) to O(N×L + T×N)

✅ Sorting efficiency: Replacing O(n²) selection sort with slices.SortFunc provides O(n log n) performance

✅ Safety enhancement: Added guard against division by zero when avgDocLength is 0 (defaults to length ratio of 1.0)

✅ Correctness fix: Using exact token matching for both TF and DF calculations instead of mixing exact matching and substring matching

The code includes appropriate guards for edge cases:

• Query validation ensures queryTerms is non-empty before scoring
• Division by avgDocLength is protected with a conditional check
• Term frequency checks prevent unnecessary IDF calculations

No bugs found in the changed code. The optimizations are sound and improve both performance and correctness.