feat!: add progress monitoring via callbacks for distributed merge

[vivek-bharathan]

No description provided.

[github-actions]

PR Review

Bug (P0): Inconsistent Mutex type in inverted index test

In rust/lance-index/src/scalar/inverted/builder.rs, the RecordingProgress test helper uses tokio::sync::Mutex (with .lock().await), while the equivalent helpers in btree.rs and index_merger.rs use std::sync::Mutex (with .lock().unwrap()). This will compile (since tokio::sync::Mutex<T: Send> is Sync), but it's needlessly inconsistent and introduces an unnecessary async lock for a simple test helper. Should use std::sync::Mutex for consistency.

Magic number (P1): * 6 in inverted index rename total

In rust/lance-index/src/scalar/inverted/builder.rs, the total renames is calculated as:

let total_renames = id_mapping
    .iter()
    .filter(|(old_id, new_id)| old_id != new_id)
    .count() as u64
    * 6;

The 6 comes from 3 file suffixes × 2 rename phases, but this is fragile and not documented. If a file type is added/removed, this silently breaks the progress total. Consider deriving it from the actual constants, e.g.:

let num_suffixes = [TOKENS_FILE, INVERT_LIST_FILE, DOCS_FILE].len() as u64;
let num_phases = 2u64; // temp + final
let total_renames = changed_count * num_suffixes * num_phases;

Minor note: RecordingProgress duplicated 3 times

The same RecordingProgress test helper is copy-pasted in btree.rs, inverted/builder.rs, and index_merger.rs (plus ivf/v2.rs). Consider extracting it into lance-testing or a shared test utility module in lance-index to avoid divergence (as already evidenced by the Mutex inconsistency above).

---

Overall: the approach of threading Arc<dyn IndexBuildProgress> through the merge paths is clean and well-tested. The test coverage is good with stage ordering assertions. The issues above are the main things to address before merge.

[codecov]

Codecov Report

❌ Patch coverage is 90.86758% with 40 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/builder.rs	89.65%	0 Missing and 12 partials ⚠️
rust/lance-index/src/scalar/btree.rs	84.93%	2 Missing and 9 partials ⚠️
...lance-index/src/vector/distributed/index_merger.rs	91.26%	1 Missing and 10 partials ⚠️
rust/lance/src/index/vector/ivf.rs	57.14%	0 Missing and 3 partials ⚠️
rust/lance/src/dataset.rs	50.00%	2 Missing ⚠️
rust/lance/src/index/vector/ivf/v2.rs	99.02%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!