feat: pluggable index cache via CacheBackend trait

[wjones127]

Previously the Session's index cache was hardcoded to Moka. This adds a CacheBackend trait so users can provide their own cache implementation. There is a default MokaCacheBackend that works the same as the existing cache.

flowchart LR
    LanceCache --> backend["dyn CacheBackend"]
    backend --> MokaCacheBackend
    backend --> CustomCacheBackend

Loading

The cache key construction is handled at the LanceCache layer, so CacheBackend implementations just receive opaque bytes for keys. They can optionally use parse_cache_key to get a unique u64 type id. This might be used by caches to figure out how they can downcast and serialize / deserialize the entry.

flowchart LR
    CacheKey --> key["&[u8]"]
    key --"parse_cache_key"--> typeid

Loading

[github-actions]

PR Review: feat: pluggable index cache via CacheBackend trait

P0: TOCTOU race in get_or_insert — dedup is broken

The check-then-register pattern in get_or_insert has a race between the two separately-locked critical sections:

// Block 1: check (drops lock at end of block)
{
    let map = self.in_flight.lock().await;
    if let Some(rx) = map.get(&cache_key) { ... }
}
// <-- gap: another task can also see an empty map here

// Block 2: register as leader (re-acquires lock)
let (tx, rx) = tokio::sync::watch::channel(None);
{
    let mut map = self.in_flight.lock().await;
    map.insert(cache_key.clone(), rx);
}

Between dropping the lock in block 1 and re-acquiring it in block 2, N tasks can all see an empty in-flight map and all decide they are the "leader." The last one to insert overwrites the earlier receivers, so:

1. The loader runs N times (dedup completely defeated).
2. Tasks that cloned an earlier receiver get RecvError when that sender drops without the value propagating through the map.

Fix: Merge the check and register into a single critical section:

let mut map = self.in_flight.lock().await;
if let Some(rx) = map.get(&cache_key) {
    let mut rx = rx.clone();
    drop(map);
    // ... wait for leader ...
} else {
    let (tx, rx) = tokio::sync::watch::channel(None);
    map.insert(cache_key.clone(), rx);
    drop(map);
    // ... run loader ...
}

The test_get_or_insert_dedup test passes by luck because tokio::task::yield_now() with a broadcast barrier doesn't reliably interleave into this gap. A test with a longer sleep or explicit synchronization at the gap point would expose this.

P1: type_tag relies on unspecified pointer identity

fn type_tag<T: 'static>() -> [u8; 8] {
    (std::any::type_name::<T>().as_ptr() as u64).to_le_bytes()
}

The Rust spec does not guarantee that type_name::<T>().as_ptr() returns the same address across calls, nor that distinct types produce distinct addresses (string deduplication/interning is compiler-implementation-defined). The old code used TypeId as a key component, which has correct identity semantics.

If two different types happen to collide (same pointer or same string literal address), the cache will silently return a wrong type and the downcast::<T>().unwrap() will panic at runtime.

Consider using TypeId bytes instead. TypeId implements Hash and Eq, so you could hash it to get a stable discriminator, or use transmute to extract its bytes (it's currently 128 bits).

P1: invalidate_prefix is fire-and-forget

pub fn invalidate_prefix(&self, prefix: &str) {
    let prefix_bytes = self.make_prefix(prefix);
    let cache = self.cache.clone();
    tokio::spawn(async move {
        cache.invalidate_prefix(&prefix_bytes).await;
    });
}

This means invalidation is not guaranteed to complete before subsequent cache reads. A caller that invalidates then immediately reads could get stale data. The old code was synchronous. If this must remain sync (non-async signature), at minimum document this caveat prominently, or consider changing the signature to async fn.

Minor notes

• DeepSizeOf for LanceCache now returns 0, which is a silent behavioral regression for any code that relies on deep_size_of() for memory accounting. Consider at least delegating to approx_size_bytes().
• WeakLanceCache::get_or_insert lost all dedup behavior — concurrent loads for the same key will all run the loader independently. The old code used moka's optionally_get_with which handled this.

[codecov]

Codecov Report

❌ Patch coverage is 69.12162% with 457 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf/v2.rs	8.37%	164 Missing ⚠️
rust/lance/src/index/vector/ivf/partition_serde.rs	84.53%	58 Missing and 58 partials ⚠️
rust/lance-index/src/vector.rs	0.00%	78 Missing ⚠️
rust/lance-core/src/cache.rs	84.97%	47 Missing and 5 partials ⚠️
rust/lance/src/session.rs	30.43%	16 Missing ⚠️
rust/lance-index/src/vector/storage.rs	0.00%	15 Missing ⚠️
rust/lance/src/dataset/builder.rs	46.66%	8 Missing ⚠️
rust/lance-index/src/scalar/inverted/index.rs	37.50%	4 Missing and 1 partial ⚠️
rust/lance/src/session/index_caches.rs	66.66%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!