feat: add serialize/deserialize for IVF PQ partition cache entries

[wjones127]

Summary

• Add zero-copy serialization/deserialization for PartitionEntry<S, ProductQuantizer>, the type cached in LanceCache for IVF_PQ V3 indices.
• Uses Arrow IPC FileDecoder for zero-copy deserialization: Arrow arrays reference slices of the original Bytes buffer directly.
• Generic over S: IvfSubIndex, so works for both FlatIndex+PQ and HNSW+PQ variants.

Test plan

• Roundtrip test for FlatIndex+PQ with 100 rows
• Distance type preservation across all relevant variants
• Empty partition roundtrip
• Truncated data returns error

🤖 Generated with Claude Code

[github-actions]

PR Review: feat: add serialize/deserialize for IVF PQ partition cache entries

Clean implementation overall — the zero-copy IPC approach is well-suited for cache serde. A few issues to flag:

P1: Integer overflow in deserialization offset arithmetic

In deserialize(), the section boundary calculations use unchecked addition:

let sub_index_end = sub_index_start + header.sub_index_len as usize;
let codebook_end = codebook_start + header.codebook_len as usize;
let storage_end = storage_start + header.storage_len as usize;

If the header contains corrupted or adversarial values (e.g., lengths close to u64::MAX), these additions can wrap around, causing storage_end to be a small value that passes the data.len() < storage_end check. The subsequent Buffer::slice_with_length calls would then read incorrect regions or panic.

Use checked_add and return an error on overflow, same as the defensive pattern already used for trailer_start / footer_start in read_ipc_zero_copy.

P1: Missing Hamming in distance type roundtrip test

test_roundtrip_preserves_distance_type covers L2, Cosine, and Dot but omits Hamming, which is handled in the distance_type_to_u8/u8_to_distance_type mapping. Either add it to the test or document why it's excluded (e.g., PQ storage doesn't support Hamming).

Minor

• The module is pub mod partition_serde — if this is only intended for internal caching, consider pub(crate) to avoid leaking it as public API.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 82.48175% with 48 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/ivf/partition_serde.rs	82.48%	23 Missing and 25 partials ⚠️

📢 Thoughts on this report? Let us know!