feat: change default file format version to 2.1

[westonpace]

No description provided.

[github-actions]

PR Review: feat: change default file format version to 2.1

The change is small and well-scoped. A few observations:

Enum ordering and Ord derivation (P1 — fragility risk)

LanceFileVersion derives Ord/PartialOrd, so variant declaration order determines comparison results. This PR reorders variants to place V2_1 before Stable (previously Stable was before V2_1). This changes the behavior of ordering-dependent methods like support_add_sub_column (self > &Self::V2_1) and support_remove_sub_column (self <= &Self::V2_1) when called on Stable.

In practice this is currently safe because Stable/Next are always resolved to concrete versions before being stored in manifests, so they should never reach those comparison methods. However, this design is fragile — nothing in the type system prevents calling support_add_sub_column() on an unresolved Stable. Consider either:

1. Having support_add_sub_column/support_remove_sub_column call .resolve() first, or
2. Adding a debug assertion that the variant isn't Stable/Next

This would prevent a subtle bug if a future caller forgets to resolve.

resolve() → Self::default() — good

Tying Stable resolution to Self::default() instead of hardcoding Self::V2_0 is a nice improvement that keeps them in sync automatically.

Test updates look correct

• Python test_metadata: relaxing exact buffer assertions to alignment-based checks makes sense since v2.1 uses different encodings with different buffer layouts.
• Rust test_insert_nested_subschemas: v2.1 no longer stores the parent struct field ID as a separate column, so removing field 0 from the expected fields/column_indices is correct.

Missing

• No migration guide or changelog note for this default change. Bumping the default file format version is user-visible — downstream consumers on older Lance versions may not be able to read files produced with the new default. Worth documenting.

[codecov]

Codecov Report

❌ Patch coverage is 98.38710% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/utils/futures.rs	98.14%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[westonpace]

Mostly minor test fixes but there was one legitimate bug that needed fixed.

[westonpace]

Miniblock means we now have two buffers, even for primitive data.

[westonpace]

Spent some time tracking this down. It is 8 now instead of 4. It seems when we read in 2.1 the arrays get allocated a buffer with 2MB of capacity even though there is only 1MB of data. Might be worth some further investigation at some point but not critical.

[westonpace]

Extra utility to add on_drop method

[westonpace]

This was a bit tricky. There was no initialization needed in 2.0 and so we never noticed this. Bascially...the I/O queue fills up and blocks, the decode thread can't get around to decoding and draining the backpressure queue because the stream has been dropped. So the scheduler hangs. This is expected. When we drop everything we should cancel all the I/O and cleanup.

However, we don't drop everything because we have a reference to the I/O scheduler here. So the fix here aborts the scheduler thread if the stream is dropped.