feat: support atomic multi-table transactions via namespace manifest by XuQianJin-Stars · Pull Request #6173 · lance-format/lance

XuQianJin-Stars · 2026-03-12T07:52:31Z

Summary

Add a new table_version_storage_enabled flag to DirectoryNamespaceBuilder that stores table versions in the __manifest table (as table_version object type) instead of relying solely on Lance's native version management. This enables atomic multi-table version creation via BatchCreateTableVersions, multi-table deletion via BatchDeleteTableVersions, and a unified atomic BatchCommitTables operation that supports mixed operations (declare, create version, delete versions, deregister) across multiple tables in a single transaction.

Key Changes

manifest.rs

Add TableVersion variant to ObjectType enum
Make str_object_id and insert_into_manifest_with_metadata public
Add batch_insert_into_manifest(): atomically inserts multiple entries using a single MergeInsert with WhenMatched::Fail semantics, so if any version already exists the entire batch is rejected
Add query_table_versions(): lists all versions for a given table from the manifest, returning version number and full metadata JSON
Add query_table_version(): queries a single version entry by table ID and version number, returning its metadata JSON
Add delete_table_versions(): removes version entries by table ID and version ranges from the manifest
Add batch_delete_table_versions_by_object_ids(): atomically deletes version entries across multiple tables in a single DeleteBuilder operation

dir.rs

Add table_version_storage_enabled field with builder method and validation (requires manifest_enabled=true)
CreateTableVersion: when enabled, also records the version entry in __manifest via insert_into_manifest_with_metadata
BatchCreateTableVersions: two-phase commit — first copies staging manifests to final locations, then atomically inserts all version entries into __manifest in a single batch operation
ListTableVersions: when enabled, queries versions from __manifest and reconstructs TableVersion from stored metadata JSON
DescribeTableVersion: when enabled and version is specified, queries directly from __manifest instead of opening the dataset
BatchDeleteTableVersions: supports two modes:
- Single-table (legacy): uses request.id + request.ranges for one table
- Multi-table (transactional): uses request.entries to atomically delete versions across multiple tables (metadata deletion is atomic; physical file cleanup is best-effort)
BatchCommitTables: new unified operation that atomically commits mixed table operations (declare_table, create_table_version, delete_table_versions, deregister_table) across multiple tables using discriminated union types from the generated lance-namespace API spec
Add comprehensive tests: batch create atomic, conflict detection, manifest requirement validation, single version recording, and more

namespace.rs

Add batch_create_table_versions trait method with default not-supported implementation
Add batch_commit_tables trait method with default not-supported implementation
Update batch_delete_table_versions doc to document both single-table (legacy) and multi-table (transactional) modes

rest.rs / rest_adapter.rs

Add REST client/server endpoints:
- POST /v1/table/version/batch_create — batch_create_table_versions
- POST /v1/table/:id/version/batch_delete — batch_delete_table_versions
- POST /v1/table/batch_commit — batch_commit_tables (unified atomic multi-table commit)

Depends On

lance-namespace PR: adds BatchCommitTables API spec and generated client code with discriminated union types (CommitTableOperation, CommitTableResult)

Testing

122 unit tests pass (including new tests for batch create, conflict detection, multi-table delete, and manifest validation)
All doc-tests pass

github-actions · 2026-03-12T07:54:05Z

PR Review: feat: support atomic multi-table transactions via namespace manifest

P0: Broken dependency path — will not build for anyone else

-lance-namespace-reqwest-client = "0.5.2"
+lance-namespace-reqwest-client = { path = "../lance-namespace/rust/lance-namespace-reqwest-client" }

This points to a local path outside the repository (../lance-namespace/...). This will break CI and every other developer's build. The Cargo.lock changes (removing source and checksum) confirm this. This must be reverted before merge.

P0: "Atomic" batch create is not actually atomic

batch_create_table_versions claims atomicity but the two-phase approach has a critical gap:

Phase 1: Copies all staging manifests to final locations (visible to readers immediately)
Phase 2: Inserts entries into __manifest via MergeInsert with WhenMatched::Fail

If Phase 2 fails (e.g., conflict on a duplicate version), Phase 1 has already written manifest files to their final paths. There is no rollback/cleanup of these orphaned files. Other readers scanning _versions/ directories will see these phantom manifests. This defeats the purpose of atomic multi-table versioning.

You need either:

Reverse the order: insert into __manifest first (as the commit point), then copy files. Readers using table_version_storage_enabled will only see versions in the manifest.
Add a rollback in Phase 2's error path that deletes the files written in Phase 1.

P1: `delete_table_versions` is O(n) per version — will be very slow for large ranges

for (start, end) in ranges {
    for version in *start..=*end {
        // scan + check existence + delete, one at a time
    }
}

Deleting a range of 1000 versions means 1000 separate scans and 1000 separate delete operations. This should use a single filter expression like object_id IN (...) or starts_with(object_id, ...) AND version BETWEEN ... with a single delete call.

P1: `query_table_versions` loads everything into memory, then sorts client-side

The scanner fetches all matching rows, collects into a Vec, sorts in Rust, then truncates. For tables with many versions, this is unnecessarily expensive. Consider pushing sort/limit into the scan if Lance supports it, or at minimum document this as a known limitation.

Minor

use serde_json; (added in both dir.rs and manifest.rs) is unnecessary in Rust 2018+ — external crates are available by name without an explicit use.
list_builder.append_null() for every entry in batch_insert_into_manifest means base_objects is always null for version entries. This is fine semantically but worth a comment explaining the intent.
In ListTableVersions, entries with missing manifest_path in the JSON metadata are silently dropped via filter_map. This should at minimum log a warning, as it could mask data corruption.

Summary: The dependency path change is a build-breaker, and the non-atomic "atomic" batch create is a correctness issue that undermines the core goal of this PR. I'd recommend fixing these before proceeding with review of the rest.

XuQianJin-Stars · 2026-03-12T08:37:07Z

hi @jackye1995 Could a new version be released for https://github.com/lance-format/lance-namespace to support the interfaces related to lance-format/lance-namespace#310 and lance-format/lance-namespace#311? The newly added interface methods in the current 0.5.2 version are all missing.

wojiaodoubao · 2026-03-12T14:14:11Z

Thanks @XuQianJin-Stars for your great work!!! I'd like to help review this pr! I will start after the namespace issue is fixed, feel free to ping me if I miss the progress.

You can also fix the fmt/clippy issue when you have time.

jackye1995 · 2026-03-13T02:07:36Z

rust/lance-namespace-impls/src/dir.rs

        }

+        // If table_version_storage_enabled is enabled, also record in __manifest
+        if self.table_version_storage_enabled {


This should be a best effort, okay to fail

jackye1995 · 2026-03-13T02:20:19Z

rust/lance-namespace-impls/src/dir.rs

    /// - Storage options are invalid
    pub async fn build(self) -> Result<DirectoryNamespace> {
+        // Validate: table_version_storage_enabled requires manifest_enabled
+        if self.table_version_storage_enabled && !self.manifest_enabled {


When enabled, we should ensure the __manifest table has a property persistedin it's metadata. This is because once we enable this, we should start to reject all writes that are not us the storage to ensure consistency. (Maybe time to develop the same feature flag system like the table format for forward compatibility)

Let's add the boolean flag for now, and we can do a more formal feature flag system in a separated PR to make the experience better, and also introduce a similar feature flag for manifest_enabled

jackye1995 · 2026-03-13T02:24:58Z

rust/lance-namespace-impls/src/dir.rs

    ) -> Result<ListTableVersionsResponse> {
+        // When table_version_storage_enabled, query from __manifest
+        if self.table_version_storage_enabled {
+            if let Some(ref manifest_ns) = self.manifest_ns {


All these table version specific logic should go to manifest.rs since it requires manifest. We can call into it here.

jackye1995 · 2026-03-13T02:55:26Z

rust/lance-namespace-impls/src/dir/manifest.rs


    /// Convert a table ID (vec of strings) to an object_id string
-    fn str_object_id(table_id: &[String]) -> String {
+    pub fn str_object_id(table_id: &[String]) -> String {


We should just rename the table_id variable to object_id, and we can use that also for cases like

let object_id = format!("{}${}", table_id, version);

below

jackye1995 · 2026-03-13T02:56:43Z

rust/lance-namespace-impls/src/dir/manifest.rs

+        // table_version object_ids are formatted as "{table_id}${version}"
+        let filter = format!(
+            "object_type = 'table_version' AND starts_with(object_id, '{}$')",
+            escaped_id


I think we should store the version number in lexigraphical way, like 00000000001 for version 1, so the versions are ordered and we can do range queries more easily

jackye1995 · 2026-03-13T03:06:35Z

rust/lance-namespace-impls/src/dir/manifest.rs

+                    ))))
+                })?;
+
+        merge_builder.when_matched(lance::dataset::WhenMatched::Fail);


nit: we should try to import things at top of module when possible ( I'm not doing a good job with that either 😆)

jackye1995 · 2026-03-13T07:12:55Z

rust/lance-namespace-impls/src/dir/manifest.rs

+    /// contains the full metadata JSON stored in the manifest (manifest_path, manifest_size,
+    /// e_tag, naming_scheme).
+    ///
+    /// **Known limitation**: All matching rows are loaded into memory, sorted in Rust,


okay sorry I missed that we need descending. Looks like we probably need to do the reverse trick like what we do for the manifest file name, but using 9999999999999 instead of long max here, so that it's ordered first in the string form.

Because Lance scan is ordered, by doing this, we should always be able to just do LIMIT 1 to get the latest table version.

oh actually, nvm it's ordered by the row address, not by the actual value. Then the current way should be fine for now, it should also work if there is a btree, but might require some additional optimization in the query planning that we can do later.

jackye1995 · 2026-03-13T07:15:10Z

rust/lance-namespace-impls/src/dir/manifest.rs

+        {
+            return Ok(Some(metadata));
+        }
+        // Fall back to legacy plain integer format


no need to fallback

jackye1995 · 2026-03-13T15:55:01Z

rust/lance-namespace-impls/src/dir/manifest.rs

+    TableVersion,
+    /// A property entry used to persist feature flags and configuration
+    /// in the __manifest table (e.g., `table_version_storage_enabled`).
+    Property,


we don't need this, when I say property, I am saying you update the __manifest table's table metadata key-value map, which can be done with dataset.update_metadata()

jackye1995 · 2026-03-13T15:55:44Z

rust/lance-namespace-impls/src/dir/manifest.rs

+    ///
+    /// Versions are stored as 20-digit zero-padded integers (e.g., `00000000000000000001`
+    /// for version 1) so that string-based range queries and sorting work correctly.
+    pub fn format_version(version: i64) -> String {


nit: should be named format_table_version

jackye1995

Overall looks great! I would suggest we get all features except for the batch_create_table_versions/batch_commit_tables in first, and then we get a few subsequent items:

add support for atomically insert and delete entries through merge_insert
add batch_commit_tables officially with updated lance-namespace crate version
add some more official feature flag setup like the table format

jackye1995 · 2026-03-16T04:38:09Z

rust/lance-namespace-impls/src/dir/manifest.rs

+    ///
+    /// The object_id is formatted as `{table_id}${zero_padded_version}`.
+    pub fn parse_version_from_object_id(object_id: &str) -> Option<i64> {
+        object_id


this can use split_object_id and then parse the last part.

jackye1995 · 2026-03-16T04:49:05Z

rust/lance-namespace-impls/src/dir.rs

+            }
+        }
+
+        // Legacy behavior: delete physical files directly (no __manifest)


nit: this is not really legacy, it's just when the feature is not enabled.

jackye1995 · 2026-03-16T04:49:43Z

rust/lance-namespace-impls/src/dir.rs

+                // Even if some file deletions fail, the versions are already removed from
+                // __manifest, so they won't be visible to readers. Leftover files are
+                // orphaned but harmless and can be cleaned up later.
+                for te in &table_entries {


this should be extracted as a helper method since it's basically used twice here and below.

jackye1995 · 2026-03-16T04:50:30Z

rust/lance-namespace-impls/src/dir.rs

+        })
+    }
+
+    async fn batch_create_table_versions(


TODO: pending merging the namespace PR to get this updated. But I think we can also just not implement this and merge the remaining, have a separated PR to actually get that implemented, if that is cleaner.

jackye1995 · 2026-03-16T04:53:19Z

rust/lance-namespace-impls/src/dir.rs

+        // Phase 1 (atomic commit point): Execute inserts and deletes atomically.
+        // Inserts use MergeInsert (WhenMatched::Fail), deletes use DeleteBuilder.
+        // We execute inserts first, then deletes. Both are individual atomic ops on __manifest.
+        if !insert_entries.is_empty() {


if we do them separately, they are not atomic. I think there are 2 ways out: (1) support merge_insert with WhenMatched::InsertOrDelete(condition), so that we can actually choose to insert or delete based on another condition. (2) do a batch commit of 2 staged transactions. I think we should probably do 1, it's a long-overdue feature in merge_insert.

jackye1995 · 2026-03-16T04:55:47Z

rust/lance-namespace-impls/src/dir/manifest.rs


    /// Insert an entry into the manifest table with metadata and base_objects
-    async fn insert_into_manifest_with_metadata(
+    pub async fn insert_into_manifest_with_metadata(


I think we can do some refactoring, instead of having another batch_insert_into_manifest, make this support inserting a list of objects (object_id + object_type)

jackye1995 · 2026-03-16T04:56:16Z

rust/lance-namespace-impls/src/dir/manifest.rs

+    /// This provides atomic multi-table version creation semantics.
+    pub async fn batch_insert_into_manifest(
+        &self,
+        entries: Vec<(String, ObjectType, Option<String>, Option<String>)>,


can make this a struct to make the meaning of each entry more clear

jackye1995 · 2026-03-16T04:57:57Z

rust/lance-namespace-impls/src/dir.rs


+        // Persist table_version_storage_enabled flag in __manifest so that once
+        // enabled, it becomes a permanent property of this namespace.
+        if self.table_version_storage_enabled {


this logic should be moved to manifest::ensure_manifest_table_up_to_date so we always have a central place to track all migration related functionalities.

codecov · 2026-03-16T07:42:40Z

Codecov Report

❌ Patch coverage is 41.80180% with 323 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	29.53%	247 Missing and 13 partials ⚠️
rust/lance-namespace-impls/src/dir.rs	73.17%	43 Missing and 1 partial ⚠️
rust/lance-namespace-impls/src/rest_adapter.rs	15.00%	17 Missing ⚠️
rust/lance-namespace-impls/src/rest.rs	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Add a new `table_version_storage_enabled` flag to `DirectoryNamespaceBuilder` that stores table versions in the `__manifest` table (as `table_version` object type) instead of relying solely on Lance's native version management. This enables atomic multi-table version creation via `BatchCreateTableVersions`. Key changes: **manifest.rs** - Add `TableVersion` variant to `ObjectType` enum - Make `str_object_id` and `insert_into_manifest_with_metadata` public - Add `batch_insert_into_manifest()`: atomically inserts multiple entries using a single MergeInsert with `WhenMatched::Fail` semantics, so if any version already exists the entire batch is rejected - Add `query_table_versions()`: lists all versions for a given table from the manifest, returning version number and full metadata JSON - Add `query_table_version()`: queries a single version entry by table ID and version number, returning its metadata JSON - Add `delete_table_versions()`: removes version entries by table ID and version ranges from the manifest **dir.rs** - Add `table_version_storage_enabled` field with builder method and validation (requires `manifest_enabled=true`) - `CreateTableVersion`: when enabled, also records the version entry in `__manifest` via `insert_into_manifest_with_metadata` - `BatchCreateTableVersions`: two-phase commit — first copies staging manifests to final locations, then atomically inserts all version entries into `__manifest` in a single batch operation - `ListTableVersions`: when enabled, queries versions from `__manifest` and reconstructs `TableVersion` from stored metadata JSON - `DescribeTableVersion`: when enabled and version is specified, queries directly from `__manifest` instead of opening the dataset - `BatchDeleteTableVersions`: after deleting physical files, syncs deletion to `__manifest` via `delete_table_versions()` - Add comprehensive tests for batch create, conflict detection, manifest requirement validation, and single version recording **namespace.rs** - Add `batch_create_table_versions` trait method with default not-supported implementation **rest.rs / rest_adapter.rs** - Add REST endpoints for `batch_create_table_versions` (POST /v1/table/version/batch_create) and `batch_delete_table_versions` (POST /v1/table/:id/version/batch_delete)

Update dir.rs to use generated tuple variant types (CommitTableOperation, CommitTableResult) instead of hand-written inline field variants: - CommitTableOperation::DeclareTable(op) pattern with op.id, op.location - CommitTableResult::DeclareTable(Box::new(CommitTableResultDeclareTable {...})) - Same pattern for CreateTableVersion, DeleteTableVersions, DeregisterTable All 122 tests pass.

…llapsible_if errors - Update lance-namespace-reqwest-client from local path to crates.io v0.6.0 - Fix 5 collapsible_if clippy errors in dir.rs and manifest.rs

- Escape <hash>_<object_id> in doc comment to fix rustdoc invalid-html-tags error - Update python/Cargo.lock for lance-namespace-reqwest-client 0.5.2 -> 0.6.0

github-actions bot added the enhancement New feature or request label Mar 12, 2026

XuQianJin-Stars force-pushed the feat/multi-table-transactions branch from 92a9bae to 8b40e86 Compare March 12, 2026 07:58

XuQianJin-Stars force-pushed the feat/multi-table-transactions branch from 67f0f07 to 4994c3b Compare March 12, 2026 08:53

jackye1995 reviewed Mar 13, 2026

View reviewed changes

XuQianJin-Stars force-pushed the feat/multi-table-transactions branch from 9512267 to f33f35d Compare March 13, 2026 05:47

jackye1995 reviewed Mar 13, 2026

View reviewed changes

XuQianJin-Stars force-pushed the feat/multi-table-transactions branch from 83a214f to 29473fe Compare March 14, 2026 03:24

jackye1995 reviewed Mar 16, 2026

View reviewed changes

github-actions bot added the python label Mar 16, 2026

wojiaodoubao mentioned this pull request Mar 17, 2026

Epic: partitioned namespace #5741

Open

7 tasks

XuQianJin-Stars added 8 commits March 18, 2026 10:24

feat: support atomic multi-table transactions via namespace manifest

f325782

fix rebase conflict

254d675

table version interface change

af7718f

fix some comments

94001c5

fix some comments

d42b26d

fix some comments

c57a748

XuQianJin-Stars added 4 commits March 18, 2026 10:45

fix some comments

eb1ea0c

fix: update lance-namespace-reqwest-client to 0.6.0 and fix clippy co…

f0a8897

…llapsible_if errors - Update lance-namespace-reqwest-client from local path to crates.io v0.6.0 - Fix 5 collapsible_if clippy errors in dir.rs and manifest.rs

fix: escape HTML tag in rustdoc and update python Cargo.lock

2c6475e

- Escape <hash>_<object_id> in doc comment to fix rustdoc invalid-html-tags error - Update python/Cargo.lock for lance-namespace-reqwest-client 0.5.2 -> 0.6.0

fix rebase conflict

7b6abd0

XuQianJin-Stars force-pushed the feat/multi-table-transactions branch from 010a7af to 7b6abd0 Compare March 18, 2026 02:47

XuQianJin-Stars added 6 commits March 18, 2026 10:53

change 0.6.0 -> 0.6.1

3387250

change 0.6.0 -> 0.6.1

882ff29

fix some comments

80a57a2

fix some comments

aa5a0ec

fix 0.6.1

4f0589b

fix 0.6.1

1380625

Conversation

XuQianJin-Stars commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Changes

manifest.rs

dir.rs

namespace.rs

rest.rs / rest_adapter.rs

Depends On

Testing

Uh oh!

github-actions bot commented Mar 12, 2026

PR Review: feat: support atomic multi-table transactions via namespace manifest

P0: Broken dependency path — will not build for anyone else

P0: "Atomic" batch create is not actually atomic

P1: delete_table_versions is O(n) per version — will be very slow for large ranges

P1: query_table_versions loads everything into memory, then sorts client-side

Minor

Uh oh!

XuQianJin-Stars commented Mar 12, 2026

Uh oh!

wojiaodoubao commented Mar 12, 2026

Uh oh!

jackye1995 Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

XuQianJin-Stars commented Mar 12, 2026 •

edited

Loading

P1: `delete_table_versions` is O(n) per version — will be very slow for large ranges

P1: `query_table_versions` loads everything into memory, then sorts client-side

jackye1995 Mar 13, 2026 •

edited

Loading

codecov bot commented Mar 16, 2026 •

edited

Loading