Epic: partitioned namespace

[wojiaodoubao]

Code Design

1. ManifestNamespace

• The ManifestNamespace will introduce an extended_properties feature, which supports storing specified properties as __manifest columns. These properties can be specified during create/describe table or namespace operations and are then set to or read from __manifest extended columns.
• ManifestNamespace provides a public insert_into_manifest method to facilitate transactional commits for PartitionedNamespace.

2. PartitionedNamespace
PartitionedNamespace is an impl of trait LanceNamespace, and has a member of ManifestNamespace. PartitionedNamespace exposes two categories of APIs:

• The first category inherits methods from LanceNamespace. All of their implementations directly delegate to ManifestNamespace.
• The second category consists of methods with partition semantics, such as plan_scan and resolve_or_create_partition_table. These methods encapsulate partition-related logic and are exposed to upper-layer engines for integration via public APIs.

Step by Step

• (lance-namespace) Add partition spec to rest.yaml. feat: add partition spec to rest.yaml lance-namespace#297
• (lance-namespace) Add table properties. feat: add table properties lance-namespace#299
• (lance) Add manifest namespace extended properties. feat: add manifest namespace extended properties #6215
• (fix) TooMuchWriteContention should be converted to a retryable namespace error fix: lance error TooMuchWriteContention should be converted to a retryable namespace error. #6177
• (lance) Multi-table transactions. feat: support atomic multi-table transactions via namespace manifest #6173
• (lance) Add partitioned namespace. feat: add partitioned namespace #5896
• (lance) Implement PartitionedNamespace. feat: add framework of partitioned namespace #5638

After PartitionedNamespace is implemented, we can start integrating with engines.

Related Links

• Discussion. Introduce Partition Table Spec in Lance Namespace lance-namespace#272

[wojiaodoubao]

Implement BatchCreateTableVersions in ManifestNamespace

@jackye1995 described the design for multi-table transactions in the discussion(#5849 (reply in thread)). In short:

• By introducing LanceNamespaceExternalManifestStore, the Lance namespace can manage commits for Lance tables.
• By implementing the BatchCreateTableVersions interface in the Lance namespace, we can support multi-table transactions.

The next step is to implement the BatchCreateTableVersions interfaces in DirectoryNamespace v2 (ManifestNamespace). After that, we will be able to use them in the partitioned namespace to achieve atomic commits across multiple partition tables.

Discussions about partitioned tables are currently scattered across multiple issues and discussions. I would like to restate the design proposal here so that others can better understand the background and participate.

Manifest Table Version Design

ManifestNamespace needs to add a table_version column to the __manifest table, and introduce a new object_type = "table_version".

Each table commit will insert a new record into __manifest:

<object_id={table_id}, object_type="table_version", table_version={version}, others=null>

The version-related methods in the manifest namespace will then become table queries:

Method	Description	Implementation
list_table_versions	List all records where object_id={table_id} and object_type="table_version".	Query the __manifest table and return all matching rows.
describe_table_version	Query records where object_id={table_id} and object_type="table_version", and return the maximum table_version.	Query the __manifest table and compute MAX(table_version).
create_table_version	Insert a new table version record.	Use merge_insert to insert <object_id={table_id}, object_type="table_version", table_version={version}, others=null>.
batch_delete_table_versions	Delete table versions within a given range.	Remove records whose table_version falls within the specified range.
batch_create_table_versions	Insert multiple table versions in a batch.	Use merge_insert to insert multiple records in batch.

Example of the new __manifest table

object_id	object_type	metadata	table_version	read_tag
v1	namespace	{}	NULL	NULL
v1$m1	namespace	{}	NULL	NULL
v1$m1$dataset	table	{}	NULL	NULL
v1$m1$dataset	table_version	NULL	1	NULL
v2	namespace	{}	NULL	NULL
v2$m2	namespace	{}	NULL	NULL
v2$m2$j3	namespace	{}	NULL	NULL
v2$m2$j3$dataset	table	{}	NULL	NULL
v2$m2$j3$dataset	table_version	NULL	1	NULL
v2$m2$j3$dataset	table_version	NULL	2	NULL