Skip to content

Add custom_metadata support for RecordBatch IPC messages #135

@rustyconover

Description

@rustyconover

Describe the enhancement requested

The Arrow IPC spec supports custom_metadata (key-value pairs) on RecordBatch messages via the Message.custom_metadata field. The Swift FlatBuffers generated code (org_apache_arrow_flatbuf_Message) already has full read/write support for this field, but the Swift RecordBatch model, reader, and writer all ignore it.

Proposed changes

  • Add a customMetadata: [String: String] property to RecordBatch (defaulting to [:] for backward compatibility)
  • Add addMetadata builder methods on RecordBatch.Builder
  • Serialize metadata into the FlatBuffers Message wrapper when writing (covers file, streaming, and Flight paths via toMessage)
  • Deserialize metadata from the FlatBuffers Message wrapper when reading (covers readFile, readStreaming, and fromMessage)

Design notes

  • [String: String] matches the pragmatic API used by pyarrow and most Arrow consumers. Duplicate keys from other implementations are deduplicated (last wins). Dictionary order is not preserved across round-trips.
  • Flight support comes for free since toMessage(batch:) delegates to writeRecordBatch.
  • ArrowTable.from(recordBatches:) does not propagate per-batch metadata, which is expected since ArrowTable is a table-level abstraction.

Files affected

File Change
Sources/Arrow/ArrowTable.swift Add customMetadata property and builder methods
Sources/Arrow/ArrowWriter.swift Serialize metadata into FlatBuffers Message
Sources/Arrow/ArrowReader.swift Deserialize metadata from FlatBuffers Message
Tests/ArrowTests/IPCTests.swift Round-trip, builder API, cross-language, and multi-batch tests

I have an implementation ready and will open a PR referencing this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions