Skip to content

feat: add read_postgres_binary table function and COPY FROM support for binary files#426

Merged
staticlibs merged 3 commits intoduckdb:mainfrom
arnoldwakim:feat/postgres-read-binary
Apr 2, 2026
Merged

feat: add read_postgres_binary table function and COPY FROM support for binary files#426
staticlibs merged 3 commits intoduckdb:mainfrom
arnoldwakim:feat/postgres-read-binary

Conversation

@arnoldwakim
Copy link
Copy Markdown
Contributor

This PR is a follow-up of #411.

It adds the read_postgres_binary table function and COPY FROM ... (FORMAT postgres_binary) for reading PostgreSQL binary dump files from file system.

It introduces PostgresBinaryFileReader which wraps PostgresBinaryParser (from #411) and reads files in chunks (defaults to 32MB but is configurable via buffer_size).
Partial rows at buffer boundaries are carried over to the next read via FindLastCompleteRow. Since the PostgreSQL binary format has no row delimiters or index, rows can span chunk boundaries, the aforementioned function scans ahead to find the last complete row in each buffer so partial rows are carried over to the next read.

The read_postgres_binary(path, columns={...}) table function is exposed through the PostgresBinaryCopyFunction's copy_from_function, which also enables COPY FROM ... (FORMAT postgres_binary).


In this PR we've included a test suite that deals with:

  • basic scalar types (bool, int, float, varchar, blob, hugeint, decimal)
  • temporal types (date, time, timestamp, interval, timetz)
  • complex types (arrays, structs)
  • error handling (missing params, bad file, empty file)
  • a roundtrip, 10k dataset, NULLs, empty result set

@staticlibs staticlibs merged commit b3e5d65 into duckdb:main Apr 2, 2026
15 checks passed
@staticlibs
Copy link
Copy Markdown
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants