Skip to content

Fix: populate PostgresType.oid for array columns to enable binary COPY#432

Merged
staticlibs merged 1 commit intoduckdb:mainfrom
onhate:fix/array-binary-copy-oid
Apr 7, 2026
Merged

Fix: populate PostgresType.oid for array columns to enable binary COPY#432
staticlibs merged 1 commit intoduckdb:mainfrom
onhate:fix/array-binary-copy-oid

Conversation

@onhate
Copy link
Copy Markdown
Contributor

@onhate onhate commented Apr 7, 2026

Summary

Fixes #431

PostgresType.oid was initialized to 0 and never populated for array child types, causing CopyRequiresText to always fall back to FORMAT TEXT for any table with array columns. This resulted in ~2x slower bulk write throughput.

Root Cause

The check in CopyRequiresText for LIST types:

if (pg_type.children[0].oid != PostgresUtils::ToPostgresOid(child_type)) {
    return true;  // always triggers because oid is always 0
}

Always evaluated as 0 != 1043 → true → FORMAT TEXT because the oid field was never set in either code path.

Changes

1. CreateEmptyPostgresType (DDL-created tables)

When DuckDB creates a table via DDL (e.g. CREATE TABLE ... VARCHAR[]), the child PostgresType.oid was left at 0. Fixed by setting it via ToPostgresOid(), which is correct since DuckDB controls the DDL and the Postgres types will match.

2. TypeToLogicalType (catalog-read tables)

When reading existing Postgres tables from the catalog, the child PostgresType.oid was also never set. Here ToPostgresOid() cannot be used because multiple Postgres type names map to the same DuckDB LogicalType (e.g. both text and varchar map to VARCHAR, but have different OIDs: 25 vs 1043). Added TypeNameToPostgresOid() to resolve the OID from the actual Postgres type name, preserving the text/varchar distinction.

Result

  • varchar[] columns now correctly use binary COPY in both paths
  • text[] columns correctly fall back to TEXT copy (since TEXTOID != VARCHAROID and the binary writer always emits VARCHAROID)
  • Other supported array types (int[], boolean[], uuid[], etc.) also benefit from binary COPY

Test plan

  • New test attach_array_binary_copy.test covering 4 scenarios:
    • DDL-created table with VARCHAR[] and INT[]
    • DDL-created table with BOOLEAN[], DATE[], DOUBLE[], UUID[]
    • Existing varchar[] table created via postgres_execute
    • Existing text[] table created via postgres_execute (verifies TEXT fallback still works)
  • All existing array and binary copy tests pass

PostgresType.oid was initialized to 0 and never populated for array
child types in both code paths:

1. CreateEmptyPostgresType (DDL-created tables): when DuckDB creates a
   table via DDL (e.g. CREATE TABLE with VARCHAR[]), the child
   PostgresType.oid was left at 0. Fixed by setting it via
   ToPostgresOid(), which is correct since DuckDB controls the DDL and
   the Postgres types will match.

2. TypeToLogicalType (catalog-read tables): when reading existing
   Postgres tables from the catalog, the child PostgresType.oid was
   also never set. Here ToPostgresOid() cannot be used because multiple
   Postgres types map to the same DuckDB LogicalType (e.g. both "text"
   and "varchar" map to VARCHAR, but have different OIDs 25 vs 1043).
   Added TypeNameToPostgresOid() to resolve the OID from the actual
   Postgres type name, preserving the text/varchar distinction.

This caused CopyRequiresText to always evaluate:
  0 != ToPostgresOid(child_type) → 0 != 1043 → true → FORMAT TEXT

With the fix:
- varchar[] columns now correctly use binary COPY in both paths
- text[] columns correctly fall back to TEXT (TEXTOID != VARCHAROID),
  which is needed because the binary writer always emits VARCHAROID

Fixes duckdb#431
@onhate
Copy link
Copy Markdown
Contributor Author

onhate commented Apr 7, 2026

Local test results (macOS ARM64, PostgreSQL 17, DuckDB v1.5.1)

Test Assertions Result
attach_array_binary_copy.test (new) 63 ✅ Pass
attach_use_binary_copy.test 22 ✅ Pass
attach_text_array.test 6 ✅ Pass
attach_types.test 401 ✅ Pass
attach_types_multidimensional_array.test 66 ✅ Pass

No regressions in existing array or binary copy tests.

@staticlibs staticlibs merged commit fa3e4f4 into duckdb:main Apr 7, 2026
16 of 17 checks passed
@staticlibs
Copy link
Copy Markdown
Collaborator

Thanks!

@onhate onhate deleted the fix/array-binary-copy-oid branch April 7, 2026 19:34
@onhate
Copy link
Copy Markdown
Contributor Author

onhate commented Apr 7, 2026

that's cool! thanks @staticlibs

staticlibs added a commit to staticlibs/duckdb that referenced this pull request Apr 8, 2026
This Postgres update includes 2 following fixes that were added
recently:

 - duckdb/duckdb-postgres#432
 - duckdb/duckdb-postgres#434
Mytherin added a commit to duckdb/duckdb that referenced this pull request Apr 9, 2026
This Postgres scanner update includes 2 following fixes that were added
recently:

 - duckdb/duckdb-postgres#432
 - duckdb/duckdb-postgres#434
evertlammerts pushed a commit to evertlammerts/duckdb that referenced this pull request Apr 10, 2026
This Postgres update includes 2 following fixes that were added
recently:

 - duckdb/duckdb-postgres#432
 - duckdb/duckdb-postgres#434
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Binary COPY is never used for tables with array columns (PostgresType.oid is never populated)

2 participants