Skip to content

Pool: add configuration and introspection function#430

Merged
staticlibs merged 3 commits intoduckdb:mainfrom
staticlibs:configure_pool_fun
Apr 6, 2026
Merged

Pool: add configuration and introspection function#430
staticlibs merged 3 commits intoduckdb:mainfrom
staticlibs:configure_pool_fun

Conversation

@staticlibs
Copy link
Copy Markdown
Collaborator

@staticlibs staticlibs commented Apr 5, 2026

When a Postgres database is being attached, a connection pool is created for this database.

This PR adds the ability to configure connection pool details for both newly created pools (newly attached databases) and also for already existing pools (for databases that are already attached).

It adds the following configuration options that only apply to databases attached after the option is set:

  • pg_pool_max_connections (UBIGINT, default: max(num_cpus, 8)): maximum number of connections that are allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used.
  • pg_pool_wait_timeout_millis (UBIGINT, default: 30000): maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken.
  • pg_pool_enable_thread_local_cache (BOOLEAN, default: true): whether to enable the connection caching in thread-local cache. Such connections are getting pinned to the threads and are not made available to other threads, while still taking the place in the pool.
  • pg_pool_max_lifetime_millis (UBIGINT, default: 0 - unlimited): maximum number of milliseconds the connection can be kept open. This number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled ('pg_pool_enable_reaper_thread' option), then this number is checked in background periodically.
  • pg_pool_idle_timeout_millis (UBIGINT, default: 0 - unlimited): maximum number of milliseconds the connection can be kept idle in the pool. This number is checked when the connection is taken from the pool. When the connection pool reaper thread is enabled ('pg_pool_enable_reaper_thread' option), then this number is checked in background periodically.
  • pg_pool_enable_reaper_thread (default: FALSE): whether to enable the connection pool reaper thread, that periodically scans the pool to check the 'max_lifetime_millis' and 'idle_timeout_millis' and closes the connection which exceed the specified values. Either 'max_lifetime_millis' or 'idle_timeout_millis' must be set to a non-zero value for this option to be effective.
  • pg_pool_health_check_query (VARCHAR, default: SELECT 1): the query that is used to check that the connection is healthy. Setting this option to an empty string disables the health check.

Pool options that existed before, these options apply to both newly created pools and also to all existing pools:

  • pg_connection_limit (default: max(num_cpus, 8)): the same as pg_pool_max_connections
  • pg_connection_cache (default: TRUE): the same as SET pg_pool_max_connections = 0 or RESET pg_pool_max_connections.

To change the configuration of an exiting pool new table function is added:

Function: postgres_configure_pool

FROM postgres_configure_pool([<optional named parameters>]) -> TABLE

Allows to change the configuration of an existing connection pool (in an attached Postgres database). Returns the current state of a single pool or of all pools.

Parameters:

  • catalog_name (VARCHAR): the name (alias) of the attached Postgres database to which pool the configuration change is applied and details are returned. When NULL (default) returns the current state of pools for all attached catalogs without changing their configuration. Must be specified and non-NULL when any other option is specified.
  • max_connections (UBIGINT): maximum number of connections that are allowed to be cached in a connection pool for each attached Postgres database. This number can be temporary exceeded when parallel scans are used.
  • wait_timeout_millis (UBIGINT): maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken.
  • enable_thread_local_cache (BOOLEAN): whether to enable the connection caching in thread-local cache. Such connections are getting pinned to the threads and are not made available to other threads, while still taking the place in the pool.
  • max_lifetime_millis (UBIGINT): maximum number of milliseconds the connection can be kept open. This number is checked when the connection is taken from the pool and returned to the pool. When the connection pool reaper thread is enabled ('enable_reaper_thread' argument), then this number is checked in background periodically.
  • idle_timeout_millis (UBIGINT): maximum number of milliseconds the connection can be kept idle in the pool. This number is checked when the connection is taken from the pool. When the connection pool reaper thread is enabled ('enable_reaper_thread' option), then this number is checked in background periodically.
  • enable_reaper_thread (BOOLEAN): whether to enable the connection pool reaper thread, that periodically scans the pool to check the 'max_lifetime_millis' and 'idle_timeout_millis' and closes the connection which exceed the specified values. Either 'max_lifetime_millis' or 'idle_timeout_millis' must be set to a non-zero value for this option to be effective.
  • health_check_query (VARCHAR): the query that is used to check that the connection is healthy. Setting this option to an empty string disables the health check.

Returns:

A table with 1 row for each pool with the following columns:

  • catalog_name (VARCHAR): the name (alias) of the attached Postgres database
  • available_connections (UBIGINT): the number of idle connection that are currently available in the pool
  • max_connections (UBIGINT): maximum number of connections that are allowed to be cached in the pool. This number can be temporary exceeded when parallel scans are used.
  • wait_timeout_millis (UBIGINT): maximum number of milliseconds to wait when acquiring a connection from a pool where all available connections are already taken
  • thread_local_cache_enabled (BOOLEAN): whether caching the connections caching in thread-local cache is enabled
  • thread_local_cache_hits (UBIGINT): the number of times connections were successfully acquired from a thread-local cache without going to the main pool
  • thread_local_cache_misses (UBIGINT): the number of times connections were not available in a thread-local cache and were taken from the main pool instead
  • max_lifetime_millis (UBIGINT): maximum number of milliseconds the connection can be kept open
  • idle_timeout_millis (UBIGINT): maximum number of milliseconds the connection can be kept idle in the pool
  • reaper_thread_running (BOOLEAN): whether the pool reaper thread is running; this thread periodically scans the pool to check the 'max_lifetime_millis' and 'idle_timeout_millis' and closes the connection which exceed the specified values
  • health_check_query (VARCHAR): the query that is used to check that the connection is healthy

Examples:

Change 2 options on a pool for the specified attached database:

ATTACH '...' AS my_db (TYPE POSTGRES);
FROM postgres_configure_pool(catalog_name='my_db', max_lifetime_millis=42000, enable_reaper_thread=TRUE);

Return current pool state for all attached databases:

FROM postgres_configure_pool();

Testing: new tests added for new configuration options and for postgres_configure_pool function.

This PR adds `postgres_configure_pool` table function that can be used
to configure the connection pool for the specified attached Postgres DB
(all for all attached Postgres DBs).

This function return the details about the configured pool.

Testing: basic test added, more test coverage pending.
@staticlibs staticlibs merged commit 0634a6a into duckdb:main Apr 6, 2026
15 checks passed
@staticlibs staticlibs deleted the configure_pool_fun branch April 6, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant