- | MYSQL |
+ SQLSERVER |
jdbc_url (str)
- connection_init_sql (list[str])
connection_properties (str)
disable_auto_commit (boolean)
fetch_size (int32)
@@ -179,7 +180,6 @@ and Beam SQL is invoked via the Managed API under the hood.
jdbc_url (str)
autosharding (boolean)
batch_size (int64)
- connection_init_sql (list[str])
connection_properties (str)
location (str)
password (str)
@@ -890,7 +890,7 @@ For more information on table properties, please visit https://iceberg.apache.or
-### `SQLSERVER` Write
+### `MYSQL` Write
@@ -932,6 +932,17 @@ For more information on table properties, please visit https://iceberg.apache.or
n/a
+
+ |
+ connection_init_sql
+ |
+
+ list[str]
+ |
+
+ Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
+ |
+
|
connection_properties
@@ -990,7 +1001,7 @@ For more information on table properties, please visit https://iceberg.apache.or
|
-### `SQLSERVER` Read
+### `MYSQL` Read
@@ -1010,6 +1021,17 @@ For more information on table properties, please visit https://iceberg.apache.or
Connection URL for the JDBC source.
+
+ |
+ connection_init_sql
+ |
+
+ list[str]
+ |
+
+ Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
+ |
+
|
connection_properties
@@ -1123,7 +1145,7 @@ For more information on table properties, please visit https://iceberg.apache.or
|
-### `MYSQL` Read
+### `SQLSERVER` Read
@@ -1143,17 +1165,6 @@ For more information on table properties, please visit https://iceberg.apache.or
Connection URL for the JDBC source.
-
- |
- connection_init_sql
- |
-
- list[str]
- |
-
- Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- |
-
|
connection_properties
@@ -1267,7 +1278,7 @@ For more information on table properties, please visit https://iceberg.apache.or
|
-### `MYSQL` Write
+### `SQLSERVER` Write
@@ -1309,17 +1320,6 @@ For more information on table properties, please visit https://iceberg.apache.or
n/a
-
- |
- connection_init_sql
- |
-
- list[str]
- |
-
- Sets the connection init sql statements used by the Driver. Only MySQL and MariaDB support this.
- |
-
|
connection_properties
@@ -1378,7 +1378,7 @@ For more information on table properties, please visit https://iceberg.apache.or
|
-### `BIGQUERY` Write
+### `BIGQUERY` Read
@@ -1389,74 +1389,63 @@ For more information on table properties, please visit https://iceberg.apache.or
|
- table
+ kms_key
|
str
|
- The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE}
- |
-
-
- |
- drop
- |
-
- list[str]
- |
-
- A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'.
+ Use this Cloud KMS key to encrypt your data
|
|
- keep
+ query
|
- list[str]
+ str
|
- A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
+ The SQL query to be executed to read from the BigQuery table.
|
|
- kms_key
+ row_restriction
|
str
|
- Use this Cloud KMS key to encrypt your data
+ Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query.
|
|
- only
+ fields
|
- str
+ list[str]
|
- The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'.
+ Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3"
|
|
- triggering_frequency_seconds
+ table
|
- int64
+ str
|
- Determines how often to 'commit' progress into BigQuery. Default is every 5 seconds.
+ The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE}
|
-### `BIGQUERY` Read
+### `BIGQUERY` Write
@@ -1467,63 +1456,74 @@ For more information on table properties, please visit https://iceberg.apache.or
|
- kms_key
+ table
|
str
|
- Use this Cloud KMS key to encrypt your data
+ The bigquery table to write to. Format: [${PROJECT}:]${DATASET}.${TABLE}
|
|
- query
+ drop
|
- str
+ list[str]
|
- The SQL query to be executed to read from the BigQuery table.
+ A list of field names to drop from the input record before writing. Is mutually exclusive with 'keep' and 'only'.
|
|
- row_restriction
+ keep
|
- str
+ list[str]
|
- Read only rows that match this filter, which must be compatible with Google standard SQL. This is not supported when reading via query.
+ A list of field names to keep in the input record. All other fields are dropped before writing. Is mutually exclusive with 'drop' and 'only'.
|
|
- fields
+ kms_key
|
- list[str]
+ str
|
- Read only the specified fields (columns) from a BigQuery table. Fields may not be returned in the order specified. If no value is specified, then all fields are returned. Example: "col1, col2, col3"
+ Use this Cloud KMS key to encrypt your data
|
|
- table
+ only
|
str
|
- The fully-qualified name of the BigQuery table to read from. Format: [${PROJECT}:]${DATASET}.${TABLE}
+ The name of a single record field that should be written. Is mutually exclusive with 'keep' and 'drop'.
+ |
+
+
+ |
+ triggering_frequency_seconds
+ |
+
+ int64
+ |
+
+ Determines how often to 'commit' progress into BigQuery. Default is every 5 seconds.
|
-### `POSTGRES` Write
+### `POSTGRES` Read
@@ -1540,173 +1540,173 @@ For more information on table properties, please visit https://iceberg.apache.or
str
- Connection URL for the JDBC sink.
+ Connection URL for the JDBC source.
|
|
- autosharding
+ connection_properties
|
- boolean
+ str
|
- If true, enables using a dynamically determined number of shards to write.
+ Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
|
|
- batch_size
+ fetch_size
|
- int64
+ int32
|
- n/a
+ This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
|
|
- connection_properties
+ location
|
str
|
- Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
+ Name of the table to read from.
|
|
- location
+ num_partitions
|
- str
+ int32
|
- Name of the table to write to.
+ The number of partitions
|
|
- password
+ output_parallelization
|
- str
+ boolean
|
- Password for the JDBC source.
+ Whether to reshuffle the resulting PCollection so results are distributed to all workers.
|
|
- username
+ partition_column
|
str
|
- Username for the JDBC source.
+ Name of a column of numeric type that will be used for partitioning.
|
|
- write_statement
+ password
|
str
|
- SQL query used to insert records into the JDBC sink.
+ Password for the JDBC source.
|
-
-
-
-### `POSTGRES` Read
-
-
-
-
- | Configuration |
- Type |
- Description |
-
|
- jdbc_url
+ read_query
|
str
|
- Connection URL for the JDBC source.
+ SQL query used to query the JDBC source.
|
|
- connection_properties
+ username
|
str
|
- Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
+ Username for the JDBC source.
|
+
+
+
+
+ | Configuration |
+ Type |
+ Description |
+
|
- fetch_size
+ jdbc_url
|
- int32
+ str
|
- This method is used to override the size of the data that is going to be fetched and loaded in memory per every database call. It should ONLY be used if the default value throws memory errors.
+ Connection URL for the JDBC sink.
|
|
- location
+ autosharding
|
- str
+ boolean
|
- Name of the table to read from.
+ If true, enables using a dynamically determined number of shards to write.
|
|
- num_partitions
+ batch_size
|
- int32
+ int64
|
- The number of partitions
+ n/a
|
|
- output_parallelization
+ connection_properties
|
- boolean
+ str
|
- Whether to reshuffle the resulting PCollection so results are distributed to all workers.
+ Used to set connection properties passed to the JDBC driver not already defined as standalone parameter (e.g. username and password can be set using parameters above accordingly). Format of the string must be "key1=value1;key2=value2;".
|
|
- partition_column
+ location
|
str
|
- Name of a column of numeric type that will be used for partitioning.
+ Name of the table to write to.
|
@@ -1722,24 +1722,24 @@ For more information on table properties, please visit https://iceberg.apache.or
|
- read_query
+ username
|
str
|
- SQL query used to query the JDBC source.
+ Username for the JDBC source.
|
|
- username
+ write_statement
|
str
|
- Username for the JDBC source.
+ SQL query used to insert records into the JDBC sink.
|
|