JDBC sink connector
Write data through jdbc. Support Batch mode and Streaming mode, support concurrent writing, support exactly-once semantics (using XA transaction guarantee).
Use Xa transactions
to ensure exactly-once
. So only support exactly-once
for the database which is support Xa transactions
. You can set is_exactly_once=true
to enable it.
name | type | required | default value |
---|---|---|---|
url | String | Yes | - |
driver | String | Yes | - |
user | String | No | - |
password | String | No | - |
query | String | Yes | - |
connection_check_timeout_sec | Int | No | 30 |
max_retries | Int | No | 3 |
batch_size | Int | No | 300 |
batch_interval_ms | Int | No | 1000 |
is_exactly_once | Boolean | No | false |
xa_data_source_class_name | String | No | - |
max_commit_attempts | Int | No | 3 |
transaction_timeout_sec | Int | No | -1 |
common-options | no | - |
The jdbc class name used to connect to the remote data source, if you use MySQL the value is com.mysql.cj.jdbc.Driver. Warn: for license compliance, you have to provide any driver yourself like MySQL JDBC Driver, e.g. copy mysql-connector-java-xxx.jar to $SEATNUNNEL_HOME/lib for Standalone.
userName
password
The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost/test
Query statement
The time in seconds to wait for the database operation used to validate the connection to complete.
The number of retries to submit failed (executeBatch)
For batch writing, when the number of buffers reaches the number of batch_size
or the time reaches batch_interval_ms
, the data will be flushed into the database
For batch writing, when the number of buffers reaches the number of batch_size
or the time reaches batch_interval_ms
, the data will be flushed into the database
Whether to enable exactly-once semantics, which will use Xa transactions. If on, you need to set xa_data_source_class_name
.
The xa data source class name of the database Driver, for example, mysql is com.mysql.cj.jdbc.MysqlXADataSource
, and please refer to appendix for other data sources
The number of retries for transaction commit failures
The timeout after the transaction is opened, the default is -1 (never timeout). Note that setting the timeout may affect exactly-once semantics
Sink plugin common parameters, please refer to Sink Common Options for details
In the case of is_exactly_once = “true”, Xa transactions are used. This requires database support, and some databases require some setup : 1 postgres needs to set max_prepared_transactions > 1
such as ALTER SYSTEM set max_prepared_transactions to 10
. 2 mysql version need >= 8.0.29
and Non-root users need to grant XA_RECOVER_ADMIN
permissions. such as grant XA_RECOVER_ADMIN on test_db.* to 'user1'@'%'
.
there are some reference value for params above.
Simple
jdbc { url = "jdbc:mysql://localhost/test" driver = "com.mysql.cj.jdbc.Driver" user = "root" password = "123456" query = "insert into test_table(name,age) values(?,?)" }
Exactly-once
jdbc { url = "jdbc:mysql://localhost/test" driver = "com.mysql.cj.jdbc.Driver" max_retries = 0 user = "root" password = "123456" query = "insert into test_table(name,age) values(?,?)" is_exactly_once = "true" xa_data_source_class_name = "com.mysql.cj.jdbc.MysqlXADataSource" }