blob: 97d756cb2e16d1ab8ca303c21fc7c68ecce18070 [file] [log] [blame] [view]
# SQL Configuration File
Before writing the sql config file, please make sure that the name of the config file should end with `.sql`.
## Structure of SQL Configuration File
The `SQL` configuration file appears as follows:
### SQL
```sql
/* config
env {
parallelism = 1
job.mode = "BATCH"
}
*/
CREATE TABLE source_table WITH (
'connector'='jdbc',
'type'='source',
'url' = 'jdbc:mysql://localhost:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'query' = 'select * from source',
'properties'= '{
useSSL = false,
rewriteBatchedStatements = true
}'
);
CREATE TABLE sink_table WITH (
'connector'='jdbc',
'type'='sink',
'url' = 'jdbc:mysql://localhost:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'generate_sink_sql' = 'true',
'database' = 'seatunnel',
'table' = 'sink'
);
INSERT INTO sink_table SELECT id, name, age, email FROM source_table;
```
## Explanation of `SQL` Configuration File
### General Configuration in SQL File
```sql
/* config
env {
parallelism = 1
job.mode = "BATCH"
}
*/
```
In the `SQL` file, common configuration sections are defined using `/* config */` comments. Inside, common configurations like `env` can be defined using `HOCON` format.
### SOURCE SQL Syntax
```sql
CREATE TABLE source_table WITH (
'connector'='jdbc',
'type'='source',
'url' = 'jdbc:mysql://localhost:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'query' = 'select * from source',
'properties' = '{
useSSL = false,
rewriteBatchedStatements = true
}'
);
```
* Using `CREATE TABLE ... WITH (...)` syntax creates a mapping for the source table. The `TABLE` name is the name of the source-mapped table, and the `WITH` syntax contains source-related configuration parameters.
* There are two fixed parameters in the WITH syntax: `connector` and `type`, representing connector plugin name (such as `jdbc`, `FakeSource`, etc.) and source type (fixed as `source`), respectively.
* Other parameter names can reference relevant configuration parameters of the corresponding connector plugin, but the format needs to be changed to `'key' = 'value',`.
* If `'value'` is a sub-configuration, you can directly use a string in `HOCON` format. Note: if using a sub-configuration in `HOCON` format, the internal property items must be separated by `,`, like this:
```sql
'properties' = '{
useSSL = false,
rewriteBatchedStatements = true
}'
```
* If using `'` within `'value'`, it needs to be escaped with `''`, like this:
```sql
'query' = 'select * from source where name = ''Joy Ding'''
```
### SINK SQL Syntax
```sql
CREATE TABLE sink_table WITH (
'connector'='jdbc',
'type'='sink',
'url' = 'jdbc:mysql://localhost:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'generate_sink_sql' = 'true',
'database' = 'seatunnel',
'table' = 'sink'
);
```
* Using `CREATE TABLE ... WITH (...)` syntax creates a mapping for the target table. The `TABLE` name is the name of the target-mapped table, and the `WITH` syntax contains sink-related configuration parameters.
* There are two fixed parameters in the `WITH` syntax: `connector` and `type`, representing connector plugin name (such as `jdbc`, `console`, etc.) and target type (fixed as `sink`), respectively.
* Other parameter names can reference relevant configuration parameters of the corresponding connector plugin, but the format needs to be changed to `'key' = 'value',`.
### INSERT INTO SELECT Syntax
```sql
INSERT INTO sink_table SELECT id, name, age, email FROM source_table;
```
* The `SELECT FROM` part is the table name of the source-mapped table. If the select field has keyword([refrence](https://github.com/JSQLParser/JSqlParser/blob/master/src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt)),you should use it like \`fieldName\`.
```sql
INSERT INTO sink_table SELECT id, name, age, email,`output` FROM source_table;
```
* The `INSERT INTO` part is the table name of the target-mapped table.
* Note: This syntax does **not support** specifying fields in `INSERT`, like this: `INSERT INTO sink_table (id, name, age, email) SELECT id, name, age, email FROM source_table;`
### INSERT INTO SELECT TABLE Syntax
```sql
INSERT INTO sink_table SELECT source_table;
```
* The `SELECT` part directly uses the name of the source-mapped table, indicating that all data from the source table will be inserted into the target table.
* Using this syntax does not generate related `transform` configurations. This syntax is generally used in multi-table synchronization scenarios. For example:
```sql
CREATE TABLE source_table WITH (
'connector'='jdbc',
'type' = 'source',
'url' = 'jdbc:mysql://127.0.0.1:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'table_list' = '[
{
table_path = "source.table1"
},
{
table_path = "source.table2",
query = "select * from source.table2"
}
]'
);
CREATE TABLE sink_table WITH (
'connector'='jdbc',
'type' = 'sink',
'url' = 'jdbc:mysql://127.0.0.1:3306/seatunnel',
'driver' = 'com.mysql.cj.jdbc.Driver',
'user' = 'root',
'password' = '123456',
'generate_sink_sql' = 'true',
'database' = 'sink'
);
INSERT INTO sink_table SELECT source_table;
```
### CREATE TABLE AS Syntax
```sql
CREATE TABLE temp1 AS SELECT id, name, age, email FROM source_table;
```
* This syntax creates a temporary table with the result of a `SELECT` query, used for `INSERT INTO` operations.
* The syntax of the `SELECT` part refers to: [SQL Transform](../transform-v2/sql.md) `query` configuration item
```sql
CREATE TABLE temp1 AS SELECT id, name, age, email FROM source_table;
INSERT INTO sink_table SELECT * FROM temp1;
```
## Example of SQL Configuration File Submission
```bash
./bin/seatunnel.sh --config ./config/sample.sql
```