commit | 32fd54e3c1595897d02a188b00badebd0b9ef4e2 | [log] [tgz] |
---|---|---|
author | wudi <676366545@qq.com> | Mon Nov 06 16:27:08 2023 +0800 |
committer | GitHub <noreply@github.com> | Mon Nov 06 16:27:08 2023 +0800 |
tree | 1a95a05638443ced43b852f576f5086d7872daa6 | |
parent | 9e682392d79aa607b68856cc93f761aedeb32dd4 [diff] |
add multisink to DorisBatchSink (#223) Support multi-table writing on DorisBatchSink Example: ```java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(1); DorisBatchSink.Builder<RecordWithMeta> builder = DorisBatchSink.builder(); final DorisReadOptions.Builder readOptionBuilder = DorisReadOptions.builder(); Properties properties = new Properties(); properties.setProperty("column_separator", ","); properties.setProperty("line_delimiter", "\n"); properties.setProperty("format", "csv"); DorisOptions.Builder dorisBuilder = DorisOptions.builder(); dorisBuilder.setFenodes("127.0.0.1:8030") .setTableIdentifier("") .setUsername("root") .setPassword(""); DorisExecutionOptions.Builder executionBuilder = DorisExecutionOptions.builder(); executionBuilder.setLabelPrefix("label") .setStreamLoadProp(properties) .setDeletable(false) .setBufferFlushMaxBytes(8 * 1024) .setBufferFlushMaxRows(10) .setBufferFlushIntervalMs(1000 * 10); builder.setDorisReadOptions(readOptionBuilder.build()) .setDorisExecutionOptions(executionBuilder.build()) .setDorisOptions(dorisBuilder.build()); //Multiple table writing RecordWithMeta record = new RecordWithMeta("test", "test_flink_tmp1", "wangwu,1"); RecordWithMeta record1 = new RecordWithMeta("test", "test_flink_tmp", "wangwu,1"); DataStreamSource<RecordWithMeta> source = env.fromCollection(Arrays.asList(record, record1)); source.sinkTo(builder.build()); ```
Flink Doris Connector now support flink version from 1.11 to 1.17.
If you wish to contribute or use a connector from flink 1.13 (and earlier), please use the branch-for-flink-before-1.13
More information about compilation and usage, please visit Flink Doris Connector
You need to copy customer_env.sh.tpl to customer_env.sh before build and you need to configure it before build.
git clone git@github.com:apache/doris-flink-connector.git cd doris-flink-connector/flink-doris-connector ./build.sh
If you find any bugs, feel free to file a GitHub issue or fix it by submitting a pull request.
Contact us through the following mailing list.
Name | Scope | |||
---|---|---|---|---|
dev@doris.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |