DataX doriswriter plug-in, used to synchronize data from other data sources to Doris through DataX.
The plug-in uses Doris' Stream Load function to synchronize and import data. It needs to be used with DataX service.
DataX is an open source version of Alibaba Cloud DataWorks data integration, an offline data synchronization tool/platform widely used in Alibaba Group. DataX implements efficient data synchronization functions between various heterogeneous data sources including MySQL, Oracle, SqlServer, Postgre, HDFS, Hive, ADS, HBase, TableStore (OTS), MaxCompute (ODPS), Hologres, DRDS, etc.
More details can be found at: https://github.com/alibaba/DataX/
The code of DataX doriswriter plug-in can be found here.
This directory is the doriswriter plug-in development environment of Alibaba DataX.
Because the doriswriter plug-in depends on some modules in the DataX code base, and these module dependencies are not submitted to the official Maven repository, when we develop the doriswriter plug-in, we need to download the complete DataX code base to facilitate our development and compilation of the doriswriter plug-in.
doriswriter/
This directory is the code directory of doriswriter, and this part of the code should be in the Doris code base.
The help doc can be found in doriswriter/doc
init-env.sh
The script mainly performs the following steps:
Git clone the DataX code base to the local
Softlink the doriswriter/
directory to DataX/doriswriter
.
Add <module>doriswriter</module>
to the original DataX/pom.xml
Change httpclient version from 4.5 to 4.5.13 in DataX/core/pom.xml
httpclient v4.5 can not handle redirect 307 correctly.
After that, developers can enter DataX/
for development. And the changes in the DataX/doriswriter
directory will be reflected in the doriswriter/
directory, which is convenient for developers to submit code.
Run init-env.sh
Modify code of doriswriter in DataX/doriswriter
if you need.
Build doriswriter
Build doriswriter along:
mvn clean install -pl plugin-rdbms-util,doriswriter -DskipTests
Build DataX:
mvn package assembly:assembly -Dmaven.test.skip=true
The output will be in target/datax/datax/
.
hdfsreader, hdfswriter and oscarwriter needs some extra jar packages. If you don't need to use these components, you can comment out the corresponding module in DataX/pom.xml.
Compilation error
If you encounter the following compilation errors:
Could not find artifact com.alibaba.datax:datax-all:pom:0.0.1-SNAPSHOT ...
You can try the following solutions:
alibaba/datax/
directory to .m2/repository/com/alibaba/
corresponding to the maven used.Commit code of doriswriter in doriswriter
if you need.
For instructions on using the doriswriter plug-in, please refer to here.