Google Git
Sign in
apache / doris-streamloader / refs/heads/master
commitfcc6652c7183b9b0171ff3673936bb333989fe92[log] [tgz]
authorcfbber <cfbber@users.noreply.github.com>Wed Feb 26 22:21:34 2025 +0800
committerGitHub <noreply@github.com>Wed Feb 26 22:21:34 2025 +0800
treecc9d6f7d2fa913a788fa352e8037043a693036e8
parentc0cd939bae6dcb06f3aa14e0a8e6cff473c15f5a [diff]
 [Feature] Support for Custom Line Break Characters (#27)
2 files changed
tree: cc9d6f7d2fa913a788fa352e8037043a693036e8
  1. .github/
  2. dist/
  3. loader/
  4. reader/
  5. report/
  6. utils/
  7. .asf.yaml
  8. .golangci.yml
  9. build.sh
  10. gen_version.go
  11. go.mod
  12. go.sum
  13. LICENSE.txt
  14. main.go
  15. NOTICE.txt
  16. README.md
README.md

Apache Doris Streamloader

A robust, high-performance and user-friendly alternative to the traditional curl-based Stream Load.

Key Features

  • Parallel Loading: Split data files automatically and perform parallel loading
  • Support for Multiple Files and Directories: Support multiple files and directories load with one shot
  • Path Traversal Support: Support path traversal when the source files are in directories
  • Resilience and Continuity: Resume loading from previous failures and cancellations
  • Automatic Retry Mechanism: Retry automatically when failure
  • Comprehensive and Concise Input Parameters

Usage

doris-streamloader --source_file={FILE_LIST} --url={FE_OR_BE_SERVER_URL}:{PORT} --header={STREAMLOAD_HEADER} --db={TARGET_DATABASE} --table={TARGET_TABLE}
  • FILE_LIST: directory or file list, support * wildcard
  • FE_OR_BE_SERVER_URL & PORT: Doris FE or BE hostname or IP and HTTP port
  • STREAMLOAD_HEADER: supports all headers as curl Stream Load does，multiple headers are separated by ‘?’
  • TARGET_DATABASE & TARGET_TABLE: indicate the target database and table where the data will be loaded

e.g.:

doris-streamloader --source_file="data.csv" --url="http://localhost:8330" --header="column_separator:|?columns:col1,col2" --db="testdb" --table="testtbl"

For additional details and options, refer to our comprehensive docs below.

Docs

User Guide

中文使用文档

Build

To build Streamloader, ensure you have golang installed (version >= 1.19.9). For example, on CentOS:

yum install golang

Then, navigate to the doris-streamloader directory and execute:

cd doris-streamloader && sh build.sh

License

Apache License, Version 2.0