Google Git
Sign in
apache / doris-spark-connector / 11f4976be1aae4c041ea66e3e83487aa2614c947
commit11f4976be1aae4c041ea66e3e83487aa2614c947[log] [tgz]
authorgnehil <adamlee489@gmail.com>Tue Sep 05 13:56:25 2023 +0800
committerGitHub <noreply@github.com>Tue Sep 05 13:56:25 2023 +0800
treee041cb1d78ea12d377bc5d2c2e83a0c7c744053b
parent71af841c7536d843e9d7babaf69e4b95298a9502 [diff]
[fix] streaming write execution plan error (#135)

* fix streaming write error and add json data pass through option
* handle stream pass through, force set read_json_by_line is true when format is json
4 files changed
tree: e041cb1d78ea12d377bc5d2c2e83a0c7c744053b
  1. .github/
  2. .mvn/
  3. spark-doris-connector/
  4. .asf.yaml
  5. .gitignore
  6. .licenserc.yaml
  7. CODE_OF_CONDUCT.md
  8. CONTRIBUTING.md
  9. CONTRIBUTING_CN.md
  10. custom_env.sh.tpl
  11. env.sh
  12. LICENSE-dependencies.txt
  13. LICENSE.txt
  14. mvnw
  15. NOTICE.txt
  16. README.md
README.md

Spark Connector for Apache Doris

License Join the Doris Community at Slack

Spark Doris Connector

More information about compilation and usage, please visit Spark Doris Connector

License

Apache License, Version 2.0

QuickStart

  1. download and compile Spark Doris Connector from https://github.com/apache/doris-spark-connector, we suggest compile Spark Doris Connector by Doris offfcial image。
$ docker pull apache/doris:build-env-ldb-toolchain-latest

  1. the result of compile jar is like：spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar

  2. download spark for https://spark.apache.org/downloads.html .if in china there have a good choice of tencent link https://mirrors.cloud.tencent.com/apache/spark/spark-3.1.2/

#download
wget https://mirrors.cloud.tencent.com/apache/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
#decompression
tar -xzvf spark-3.1.2-bin-hadoop3.2.tgz
  1. config Spark environment
vim /etc/profile
export SPARK_HOME=/your_parh/spark-3.1.2-bin-hadoop3.2
export PATH=$PATH:$SPARK_HOME/bin
source /etc/profile
  1. copy spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar to spark jars directory。
cp /your_path/spark-doris-connector/target/spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar  $SPARK_HOME/jars

  1. created doris database and table。

    create database mongo_doris;
use mongo_doris;
CREATE TABLE data_sync_test_simple
 (
         _id VARCHAR(32) DEFAULT '',
         id VARCHAR(32) DEFAULT '',
         user_name VARCHAR(32) DEFAULT '',
         member_list VARCHAR(32) DEFAULT ''
 )
 DUPLICATE KEY(_id)
 DISTRIBUTED BY HASH(_id) BUCKETS 10
 PROPERTIES("replication_num" = "1");
INSERT INTO data_sync_test_simple VALUES ('1','1','alex','123');
    1. Input this coed in spark-shell.
import org.apache.doris.spark._
val dorisSparkRDD = sc.dorisRDD(
  tableIdentifier = Some("mongo_doris.data_sync_test"),
  cfg = Some(Map(
    "doris.fenodes" -> "127.0.0.1:8030",
    "doris.request.auth.user" -> "root",
    "doris.request.auth.password" -> ""
  ))
)
dorisSparkRDD.collect()
  • mongo_doris:doris database name
  • data_sync_test:doris table mame.
  • doris.fenodes:doris FE IP:http_port
  • doris.request.auth.user:doris user name.
  • doris.request.auth.password:doris password
  1. if Spark is Cluster model,upload Jar to HDFS，add doris-spark-connector jar HDFS URL in spark.yarn.jars.
spark.yarn.jars=hdfs:///spark-jars/doris-spark-connector-3.1.2-2.12-1.0.0.jar

Link：https://github.com/apache/doris/discussions/9486

  1. in pyspark,input this code in pyspark shell command.
dorisSparkDF = spark.read.format("doris")
.option("doris.table.identifier", "mongo_doris.data_sync_test")
.option("doris.fenodes", "127.0.0.1:8030")
.option("user", "root")
.option("password", "")
.load()
# show 5 lines data 
dorisSparkDF.show(5)

Report issues or submit pull request

If you find any bugs, feel free to file a GitHub issue or fix it by submitting a pull request.

Contact Us

Contact us through the following mailing list.

NameScope
dev@doris.apache.orgDevelopment-related discussionsSubscribeUnsubscribeArchives

Links