tag | 4ee6e4cc8ff99afd7fdfd1992bb7369a0ccd5a54 | |
---|---|---|
tagger | jiafeng.zhang <zhangjf1@gmail.com> | Fri Jul 08 12:56:10 2022 +0800 |
object | d361d3d79f12c8c48e8fcb254fb70cfb493a0c9c |
[maven-release-plugin] copy for tag 3.2_2.12-1.1.0
commit | d361d3d79f12c8c48e8fcb254fb70cfb493a0c9c | [log] [tgz] |
---|---|---|
author | jiafeng.zhang <zhangjf1@gmail.com> | Fri Jul 08 12:56:10 2022 +0800 |
committer | jiafeng.zhang <zhangjf1@gmail.com> | Fri Jul 08 12:56:10 2022 +0800 |
tree | 27a9d2ec9adaeacd0fe08a69b99e6da85c8bcf05 | |
parent | 6f9c1b638e132471e93264551dcfdf14b5dc997f [diff] |
[maven-release-plugin] prepare release 3.2_2.12-1.1.0
More information about compilation and usage, please visit Spark Doris Connector
$ docker pull apache/incubator-doris:build-env-ldb-toolchain-latest
the result of compile jar is like:spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar
download spark for https://spark.apache.org/downloads.html .if in china there have a good choice of tencent link https://mirrors.cloud.tencent.com/apache/spark/spark-3.1.2/
#download wget https://mirrors.cloud.tencent.com/apache/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz #decompression tar -xzvf spark-3.1.2-bin-hadoop3.2.tgz
vim /etc/profile export SPARK_HOME=/your_parh/spark-3.1.2-bin-hadoop3.2 export PATH=$PATH:$SPARK_HOME/bin source /etc/profile
cp /your_path/spark-doris-connector/target/spark-doris-connector-3.1_2.12-1.0.0-SNAPSHOT.jar $SPARK_HOME/jars
created doris database and table。
create database mongo_doris; use mongo_doris; CREATE TABLE data_sync_test_simple ( _id VARCHAR(32) DEFAULT '', id VARCHAR(32) DEFAULT '', user_name VARCHAR(32) DEFAULT '', member_list VARCHAR(32) DEFAULT '' ) DUPLICATE KEY(_id) DISTRIBUTED BY HASH(_id) BUCKETS 10 PROPERTIES("replication_num" = "1"); INSERT INTO data_sync_test_simple VALUES ('1','1','alex','123');
import org.apache.doris.spark._ val dorisSparkRDD = sc.dorisRDD( tableIdentifier = Some("mongo_doris.data_sync_test"), cfg = Some(Map( "doris.fenodes" -> "127.0.0.1:8030", "doris.request.auth.user" -> "root", "doris.request.auth.password" -> "" )) ) dorisSparkRDD.collect()
spark.yarn.jars=hdfs:///spark-jars/doris-spark-connector-3.1.2-2.12-1.0.0.jar
Link:https://github.com/apache/doris/discussions/9486
dorisSparkDF = spark.read.format("doris") .option("doris.table.identifier", "mongo_doris.data_sync_test") .option("doris.fenodes", "127.0.0.1:8030") .option("user", "root") .option("password", "") .load() # show 5 lines data dorisSparkDF.show(5)
If you find any bugs, feel free to file a GitHub issue or fix it by submitting a pull request.
Contact us through the following mailing list.
Name | Scope | |||
---|---|---|---|---|
dev@doris.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |