commit | 98ae2cb5104f34a6914f777601338e58f8d89191 | [log] [tgz] |
---|---|---|
author | Alexey Kudinkin <alexey@infinilake.com> | Tue Jun 07 16:30:46 2022 -0700 |
committer | Y Ethan Guo <ethan.guoyihua@gmail.com> | Tue Jun 07 17:04:01 2022 -0700 |
tree | 8e7a19f0812a4424ba80f481896a06c4daad2678 | |
parent | 06ddb7296c14c5a39c706da604e4d766c24ff6d7 [diff] |
[HUDI-4178] Addressing performance regressions in Spark DataSourceV2 Integration (#5737) There are multiple issues with our current DataSource V2 integrations: b/c we advertise Hudi tables as V2, Spark expects it to implement certain APIs which are not implemented at the moment, instead we're using custom Resolution rule (in HoodieSpark3Analysis) to instead manually fallback to V1 APIs. This commit fixes the issue by reverting DSv2 APIs and making Spark use V1, except for schema evaluation logic.
Apache Hudi (pronounced Hoodie) stands for Hadoop Upserts Deletes and Incrementals
. Hudi manages the storage of large analytical datasets on DFS (Cloud stores, HDFS or any Hadoop FileSystem compatible storage).
Hudi supports three types of queries:
Learn more about Hudi at https://hudi.apache.org
Prerequisites for building Apache Hudi:
# Checkout code and build git clone https://github.com/apache/hudi.git && cd hudi mvn clean package -DskipTests # Start command spark-2.4.4-bin-hadoop2.7/bin/spark-shell \ --jars `ls packaging/hudi-spark-bundle/target/hudi-spark-bundle_2.11-*.*.*-SNAPSHOT.jar` \ --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
To build for integration tests that include hudi-integ-test-bundle
, use -Dintegration-tests
.
To build the Javadoc for all Java and Scala classes:
# Javadoc generated under target/site/apidocs mvn clean javadoc:aggregate -Pjavadocs
The default Spark version supported is 2.4.4. Refer to the table below for building with different Spark and Scala versions.
Maven build options | Expected Spark bundle jar name | Notes |
---|---|---|
(empty) | hudi-spark-bundle_2.11 (legacy bundle name) | For Spark 2.4.4 and Scala 2.11 (default options) |
-Dspark2.4 | hudi-spark2.4-bundle_2.11 | For Spark 2.4.4 and Scala 2.11 (same as default) |
-Dspark2.4 -Dscala-2.12 | hudi-spark2.4-bundle_2.12 | For Spark 2.4.4 and Scala 2.12 |
-Dspark3.1 -Dscala-2.12 | hudi-spark3.1-bundle_2.12 | For Spark 3.1.x and Scala 2.12 |
-Dspark3.2 -Dscala-2.12 | hudi-spark3.2-bundle_2.12 | For Spark 3.2.x and Scala 2.12 |
-Dspark3 | hudi-spark3-bundle_2.12 (legacy bundle name) | For Spark 3.2.x and Scala 2.12 |
-Dscala-2.12 | hudi-spark-bundle_2.12 (legacy bundle name) | For Spark 2.4.4 and Scala 2.12 |
For example,
# Build against Spark 3.2.x mvn clean package -DskipTests -Dspark3.2 -Dscala-2.12 # Build against Spark 3.1.x mvn clean package -DskipTests -Dspark3.1 -Dscala-2.12 # Build against Spark 2.4.4 and Scala 2.12 mvn clean package -DskipTests -Dspark2.4 -Dscala-2.12
Starting from versions 0.11, Hudi no longer requires spark-avro
to be specified using --packages
The default Flink version supported is 1.14. Refer to the table below for building with different Flink and Scala versions.
Maven build options | Expected Flink bundle jar name | Notes |
---|---|---|
(empty) | hudi-flink1.14-bundle_2.11 | For Flink 1.14 and Scala 2.11 (default options) |
-Dflink1.14 | hudi-flink1.14-bundle_2.11 | For Flink 1.14 and Scala 2.11 (same as default) |
-Dflink1.14 -Dscala-2.12 | hudi-flink1.14-bundle_2.12 | For Flink 1.14 and Scala 2.12 |
-Dflink1.13 | hudi-flink1.13-bundle_2.11 | For Flink 1.13 and Scala 2.11 |
-Dflink1.13 -Dscala-2.12 | hudi-flink1.13-bundle_2.12 | For Flink 1.13 and Scala 2.12 |
Unit tests can be run with maven profile unit-tests
.
mvn -Punit-tests test
Functional tests, which are tagged with @Tag("functional")
, can be run with maven profile functional-tests
.
mvn -Pfunctional-tests test
To run tests with spark event logging enabled, define the Spark event log directory. This allows visualizing test DAG and stages using Spark History Server UI.
mvn -Punit-tests test -DSPARK_EVLOG_DIR=/path/for/spark/event/log
Please visit https://hudi.apache.org/docs/quick-start-guide.html to quickly explore Hudi's capabilities using spark-shell.