This page describes in detail how to run end to end tests on a hudi dataset that helps in improving our confidence in a release as well as perform large scale performance benchmarks.
hdfs, parquet, spark, hive and avro.payload size, number of updates, number of inserts, number of partitionsinsert, bulk_insert, upsert, compact, queryThe Hudi test suite runs as a long running spark job. The suite is divided into the following high level components :
This component does the work of generating the workload; inserts, upserts etc.
Depending on the type of workload generated, data is either ingested into the target hudi dataset or the corresponding workload operation is executed. For example compaction does not necessarily need a workload to be generated/ingested but can require an execution.
The test suite supports different types of operations besides ingestion such as Hive Query execution, Clean action etc.
org.apache.hudi.testsuite.HoodieTestSuiteJob.java - Entry Point of the hudi test suite job. This class wraps all the functionalities required to run a configurable integration suite.
org.apache.hudi.testsuite.HoodieTestSuiteJob.HoodieTestSuiteConfig - Config class that drives the behavior of the integration test suite. This class extends from com.uber.hoodie.utilities.DeltaStreamerConfig. Look at link#HudiDeltaStreamer page to learn about all the available configs applicable to your test suite.
There are 2 ways to generate a workload pattern
1.Programatically
Choose to write up the entire DAG of operations programatically, take a look at WorkflowDagGenerator class. Once you're ready with the DAG you want to execute, simply pass the class name as follows:
spark-submit ... ... --class org.apache.hudi.testsuite.HoodieTestSuiteJob --workload-generator-classname org.apache.hudi.testsuite.dag.scheduler.<your_workflowdaggenerator> ...
2.YAML file
Choose to write up the entire DAG of operations in YAML, take a look at complex-workload-dag-cow.yaml or complex-workload-dag-mor.yaml. Once you're ready with the DAG you want to execute, simply pass the yaml file path as follows:
spark-submit ... ... --class org.apache.hudi.testsuite.HoodieTestSuiteJob --workload-yaml-path /path/to/your-workflow-dag.yaml ...
The test suite can be found in the hudi-integ-test module. Use the prepare_integration_suite.sh script to build the test suite, you can provide different parameters to the script.
shell$ ./prepare_integration_suite.sh --help Usage: prepare_integration_suite.sh --spark-command, prints the spark command -h, hdfs-version -s, spark version -p, parquet version -a, avro version -s, hive version
shell$ ./prepare_integration_suite.sh .... .... Final command : mvn clean install -DskipTests
Copy over the necessary files and jars that are required to your cluster and then run the following spark-submit command after replacing the correct values for the parameters. NOTE : The properties-file should have all the necessary information required to ingest into a Hudi dataset. For more information on what properties need to be set, take a look at the test suite section under demo steps.
shell$ ./prepare_integration_suite.sh --spark-command spark-submit --packages com.databricks:spark-avro_2.11:4.0.0 --master prepare_integration_suite.sh --deploy-mode --properties-file --class org.apache.hudi.integ.testsuite.HoodieTestSuiteJob target/hudi-integ-test-0.6 .0-SNAPSHOT.jar --source-class --source-ordering-field --input-base-path --target-base-path --target-table --props --storage-type --payload-class --workload-yaml-path --input-file-size --<deltastreamer-ingest>
Take a look at the TestHoodieTestSuiteJob to check how you can run the entire suite using JUnit.
docker exec -it adhoc-2 /bin/bash # COPY_ON_WRITE tables ========================= ## Run the following command to start the test suite spark-submit \ --packages com.databricks:spark-avro_2.11:4.0.0 \ --conf spark.task.cpus=1 \ --conf spark.executor.cores=1 \ --conf spark.task.maxFailures=100 \ --conf spark.memory.fraction=0.4 \ --conf spark.rdd.compress=true \ --conf spark.kryoserializer.buffer.max=2000m \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.memory.storageFraction=0.1 \ --conf spark.shuffle.service.enabled=true \ --conf spark.sql.hive.convertMetastoreParquet=false \ --conf spark.ui.port=5555 \ --conf spark.driver.maxResultSize=12g \ --conf spark.executor.heartbeatInterval=120s \ --conf spark.network.timeout=600s \ --conf spark.eventLog.overwrite=true \ --conf spark.eventLog.enabled=true \ --conf spark.yarn.max.executor.failures=10 \ --conf spark.sql.catalogImplementation=hive \ --conf spark.sql.shuffle.partitions=1000 \ --class org.apache.hudi.testsuite.HoodieTestSuiteJob $HUDI_TEST_SUITE_BUNDLE \ --source-ordering-field timestamp \ --target-base-path /user/hive/warehouse/hudi-integ-test-suite/output \ --input-base-path /user/hive/warehouse/hudi-integ-test-suite/input \ --target-table test_table \ --props /var/hoodie/ws/docker/demo/config/testsuite/test-source.properties \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ --source-limit 300000 \ --source-class org.apache.hudi.utilities.sources.AvroDFSSource \ --input-file-size 125829120 \ --workload-yaml-path /var/hoodie/ws/docker/demo/config/testsuite/complex-workflow-dag-cow.yaml \ --storage-type COPY_ON_WRITE \ --compact-scheduling-minshare 1 \ --hoodie-conf "hoodie.deltastreamer.source.test.num_partitions=100" \ --hoodie-conf "hoodie.deltastreamer.source.test.datagen.use_rocksdb_for_storing_existing_keys=false" \ --hoodie-conf "hoodie.deltastreamer.source.test.max_unique_records=100000000" \ --hoodie-conf "hoodie.embed.timeline.server=false" \ --hoodie-conf "hoodie.datasource.write.recordkey.field=_row_key" \ --hoodie-conf "hoodie.deltastreamer.source.dfs.root=/user/hive/warehouse/hudi-integ-test-suite/input" \ --hoodie-conf "hoodie.datasource.write.keygenerator.class=org.apache.hudi.ComplexKeyGenerator" \ --hoodie-conf "hoodie.datasource.write.partitionpath.field=timestamp" \ --hoodie-conf "hoodie.deltastreamer.schemaprovider.source.schema.file=/var/hoodie/ws/docker/demo/config/testsuite/source.avsc" \ --hoodie-conf "hoodie.datasource.hive_sync.assume_date_partitioning=false" \ --hoodie-conf "hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://hiveserver:10000/" \ --hoodie-conf "hoodie.datasource.hive_sync.database=testdb" \ --hoodie-conf "hoodie.datasource.hive_sync.table=test_table" \ --hoodie-conf "hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor" \ --hoodie-conf "hoodie.datasource.hive_sync.assume_date_partitioning=true" \ --hoodie-conf "hoodie.datasource.write.keytranslator.class=org.apache.hudi.DayBasedPartitionPathKeyTranslator" \ --hoodie-conf "hoodie.deltastreamer.schemaprovider.target.schema.file=/var/hoodie/ws/docker/demo/config/testsuite/source.avsc" ... ... 2019-11-03 05:44:47 INFO DagScheduler:69 - ----------- Finished workloads ---------- 2019-11-03 05:44:47 INFO HoodieTestSuiteJob:138 - Finished scheduling all tasks ... 2019-11-03 05:44:48 INFO SparkContext:54 - Successfully stopped SparkContext # MERGE_ON_READ tables ========================= ## Run the following command to start the test suite spark-submit \ --packages com.databricks:spark-avro_2.11:4.0.0 \ --conf spark.task.cpus=1 \ --conf spark.executor.cores=1 \ --conf spark.task.maxFailures=100 \ --conf spark.memory.fraction=0.4 \ --conf spark.rdd.compress=true \ --conf spark.kryoserializer.buffer.max=2000m \ --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ --conf spark.memory.storageFraction=0.1 \ --conf spark.shuffle.service.enabled=true \ --conf spark.sql.hive.convertMetastoreParquet=false \ --conf spark.ui.port=5555 \ --conf spark.driver.maxResultSize=12g \ --conf spark.executor.heartbeatInterval=120s \ --conf spark.network.timeout=600s \ --conf spark.eventLog.overwrite=true \ --conf spark.eventLog.enabled=true \ --conf spark.yarn.max.executor.failures=10 \ --conf spark.sql.catalogImplementation=hive \ --conf spark.sql.shuffle.partitions=1000 \ --class org.apache.hudi.testsuite.HoodieTestSuiteJob $HUDI_TEST_SUITE_BUNDLE \ --source-ordering-field timestamp \ --target-base-path /user/hive/warehouse/hudi-integ-test-suite/output \ --input-base-path /user/hive/warehouse/hudi-integ-test-suite/input \ --target-table test_table \ --props /var/hoodie/ws/docker/demo/config/testsuite/test-source.properties \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \ --source-limit 300000 \ --source-class org.apache.hudi.utilities.sources.AvroDFSSource \ --input-file-size 125829120 \ --workload-yaml-path /var/hoodie/ws/docker/demo/config/testsuite/complex-workflow-dag-mor.yaml \ --storage-type MERGE_ON_READ \ --compact-scheduling-minshare 1 \ --hoodie-conf "hoodie.deltastreamer.source.test.num_partitions=100" \ --hoodie-conf "hoodie.deltastreamer.source.test.datagen.use_rocksdb_for_storing_existing_keys=false" \ --hoodie-conf "hoodie.deltastreamer.source.test.max_unique_records=100000000" \ --hoodie-conf "hoodie.embed.timeline.server=false" \ --hoodie-conf "hoodie.datasource.write.recordkey.field=_row_key" \ --hoodie-conf "hoodie.deltastreamer.source.dfs.root=/user/hive/warehouse/hudi-integ-test-suite/input" \ --hoodie-conf "hoodie.datasource.write.keygenerator.class=org.apache.hudi.ComplexKeyGenerator" \ --hoodie-conf "hoodie.datasource.write.partitionpath.field=timestamp" \ --hoodie-conf "hoodie.deltastreamer.schemaprovider.source.schema.file=/var/hoodie/ws/docker/demo/config/testsuite/source.avsc" \ --hoodie-conf "hoodie.datasource.hive_sync.assume_date_partitioning=false" \ --hoodie-conf "hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://hiveserver:10000/" \ --hoodie-conf "hoodie.datasource.hive_sync.database=testdb" \ --hoodie-conf "hoodie.datasource.hive_sync.table=test_table" \ --hoodie-conf "hoodie.datasource.hive_sync.partition_extractor_class=org.apache.hudi.hive.NonPartitionedExtractor" \ --hoodie-conf "hoodie.datasource.hive_sync.assume_date_partitioning=true" \ --hoodie-conf "hoodie.datasource.write.keytranslator.class=org.apache.hudi.DayBasedPartitionPathKeyTranslator" \ --hoodie-conf "hoodie.deltastreamer.schemaprovider.target.schema.file=/var/hoodie/ws/docker/demo/config/testsuite/source.avsc" ... ... 2019-11-03 05:44:47 INFO DagScheduler:69 - ----------- Finished workloads ---------- 2019-11-03 05:44:47 INFO HoodieTestSuiteJob:138 - Finished scheduling all tasks ... 2019-11-03 05:44:48 INFO SparkContext:54 - Successfully stopped SparkContext