| This is a overview of the benchmark workflow and the scripts involved. The workflow is as follows: |
| |
| 1) Create base benchmark schema and load data into these tables. |
| 2) Create extended benchmark schema (different file formats, compression, etc) |
| and load data by copying from tables created in 1) using INSERT statements. |
| 3) Run the benchmarks using $IMPALA_HOME/bin/run_benchmark.py |
| |
| The *.sql scripts to create the extended benchmarks schema and data loading are dynamically generated |
| using the generate_benchmark_statements.rb script. This script reads in files that describe what |
| combinations of data set, file format, compression algorithm to be used and outputs the query |
| files. |
| |
| The input to the generate_benchmark_statements.rb script is generated using the |
| generate_test_vectors.rb script. This script looks at the different dimension values (defined |
| in benchmark_dimensions.yaml) such as file format = rcfile, sequence file, text and outputs |
| a set of test vectors. It outputs both an exhaustive and reduced set of combinations. |
| |
| Currently, a pre-generated set of vectors is checked in along with the *.sql files so these |
| scripts don't need to be run unless there is a new dimension added/removed. These can be viewed |
| at: benchmark_*.vector and create-benchmark*-generated.sql. |
| |
| For more information about these scripts please view the comments within the scripts themselves. |
