Comet Benchmarking Scripts

This directory contains scripts used for generating benchmark results that are published in this repository and in the Comet documentation.

For full instructions on running these benchmarks on an EC2 instance, see the Comet Benchmarking on EC2 Guide.

Example usage

Set Spark environment variables:

export SPARK_HOME=/opt/spark-3.5.3-bin-hadoop3/
export SPARK_MASTER=spark://yourhostname:7077

Set path to queries and data:

export TPCH_QUERIES=/mnt/bigdata/tpch/queries/
export TPCH_DATA=/mnt/bigdata/tpch/sf100/

Run Spark benchmark:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
sudo ./drop-caches.sh
./spark-tpch.sh

Run Comet benchmark:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
sudo ./drop-caches.sh
./comet-tpch.sh

Run Gluten benchmark:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export GLUTEN_JAR=/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar
sudo ./drop-caches.sh
./gluten-tpch.sh

Generating charts:

python3 generate-comparison.py --benchmark tpch --labels "Spark 3.5.3" "Comet 0.9.0" "Gluten 1.4.0" --title "TPC-H @ 100 GB (single executor, 8 cores, local Parquet files)" spark-tpch-1752338506381.json comet-tpch-1752337818039.json gluten-tpch-1752337474344.json

Iceberg Benchmarking

Comet includes native Iceberg support via iceberg-rust integration. This enables benchmarking TPC-H queries against Iceberg tables with native scan acceleration.

Prerequisites

Download the Iceberg Spark runtime JAR (required for running the benchmark):

wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.8.1/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar

Note: Table creation uses --packages which auto-downloads the dependency.

Create Iceberg TPC-H tables

Convert existing Parquet TPC-H data to Iceberg format:

export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
export ICEBERG_CATALOG=${ICEBERG_CATALOG:-local}

$SPARK_HOME/bin/spark-submit \
    --master $SPARK_MASTER \
    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \
    --conf spark.driver.memory=8G \
    --conf spark.executor.instances=1 \
    --conf spark.executor.cores=8 \
    --conf spark.cores.max=8 \
    --conf spark.executor.memory=16g \
    --conf spark.sql.catalog.${ICEBERG_CATALOG}=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.${ICEBERG_CATALOG}.type=hadoop \
    --conf spark.sql.catalog.${ICEBERG_CATALOG}.warehouse=$ICEBERG_WAREHOUSE \
    create-iceberg-tpch.py \
    --parquet-path $TPCH_DATA \
    --catalog $ICEBERG_CATALOG \
    --database tpch

Run Iceberg benchmark

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
export TPCH_QUERIES=/mnt/bigdata/tpch/queries/
sudo ./drop-caches.sh
./comet-tpch-iceberg.sh

The benchmark uses spark.comet.scan.icebergNative.enabled=true to enable Comet's native iceberg-rust integration. Verify native scanning is active by checking for CometIcebergNativeScanExec in the physical plan output.

Iceberg-specific options

Comparing Parquet vs Iceberg performance

Run both benchmarks and compare:

python3 generate-comparison.py --benchmark tpch \
    --labels "Comet (Parquet)" "Comet (Iceberg)" \
    --title "TPC-H @ 100 GB: Parquet vs Iceberg" \
    comet-tpch-*.json comet-iceberg-tpch-*.json