Comet Benchmarking Scripts

This directory contains scripts used for generating benchmark results that are published in this repository and in the Comet documentation.

For full instructions on running these benchmarks on an EC2 instance, see the Comet Benchmarking on EC2 Guide.

Setup

TPC queries are bundled in benchmarks/tpc/queries/ (derived from TPC-H/DS under the TPC Fair Use Policy).

Usage

All benchmarks are run via run.py:

python3 run.py --engine <engine> --benchmark <tpch|tpcds> [options]

Available engines: spark, comet, comet-iceberg, gluten

Example usage

Set Spark environment variables:

export SPARK_HOME=/opt/spark-3.5.3-bin-hadoop3/
export SPARK_MASTER=spark://yourhostname:7077

Set path to data (TPC queries are bundled in benchmarks/tpc/queries/):

export TPCH_DATA=/mnt/bigdata/tpch/sf100/

Run Spark benchmark:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
sudo ./drop-caches.sh
python3 run.py --engine spark --benchmark tpch

Run Comet benchmark:

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
sudo ./drop-caches.sh
python3 run.py --engine comet --benchmark tpch

Run Gluten benchmark:

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export GLUTEN_JAR=/opt/gluten/gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar
sudo ./drop-caches.sh
python3 run.py --engine gluten --benchmark tpch

Preview a command without running it:

python3 run.py --engine comet --benchmark tpch --dry-run

Generating charts:

python3 generate-comparison.py --benchmark tpch --labels "Spark 3.5.3" "Comet 0.9.0" "Gluten 1.4.0" --title "TPC-H @ 100 GB (single executor, 8 cores, local Parquet files)" spark-tpch-1752338506381.json comet-tpch-1752337818039.json gluten-tpch-1752337474344.json

Engine Configuration

Each engine is defined by a TOML file in engines/. The config specifies JARs, Spark conf overrides, required environment variables, and optional defaults/exports. See existing files for examples.

Iceberg Benchmarking

Comet includes native Iceberg support via iceberg-rust integration. This enables benchmarking TPC-H queries against Iceberg tables with native scan acceleration.

Prerequisites

Download the Iceberg Spark runtime JAR (required for running the benchmark):

wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.8.1/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar

Note: Table creation uses --packages which auto-downloads the dependency.

Create Iceberg tables

Convert existing Parquet data to Iceberg format using create-iceberg-tables.py. The script configures the Iceberg catalog automatically -- no --conf flags needed.

export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
mkdir -p $ICEBERG_WAREHOUSE

# TPC-H
$SPARK_HOME/bin/spark-submit \
    --master $SPARK_MASTER \
    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \
    --conf spark.driver.memory=8G \
    --conf spark.executor.instances=2 \
    --conf spark.executor.cores=8 \
    --conf spark.cores.max=16 \
    --conf spark.executor.memory=16g \
    create-iceberg-tables.py \
    --benchmark tpch \
    --parquet-path $TPCH_DATA \
    --warehouse $ICEBERG_WAREHOUSE

# TPC-DS
$SPARK_HOME/bin/spark-submit \
    --master $SPARK_MASTER \
    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.8.1 \
    --conf spark.driver.memory=8G \
    --conf spark.executor.instances=2 \
    --conf spark.executor.cores=8 \
    --conf spark.cores.max=16 \
    --conf spark.executor.memory=16g \
    create-iceberg-tables.py \
    --benchmark tpcds \
    --parquet-path $TPCDS_DATA \
    --warehouse $ICEBERG_WAREHOUSE

Run Iceberg benchmark

export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar
export ICEBERG_JAR=/path/to/iceberg-spark-runtime-3.5_2.12-1.8.1.jar
export ICEBERG_WAREHOUSE=/mnt/bigdata/iceberg-warehouse
sudo ./drop-caches.sh
python3 run.py --engine comet-iceberg --benchmark tpch

The benchmark uses Comet's native iceberg-rust integration, which is enabled by default. Verify native scanning is active by checking for CometIcebergNativeScanExec in the physical plan output.

create-iceberg-tables.py options

Running with Docker

A Docker Compose setup is provided in infra/docker/ for running benchmarks in an isolated Spark standalone cluster. The Docker image supports both Linux (amd64) and macOS (arm64) via architecture-agnostic Java symlinks created at build time.

Build the image

The image must be built for the correct platform to match the native libraries in the engine JARs (e.g. Comet bundles libcomet.so for a specific OS/arch).

docker build -t comet-bench -f benchmarks/tpc/infra/docker/Dockerfile .

Building a compatible Comet JAR

The Comet JAR contains platform-specific native libraries (libcomet.so / libcomet.dylib). A JAR built on the host may not work inside the Docker container due to OS, architecture, or glibc version mismatches. Use Dockerfile.build-comet to build a JAR with compatible native libraries:

• macOS (Apple Silicon): The host JAR contains darwin/aarch64 libraries which won't work in Linux containers. You must use the build Dockerfile.
• Linux: If your host glibc version differs from the container's, the native library will fail to load with a GLIBC_x.xx not found error. The build Dockerfile uses Ubuntu 20.04 (glibc 2.31) for broad compatibility. Use it if you see UnsatisfiedLinkError mentioning glibc when running benchmarks.

mkdir -p output
docker build -t comet-builder \
    -f benchmarks/tpc/infra/docker/Dockerfile.build-comet .
docker run --rm -v $(pwd)/output:/output comet-builder
export COMET_JAR=$(pwd)/output/comet-spark-spark3.5_2.12-*.jar

Platform notes

macOS (Apple Silicon): Docker Desktop is required.

• Memory: Docker Desktop defaults to a small memory allocation (often 8 GB) which is not enough for Spark benchmarks. Go to Docker Desktop > Settings > Resources > Memory and increase it to at least 48 GB (each worker requests 16 GB for its executor plus overhead, and the driver needs 8 GB). Without enough memory, executors will be OOM-killed (exit code 137).
• File Sharing: You may need to add your data directory (e.g. /opt) to Docker Desktop > Settings > Resources > File Sharing before mounting host volumes.

Linux (amd64): Docker uses cgroup memory limits directly without a VM layer. No special Docker configuration is needed, but you may still need to build the Comet JAR using Dockerfile.build-comet (see above) if your host glibc version doesn‘t match the container’s.

The Docker image auto-detects the container architecture (amd64/arm64) and sets up arch-agnostic Java symlinks. The compose file uses BENCH_JAVA_HOME (not JAVA_HOME) to avoid inheriting the host's Java path into the container.

Start the cluster

Set environment variables pointing to your host paths, then start the Spark master and two workers:

export DATA_DIR=/mnt/bigdata/tpch/sf100
export RESULTS_DIR=/tmp/bench-results
export COMET_JAR=/opt/comet/comet-spark-spark3.5_2.12-0.10.0.jar

mkdir -p $RESULTS_DIR/spark-events
docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml up -d

Set COMET_JAR, GLUTEN_JAR, or ICEBERG_JAR to the host path of the engine JAR you want to use. Each JAR is mounted individually into the container, so you can easily switch between versions by changing the path and restarting.

Run benchmarks

Use docker compose run --rm to execute benchmarks. The --rm flag removes the container when it exits, preventing port conflicts on subsequent runs. Pass --no-restart since the cluster is already managed by Compose, and --output /results so that output files land in the mounted results directory:

docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \
    run --rm -p 4040:4040 bench \
    python3 /opt/benchmarks/run.py \
    --engine comet --benchmark tpch --output /results --no-restart

The -p 4040:4040 flag exposes the Spark Application UI on the host. The following UIs are available during a benchmark run:

Note: The Master UI links to the Application UI using the container's internal hostname, which is not reachable from the host. Use http://localhost:4040 directly to access the Application UI.

The Spark Application UI is only available while a benchmark is running. To inspect completed runs, uncomment the history-server service in docker-compose.yml and restart the cluster. The History Server reads event logs from $RESULTS_DIR/spark-events.

For Gluten (requires Java 8), you must restart the entire cluster with JAVA_HOME set so that all services (master, workers, and bench) use Java 8:

export BENCH_JAVA_HOME=/usr/lib/jvm/java-8-openjdk
docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml down
docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml up -d

docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \
    run --rm bench \
    python3 /opt/benchmarks/run.py \
    --engine gluten --benchmark tpch --output /results --no-restart

Important: Only passing -e JAVA_HOME=... to the bench container is not sufficient -- the workers also need Java 8 or Gluten will fail at runtime with sun.misc.Unsafe errors. Unset BENCH_JAVA_HOME (or switch it back to Java 17) and restart the cluster before running Comet or Spark benchmarks.

Memory limits

Two compose files are provided for different hardware profiles:

docker-compose.yml (workstation default):

Configure via environment variables: WORKER_MEM_LIMIT (default: 32g per worker), BENCH_MEM_LIMIT (default: 10g), WORKER_MEMORY (default: 16g, Spark executor memory), WORKER_CORES (default: 8).

Running on a laptop with small scale factors

For local development or testing with small scale factors (e.g. SF1 or SF10), use the laptop compose file which runs a single worker with reduced memory:

docker compose -f benchmarks/tpc/infra/docker/docker-compose-laptop.yml up -d

This starts one worker (4 GB executor inside an 8 GB container) and a 4 GB bench container, totaling approximately 12 GB of memory.

The benchmark scripts request 2 executor instances and 16 max cores by default (run.py). Spark will simply use whatever resources are available on the single worker, so no script changes are needed.

Comparing Parquet vs Iceberg performance

Run both benchmarks and compare:

python3 generate-comparison.py --benchmark tpch \
    --labels "Comet (Parquet)" "Comet (Iceberg)" \
    --title "TPC-H @ 100 GB: Parquet vs Iceberg" \
    comet-tpch-*.json comet-iceberg-tpch-*.json

Java Flight Recorder Profiling

Use the --jfr flag to capture JFR profiles from the Spark driver and executors. JFR is built into JDK 11+ so no additional dependencies are needed.

python3 run.py --engine comet --benchmark tpch --jfr

JFR recordings are written to /results/jfr/ by default (configurable with --jfr-dir). The driver writes driver.jfr and each executor writes executor.jfr (JFR appends the PID when multiple executors share a path).

With Docker Compose, the /results volume is shared across all containers, so JFR files from both driver and executors are collected in $RESULTS_DIR/jfr/ on the host:

docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \
    run --rm bench \
    python3 /opt/benchmarks/run.py \
    --engine comet --benchmark tpch --output /results --no-restart --jfr

Open the .jfr files with JDK Mission Control, IntelliJ IDEA's profiler, or jfr CLI tool (jfr summary driver.jfr).

async-profiler Profiling

Use the --async-profiler flag to capture profiles with async-profiler. Unlike JFR, async-profiler can profile both Java and native (Rust/C++) code in the same flame graph, making it especially useful for profiling Comet workloads.

Prerequisites

async-profiler must be installed on every node where the driver or executors run. Set ASYNC_PROFILER_HOME to the installation directory:

# Download and extract (Linux x64 example)
wget https://github.com/async-profiler/async-profiler/releases/download/v3.0/async-profiler-3.0-linux-x64.tar.gz
tar xzf async-profiler-3.0-linux-x64.tar.gz -C /opt/async-profiler --strip-components=1
export ASYNC_PROFILER_HOME=/opt/async-profiler

On Linux, perf_event_paranoid must be set to allow profiling:

sudo sysctl kernel.perf_event_paranoid=1   # or 0 / -1 for full access
sudo sysctl kernel.kptr_restrict=0          # optional: enable kernel symbols

Basic usage

python3 run.py --engine comet --benchmark tpch --async-profiler

This produces HTML flame graphs in /results/async-profiler/ by default (driver.html and executor.html).

Choosing events and output format

# Wall-clock profiling (includes time spent waiting/sleeping)
python3 run.py --engine comet --benchmark tpch \
    --async-profiler --async-profiler-event wall

# Allocation profiling with JFR output
python3 run.py --engine comet --benchmark tpch \
    --async-profiler --async-profiler-event alloc --async-profiler-format jfr

# Lock contention profiling
python3 run.py --engine comet --benchmark tpch \
    --async-profiler --async-profiler-event lock

Docker usage

The Docker image includes async-profiler pre-installed at /opt/async-profiler. The ASYNC_PROFILER_HOME environment variable is already set in the compose files, so no extra configuration is needed:

docker compose -f benchmarks/tpc/infra/docker/docker-compose.yml \
    run --rm bench \
    python3 /opt/benchmarks/run.py \
    --engine comet --benchmark tpch --output /results --no-restart --async-profiler

Output files are collected in $RESULTS_DIR/async-profiler/ on the host.

Note: On Linux, the Docker container needs --privileged or SYS_PTRACE capability and perf_event_paranoid <= 1 on the host for cpu/wall events. Allocation (alloc) and lock (lock) events work without special privileges.