This crate loads generated TPC-DS Parquet data into Apache Paimon and runs the same external query files against Paimon or Parquet through Apache DataFusion.
Disclosure: This is a TPC-DS-derived non-TPC benchmark. Its results are not official TPC results and must not be compared with official TPC results.
The crate does not include or download TPC tools, generated data, or query text. Obtain those materials separately and follow their licenses.
apache/datafusion-benchmarks.datafusion-benchmarks/tpcds/queries.The expected generated-data layout is:
/data/tpcds-sf1000/ call_center.parquet/ catalog_page.parquet/ ... web_site.parquet/
The upstream generation flow uses scale factor 1000 for approximately 1 TB of uncompressed generated data. Choose the generator partition count for the target machine and retain it in the test notes. For example:
tpctools generate --benchmark tpcds \ --scale 1000 \ --partitions 64 \ --generator-path /path/to/DSGen-software-code/tools \ --output /data/tpcds python3 tpcdsgen.py convert --scale-factor 1000 --partitions 64
The upstream conversion script currently contains environment-specific paths; inspect and update them before conversion.
cargo build --release -p paimon-tpcds-bench target/release/paimon-tpcds-bench --help
Keep the exact paimon-rust commit, generated-data scale, generator version, file counts, physical bytes, operating system, CPU, memory, and storage model with every published report.
Do not start with SF1000. Use progressively larger datasets:
Use --tables and --query to isolate failures:
target/release/paimon-tpcds-bench load \ --data /data/tpcds-sf10 \ --warehouse /data/paimon-sf10 \ --tables store_sales,date_dim,item target/release/paimon-tpcds-bench run \ --source paimon \ --warehouse /data/paimon-sf10 \ --queries /src/datafusion-benchmarks/tpcds/queries \ --query 1,3-5 \ --output results/paimon-sf10.json
Query selection accepts comma-separated numbers and inclusive ranges. Query files may contain multiple SQL statements; their statements execute sequentially and the file remains one timed benchmark unit.
The loader infers each Parquet schema and creates an unpartitioned append-only Paimon table. It processes and commits one table at a time.
target/release/paimon-tpcds-bench load \ --data /data/tpcds-sf1000 \ --warehouse /data/paimon-sf1000 \ --database tpcds \ --if-exists error \ --target-partitions 64 \ --memory-limit-gib 192 \ --spill-dir /nvme/datafusion-spill \ --max-spill-gib 1024
--if-exists is deliberately explicit:
error stops before changing an existing table;skip leaves the existing table untouched;overwrite runs INSERT OVERWRITE from the Parquet source.The loader's elapsed time is operational information, not part of query performance results.
target/release/paimon-tpcds-bench run \ --source paimon \ --warehouse /data/paimon-sf1000 \ --database tpcds \ --queries /src/datafusion-benchmarks/tpcds/queries \ --output results/datafusion-paimon-sf1000.json \ --warmup 1 \ --iterations 3 \ --target-partitions 64 \ --memory-limit-gib 192 \ --spill-dir /nvme/datafusion-spill \ --max-spill-gib 1024
Use the same binary, runtime settings, query files, and generated data. The --warehouse path is only a lightweight catalog location for session-scoped Parquet tables; use an empty location separate from the measured data.
target/release/paimon-tpcds-bench run \ --source parquet \ --data /data/tpcds-sf1000 \ --warehouse /data/parquet-benchmark-catalog \ --database tpcds \ --queries /src/datafusion-benchmarks/tpcds/queries \ --output results/datafusion-parquet-sf1000.json \ --warmup 1 \ --iterations 3 \ --target-partitions 64 \ --parquet-pushdown-filters \ --memory-limit-gib 192 \ --spill-dir /nvme/datafusion-spill \ --max-spill-gib 1024
This is an end-to-end source comparison. Loading the data into Paimon rewrites the physical files, so it is not a pure measurement of catalog or manifest overhead.
--parquet-pushdown-filters only controls DataFusion's Parquet reader. Paimon always receives supported predicates for conservative pruning, while exact row filtering stays in the parent DataFusion operator for benchmark runs.
Run and label cold and warm experiments separately:
--warmup 1 or more and report only measured iterations.--warmup 0, start from a documented cache state, and perform OS page-cache eviction outside this tool only when the test operator can do so safely.The runner does not drop the OS page cache. Do not mix cold and warm timings in one aggregate.
The versioned report records:
operator_peak_memory_bytes is the sum of available operator metrics, not a process-wide peak RSS. A zero value can mean that the physical operators did not publish that metric. Paimon scans currently do not publish a precise bytes_scanned execution metric, so that field may also be zero.
The report is written even when a measured query fails; the command then exits non-zero. Preserve failed queries in comparisons instead of silently excluding them.
Report per-query distributions and failure counts. Useful summaries include median, p95, total elapsed time, and geometric mean over the same successful query set. Never hide OOM, timeout, unsupported SQL, or correctness failures by computing an aggregate only from the remaining queries.