Google Git Google Git
Sign in
apache/beam/refs/heads/fix-bigtable-it-idempotent-setup-commit/./sdks/python/apache_beam/testing/benchmarks/tpcds
tree: 0eb3c4faa086264335817a600011f0832060cb7e [search]
  1. __init__.py
  2. build.gradle
  3. README.md
  4. tpcds_launcher.py
  5. tpcds_queries.py
  6. tpcds_schemas.py
  7. tpcds_test.py
sdks/python/apache_beam/testing/benchmarks/tpcds/README.md

Apache Beam Python TPC-DS Benchmark & Evaluation Suite

This package provides a Python TPC-DS benchmark and correctness suite that uses:

  • Cross-language Beam SQL Expansion Service (:sdks:java:extensions:sql:expansion-service:shadowJar) via apache_beam.transforms.sql.SqlTransform
  • Prism Runner via PrismRunner
  • Java TPC-DS schemas and queries copied into tpcds/build via Gradle (copyTpcdsResources)
  • GCS TPC-DS datasets (gs://beam-tpcds/datasets/text/10MB, 1GB, 10GB, 100GB)

Running via Gradle

The Gradle tasks in :sdks:python:apache_beam:testing:benchmarks:tpcds automatically copy the Java TPC-DS schemas and queries into tpcds/build and build the Java SQL expansion service shadow jar before running Python.

1. Run the Supported TPC-DS Queries on the 10MB Dataset

./gradlew :sdks:python:apache_beam:testing:benchmarks:tpcds:run \
  -Ptpcds.args="--queries=supported --data_size=10MB"

2. Run the 13 Canonical Multi-Table Queries

./gradlew :sdks:python:apache_beam:testing:benchmarks:tpcds:run \
  -Ptpcds.args="--queries=canonical13 --data_size=10MB"

3. Run Specific Queries on 1GB GCS Data

./gradlew :sdks:python:apache_beam:testing:benchmarks:tpcds:run \
  -Ptpcds.args="--queries=3,7,42,43,52,55,96 --data_size=1GB --results_directory=/tmp/tpcds_results"

4. Run the Pytest Evaluation Suite

./gradlew :sdks:python:apache_beam:testing:benchmarks:tpcds:pytest