GitHub Workflows

GitHub Actions only loads *.yml / *.yaml files in this directory as workflows. This README is ignored by the runner.

Pipeline overview

A single umbrella workflow (ci.yml) orchestrates everything that runs on pull requests and pushes to main. The umbrella runs cheap preflight checks first, computes which heavy jobs are relevant to the change, and only then fans out to the long-running test/build workflows. Each long workflow is a workflow_call reusable invoked from the umbrella.

                        pull_request | push to main | workflow_dispatch
                                            |
                                            v
                                +-----------------------+
                                |       preflight       |  ubuntu-slim
                                |  (RAT, prettier,      |
                                |   missing-suites,     |
                                |   actionlint)         |
                                +-----------+-----------+
                                            |  on success
                                            v
                                +-----------------------+
                                |        changes        |  ubuntu-slim
                                | (compute-changes.py:  |
                                |  one boolean per      |
                                |  heavy job)           |
                                +-----------+-----------+
                                            |
        +-----------+-----------+-----------+-----------+-----------+-----------+
        |           |           |           |           |           |           |
        v           v           v           v           v           v           v
  pr_build_     pr_build_   pr_benchmark_  docs    spark_3_5    spark_4_0   iceberg_1_10
   linux         macos        check       (push)   (PR+push)   (PR+push)    (PR+push)
  (PR+push)    (PR+push)    (PR+push)
                                                       |           |           |
                                                       v           v           v
                                            spark_3_4 / spark_4_1   iceberg_1_8 / 1_9
                                            (push or PR + label)    (push only)

  reusable workflows invoked via `uses:`:
    pr_build_linux.yml         spark_sql_test_reusable.yml
    pr_build_macos.yml         iceberg_spark_test_reusable.yml
    pr_benchmark_check.yml
    docs.yaml

What runs when

A heavy job appears in the PR‘s checks list as a skipped entry whenever its path filter or event criteria don’t match. Skipped checks count as passing for branch protection.

Standalone workflows (not under the umbrella)

These workflows have their own triggers because they fire on events the umbrella doesn't watch, or operate independently of the rest of CI:

Reusable workflows (called by ci.yml)

Modifying path filters

Each long workflow's “what files trigger me” rules live in the FILTERS dict at the top of dev/ci/compute-changes.py. The changes job in ci.yml invokes that script and the gate if: on each long job consumes needs.changes.outputs.<name>. When adding a new test suite or moving sources, update the relevant filter entry there.

Branch protection

Required-check names changed when these workflows were consolidated. The umbrella exposes per-job names like CI / pr_build_linux / Lint, CI / spark_3_5 / linux-test (...), etc. Update repository branch protection rules to point at the new names; the old standalone workflow names (Spark SQL Tests (Spark 3.5), PR Build (Linux), ...) no longer exist as top-level workflows.