GitHub Workflows

GitHub Actions only loads *.yml / *.yaml files in this directory as workflows. This README is ignored by the runner.

Pipeline overview

A single umbrella workflow (ci.yml) orchestrates everything that runs on pull requests and pushes to main. The umbrella runs cheap preflight checks first, computes which heavy jobs are relevant to the change, and only then fans out to the long-running test/build workflows. Each long workflow is a workflow_call reusable invoked from the umbrella.

                        pull_request | push to main | workflow_dispatch
                                            |
                                            v
                                +-----------------------+
                                |       preflight       |  ubuntu-slim
                                |  (RAT, prettier,      |
                                |   missing-suites,     |
                                |   actionlint)         |
                                +-----------+-----------+
                                            |  on success
                                            v
                                +-----------------------+
                                |        changes        |  ubuntu-slim
                                | (compute-changes.py:  |
                                |  one boolean per      |
                                |  heavy job)           |
                                +-----------+-----------+
                                            |
        +-----------------------------------+-----------------------------------+
        |                                   |                                   |
        v                                   v                                   v
  every PR + push                     push to main only         PR with label, or push
  ---------------                     -----------------         ----------------------
  pr_build_linux                      docs                      spark_3_4    run-spark-3.4-tests
  pr_build_macos                                                spark_4_0    run-spark-4.0-tests
  pr_benchmark_check                                            iceberg_1_8  run-iceberg-tests
  spark_3_5                                                     iceberg_1_9  run-iceberg-tests
  spark_4_1                                                     iceberg_1_10 run-iceberg-tests
  iceberg_1_11

  reusable workflows invoked via `uses:`:
    pr_build_linux.yml         spark_sql_test_reusable.yml
    pr_build_macos.yml         iceberg_spark_test_reusable.yml
    pr_benchmark_check.yml
    docs.yaml

What runs when

A heavy job appears in the PR‘s checks list as a skipped entry whenever its path filter or event criteria don’t match. Skipped checks count as passing for branch protection, so a name that can report skipped is not safe to make a required check.

Label events

ci.yml also fires on pull_request.types: [labeled], so applying run-spark-3.4-tests, run-spark-4.0-tests or run-iceberg-tests starts the job that label gates without needing a new push. GitHub cannot filter a pull_request trigger by label name, so every label added to a PR starts a run, including labels that gate nothing.

Two rules keep those runs from corrupting the PR's status:

• preflight and changes carry no event guard and run every time. A job held back by if: still publishes a check run under its own name with conclusion skipped, and the newest check run for a name is what the merge box, gh pr checks and required-status-check evaluation read. Guarding preflight on the label name used to let any unrelated label overwrite the commit run's real Preflight verdict with skipped, see #5007.
• On a labeled event, POLICY reports false for every job the new label does not gate. Without that, applying a single label re-ran the entire heavy pipeline at a commit that had already been tested.

run-spark-4.1-tests gates nothing: spark_4_1 already runs on every PR.

Standalone workflows (not under the umbrella)

These workflows have their own triggers because they fire on events the umbrella doesn't watch, or operate independently of the rest of CI:

Reusable workflows (called by ci.yml)

Changing what runs when

Every heavy job in ci.yml is gated on exactly one thing:

if: needs.changes.outputs.spark_3_5 == 'true'

That single boolean folds together two separate decisions, both of which live in dev/ci/compute-changes.py:

• FILTERS — which files the job covers. Pattern semantics match dorny/picomatch (** spans path segments, * stays within one, a leading ! excludes).
• POLICY — which events may run it. "pr" for every pull request, "push" for push to main, "label:<name>" for opt-in on a labelled pull request. "pr" and "label:" are mutually exclusive. workflow_dispatch always runs everything.

So adding a suite, moving sources, or changing when something runs is an edit to one of those two tables, not to ten ${{ }} expressions. Keeping the policy in Python is also what makes it testable: GitHub expressions cannot be exercised outside a real workflow run, whereas POLICY_CASES in dev/ci/check-ci-config.py pins the expected job set for each event shape.

A file that a job reads but that no filter lists is silent: the job skips, and the edit merges with only preflight having looked at it. The shared build inputs (mvnw, .mvn/**, the local composite actions) are pinned by a routing table in dev/ci/check-ci-config.py, which preflight runs.

Artifact names must be unique per producer

Artifact names are scoped to the workflow run, not to the calling workflow. ci.yml calls spark_sql_test_reusable.yml once per Spark version and iceberg_spark_test_reusable.yml once per Iceberg version, all inside the same run, so an unqualified name like native-lib-linux would be claimed by several producers at once. That breaks two things:

• download-artifact resolves a name to the highest matching artifact ID. Nothing ties it to the producer the consumer declared in needs.
• upload-artifact with overwrite: true deletes the newest record with that name before uploading, which can be a sibling's finished artifact. The retry wrapper below forces overwrite on attempts 2 and 3.

So every artifact published by a reusable workflow that ci.yml calls more than once carries its version inputs, e.g. native-lib-spark-4.1.3-jdk17. dev/ci/check-ci-config.py enforces this, and also that every download-artifact name is produced by an upload in the same workflow.

Retrying flaky network operations

Maven. .mvn/maven.config tunes the Maven Resolver HTTP transport: six retries instead of three, 408/429/500/502/503/504 retryable instead of only 429/503, a 30s connect timeout and a 10 minute socket read timeout. The wrapper pins maven.multiModuleProjectDirectory to the directory holding .mvn, so one file covers every mvnw invocation in CI (including cd spark && ../mvnw ...) with no per-workflow wiring.

The Wagon transport (-Dmaven.resolver.transport=wagon, maven.wagon.http.*) was evaluated and rejected: it is deprecated in Resolver 1.9 and removed in Maven 4, its retry knobs mirror the native transport‘s, and its service-unavailable retry strategy defaults to none, so adopting it would first have to buy back the 429/503 retry we already get. The one thing it can still do that the native transport cannot is shrink HttpClient’s non-retryable exception list (retryHandler.class=default plus retryHandler.nonRetryableClasses=...), the only way to retry a connect or read timeout. We size those timeouts not to fire instead.

Artifact upload. actions/upload-artifact fails the job when FinalizeArtifact returns (403) Forbidden: Error from intermediary, even though the content already uploaded. Its client only retries 429/500/502/503/504, exposes no input to widen that, and Actions has no built-in step retry. Use ./.github/actions/upload-artifact-retry instead for any artifact a later job consumes: same inputs and outputs, three attempts, 15s then 45s backoff. Attempts 2 and 3 force overwrite: true, so the name must belong to exactly one producer in the run (see above). The diagnostic uploads inside ./.github/actions/java-test stay on the plain action, since a local action calling another local action is untested here and those run only on already-failing jobs.

Maven wrapper bootstrap. ./.github/actions/java-test retries ./mvnw --version with exponential backoff, so a failed download of the Maven distribution does not surface as a test failure.

Branch protection

Required-check names changed when these workflows were consolidated. The umbrella exposes per-job names like CI / pr_build_linux / Lint, CI / spark_3_5 / linux-test (...), etc. Update repository branch protection rules to point at the new names; the old standalone workflow names (Spark SQL Tests (Spark 3.5), PR Build (Linux), ...) no longer exist as top-level workflows.

.asf.yaml currently declares no required_status_checks for main, only required_approving_review_count: 1. Anything added there must be a name that never legitimately reports skipped, because GitHub counts a skipped check as passing. The bare caller-job names (PR Build (Linux), Spark SQL Tests (Spark 3.5), ...) are not such names: a reusable workflow that actually runs publishes only its child jobs, so the bare name shows up solely when the caller was skipped.