tree: 2745ba533c09b8d699750c517aaf698a6075dfe9 [search]
  1. images/
  2. benchmark.yml
  3. branch35_scheduler.yml
  4. branch40_scheduler.yml
  5. branch41_scheduler.yml
  6. branch42_scheduler.yml
  7. branch43_scheduler.yml
  8. branch44_scheduler.yml
  9. branch4x_scheduler.yml
  10. build_and_test.yml
  11. build_codegen_jdk.yml
  12. build_coverage.yml
  13. build_infra_images_cache.yml
  14. build_java17.yml
  15. build_java21.yml
  16. build_java25.yml
  17. build_main.yml
  18. build_maven.yml
  19. build_maven_java21.yml
  20. build_maven_java21_arm.yml
  21. build_maven_java21_macos26.yml
  22. build_maven_java25.yml
  23. build_non_ansi.yml
  24. build_python_3.11.yml
  25. build_python_3.12_arm.yml
  26. build_python_3.12_classic_only.yml
  27. build_python_3.12_macos26.yml
  28. build_python_3.12_pandas_3.yml
  29. build_python_3.13.yml
  30. build_python_3.14.yml
  31. build_python_3.14_nogil.yml
  32. build_python_3.15.yml
  33. build_python_3.9.yml
  34. build_python_connect.yml
  35. build_python_connect40.yml
  36. build_python_minimum.yml
  37. build_python_pypy3.10.yml
  38. build_rockdb_as_ui_backend.yml
  39. build_scala213.yml
  40. build_sparkr_window.yml
  41. build_uds.yml
  42. labeler.yml
  43. maven_test.yml
  44. pages.yml
  45. publish_snapshot.yml
  46. python_hosted_runner_test.yml
  47. README.md
  48. release.yml
  49. stale.yml
  50. test_report.yml
  51. update_build_status.js
  52. update_build_status.yml
.github/workflows/README.md

Continuous Integration on GitHub Actions

The continuous integration build is critical to the project's health and development velocity. It enables contributors to make changes to Spark with the confidence that they have not inadvertently broken anything.

There are several considerations to keep in mind when making changes to the build.

Resource usage limits

The Apache Software Foundation (ASF) imposes resource usage limits on GitHub Actions, including on the number of concurrent running jobs as well as on the total number of run minutes used per week and rolling five-day window. When we hit these limits it can cause jobs to queue up, bringing development to a crawl.

Scheduled workflow cadence

Use daily maintenance schedules and three-day or weekly build schedules to limit runner consumption while retaining regular coverage. Reserve 04 and 12 UTC for daily workflows and 03, 09, 15, and 21 UTC for weekly workflows; distribute three-day workflows across the remaining hours and day-of-month phases, and spread weekly workflows across the days of the week. Balance expensive builds across phases and start times, and use minute 17 for rescheduled builds to avoid GitHub Actions' busy start-of-hour scheduling period; the specific minute is arbitrary. The build-status updater keeps its separate 15-minute schedule.

These assignments spread runner demand without ranking jobs by importance, and individual intervals can be shortened when faster feedback is needed. Day-of-month phases restart monthly, so intervals vary at month boundaries, and delayed starts or long runs can still overlap.

CI dashboard

See the Apache CI dashboard for an overview of Spark's CI usage.

Push heavy workflows out to forks

Workflows triggered by on: pull_request count against the base repo‘s (i.e. this repo’s) activity limits. For this reason it‘s important to use on: push for heavy workflows. on: push workflows count against the fork’s limits and not ours, enabling more jobs to run concurrently.

Downsides of avoiding the pull_request workflow trigger

The pull_request workflow trigger provides key benefits that make working with workflows easier, including:

  1. The workflow runs against the target repo's context (i.e. apache/spark).
  2. GitHub provides a stable merge reference.
  3. PR metadata is readily available via github.event.pull_request.

Without these benefits, some GitHub Actions patterns become more difficult or impossible to implement. We've implemented our own custom checkout pattern via the checkout-and-sync composite action.

If in the future GitHub somehow allows workflows to run on: pull_request while counting those jobs against the forks' limits instead of ours, that would help eliminate various distortions of our build caused by the move away from on: pull_request. We would get the best functionality from GitHub Actions without hitting job run limits.

Composite actions

If you find yourself repeating the same set of instructions across multiple jobs, extract them into a reusable composite action under actions/.

Security considerations

Keep these security considerations in mind when authoring workflows, as it's possible for malicious forks to use poorly written workflows to steal secrets or merge unapproved code.