| <!-- |
| Licensed to the Apache Software Foundation (ASF) under one |
| or more contributor license agreements. See the NOTICE file |
| distributed with this work for additional information |
| regarding copyright ownership. The ASF licenses this file |
| to you under the Apache License, Version 2.0 (the |
| "License"); you may not use this file except in compliance |
| with the License. You may obtain a copy of the License at |
| |
| http://www.apache.org/licenses/LICENSE-2.0 |
| |
| Unless required by applicable law or agreed to in writing, |
| software distributed under the License is distributed on an |
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| KIND, either express or implied. See the License for the |
| specific language governing permissions and limitations |
| under the License. |
| --> |
| # ASF GitHub Actions Repository |
| |
| This repository hosts GitHub Actions developed by the ASF community and approved for any ASF top level project to use. It also manages the organization wide allow list of GitHub Actions via 'Configuration as Code'. |
| |
| - [Checking the Action Usage in an ASF Project](#checking-the-action-usage-in-an-asf-project) |
| - [Submitting an Action](#submitting-an-action) |
| - [Available GitHub Actions](#available-github-actions) |
| - [Versioning and Pinning Actions](#versioning-and-pinning-actions) |
| - [Organization-wide GitHub Actions Allow List](#management-of-organization-wide-github-actions-allow-list) |
| - [Pipeline Overview](#pipeline-overview) |
| - [Adding a New Action](#adding-a-new-action-to-the-allow-list) |
| - [Reviewing](#reviewing) |
| - [Updating Version of Already Approved Action](#updating-version-of-already-approved-action) |
| - [Automated Verification in CI](#automated-verification-in-ci) |
| - [Dependabot Cooldown Period](#dependabot-cooldown-period) |
| - [Manual Version Addition](#manual-addition-of-specific-versions) |
| - [Automatic Expiration of Old Versions](#automatic-expiration-of-old-versions) |
| - [Removing a Version](#removing-a-version-manually) |
| - [Auditing Repositories for Actions Security Tooling](#auditing-repositories-for-actions-security-tooling) |
| - [Snapshotting Queued and Running Actions Jobs](#snapshotting-queued-and-running-actions-jobs) |
| |
| ## Checking the Action Usage in an ASF Project |
| |
| You can let your CI workflows check if the Actions used in your project are approved for use in the ASF. |
| |
| An example workflow that can be used as a template for your project's CI can be found |
| [here `allowlist-check/README.md`](allowlist-check/README.md). |
| It is usually enough to add the following job to an existing `.github/workflows/ci.yml` file: |
| ```yaml |
| jobs: |
| asf-allowlist-check: |
| runs-on: ubuntu-latest |
| steps: |
| - uses: actions/checkout@v4 |
| with: |
| persist-credentials: false |
| - uses: apache/infrastructure-actions/allowlist-check@main |
| ``` |
| |
| When calling the `check-project-actions` workflow from a `push` or `pull_request` event, it should work |
| automatically against the "right" reference. See the sample workflow linked above for more details. |
| |
| To pin to an immutable, Dependabot-trackable version instead of `@main`, see |
| [Versioning and Pinning Actions](#versioning-and-pinning-actions). |
| |
| ## Submitting an Action |
| |
| To contribute a GitHub Action to this repository: |
| |
| 1. **Fork** this repository |
| 2. **Add your action code**: |
| - Create a subdirectory for your proposed GHA at the root level (e.g., `/MyNewAction`) |
| - Add all required files for your action in this subdirectory |
| - Include a comprehensive README.md that explains: |
| - What the action does |
| - Required inputs and available outputs |
| - Example usage configurations |
| - Any special considerations or limitations |
| 3. **Create a pull request** to merge your branch into the main branch |
| |
| ## Reviewing |
| |
| The Infrastructure team will review each proposed Action based on: |
| - Overall usefulness to the ASF community |
| - Maintenance complexity |
| - Security considerations |
| - Code quality |
| |
| Once approved, the Infrastructure team will merge the pull request and add the new Action to the list of available Actions for all ASF projects. |
| |
| We highly appreciate contributed reviews, especially from people |
| associated with the projects that (would like to) use a particular |
| action, even if they're not committers on _this_ project: you're |
| especially qualified to judge and vouch for the safety and |
| correctness of the action. |
| |
| ## Available GitHub Actions |
| |
| - [ASF Infrastructure Pelican Action](/pelican/README.md): Generate and publish project websites with GitHub Actions |
| - [Stash Action](/stash/README.md): Manage large build caches |
| - [ASF Allowlist Check](/allowlist-check/README.md): Verify workflow action refs are on the ASF allowlist |
| |
| ## Versioning and Pinning Actions |
| |
| The actions in this repo are a *monorepo of actions*, and each one is released |
| under its own **path-prefixed tag** so you can pin a specific version and let |
| Dependabot propose bumps. The tag prefix is the action's leaf directory name, |
| which you repeat after the `@`: |
| |
| | Action | Pin it like this | |
| | --------------------- | ----------------------------------------------------------------------------- | |
| | `allowlist-check` | `apache/infrastructure-actions/allowlist-check@<sha> # allowlist-check/v1.2.3`| |
| | `pelican` | `apache/infrastructure-actions/pelican@<sha> # pelican/v1.2.3` | |
| | `stash/save` | `apache/infrastructure-actions/stash/save@<sha> # save/v1.2.3` | |
| | `stash/restore` | `apache/infrastructure-actions/stash/restore@<sha> # restore/v1.2.3` | |
| |
| Pinning to a commit SHA with the version in a trailing comment is the |
| recommended, [Zizmor](https://zizmor.sh/)-friendly form: the SHA is immutable, |
| and Dependabot's `github_actions` ecosystem recognises the `# <prefix>/vX.Y.Z` |
| comment and opens a PR (updating both the SHA and the comment) when a newer |
| tag for that prefix is published. Support for this monorepo leaf-prefix scheme |
| was added to Dependabot in |
| [dependabot/dependabot-core#11286](https://github.com/dependabot/dependabot-core/pull/11286), |
| contributed specifically for this repository. |
| |
| A release is the tag pair (`<prefix>/vX.Y.Z` plus the moving `<prefix>/vN`) and |
| nothing more — this repo publishes no GitHub Release objects, because |
| Dependabot resolves versions from the tags themselves. |
| |
| Tracking `@main` (as in the quick-start above) also works and always gives you |
| the latest code, but it drifts from any pinned SHA and Zizmor will flag the |
| unpinned ref. See [RELEASING.md](RELEASING.md) for how releases are cut. |
| |
| ## Management of Organization-wide GitHub Actions Allow List |
| |
| As stated in the [ASF GitHub Actions Policy](https://infra.apache.org/github-actions-policy.html), GitHub Actions from external sources are blocked by default in all `apache/*` repositories. Only actions from the following namespaces are automatically allowed: |
| - `apache/*` |
| - `github/*` |
| - `actions/*` |
| |
| All other actions must be explicitly added to the allow list after undergoing a security review. This review process applies to both new actions and new versions of previously approved actions (though reviews for new versions are typically expedited). |
| |
| `actions.yml` is the source of truth for approved actions. From it, two generated files are kept in sync automatically: `approved_patterns.yml` (consumed by the ASF org-wide allow list) and `.github/actions/for-dependabot-triggered-reviews/action.yml` (the composite action Dependabot watches, so it can propose version bumps). The sections below describe the two entry points — manual PRs to add a new action, and the Dependabot-driven flow for updating versions of already-approved actions — and the workflows that implement each. |
| |
| #### Pipeline Overview |
| |
| The diagram below summarizes every entry point, workflow and generated file involved in keeping the allow list in shape. Each subsequent section zooms in on one slice of this flow. |
| |
| ```mermaid |
| graph LR |
| human["Human PR<br/>(add action / older version /<br/>urgent removal)"] |
| dependabot["Dependabot PR<br/>(version bump)"] |
| cron["Daily 02:04 UTC"] |
| |
| actions["actions.yml<br/><i>source of truth</i>"] |
| composite[".github/actions/<br/>for-dependabot-triggered-reviews/<br/>action.yml"] |
| approved["approved_patterns.yml<br/><i>ASF org allow list</i>"] |
| |
| human-->actions |
| dependabot-->composite |
| dependabot-.verified by.-verify["<b>verify</b> job<br/>(rebuild & diff)"] |
| |
| composite=="<b>update</b> job<br/>(on merge)"==>actions |
| cron=="<b>remove_expired</b> job"==>actions |
| |
| actions=="<b>update</b> job"==>composite |
| actions=="<b>update</b> job<br/>(cap check + regen)"==>approved |
| |
| classDef source fill:#fff3b0,stroke:#8a6d0b,color:#333 |
| classDef generated fill:#e0f0ff,stroke:#2563a6,color:#333 |
| classDef trigger fill:#f3e0ff,stroke:#6a1b9a,color:#333 |
| classDef job fill:#e6ffe6,stroke:#1b5e20,color:#333 |
| class actions source |
| class composite,approved generated |
| class human,dependabot,cron trigger |
| class verify job |
| ``` |
| |
| Solid arrows (`==>`) are regeneration edges — the "source → generated" flows that keep `actions.yml`, `approved_patterns.yml` and the dependabot composite in sync. Thin arrows feed the pipeline with new content (human or Dependabot PRs, cron), and dotted arrows are observer jobs that verify rather than mutate. Bold labels are job names (rather than workflow filenames) — `update` lives in `update.yml`, `verify` in `verify_dependabot_action.yml` / `verify_manual_action.yml`, `remove_expired` in `remove_expired.yml`. |
| |
| > [!NOTE] |
| > The 800/1000-entry cap on `approved_patterns.yml` is enforced as a step inside the `update` job. It runs after regeneration and before commit/push, so an over-cap state never lands on `main`. The push uses the `ALLOWLIST_WORKFLOW_TOKEN` PAT because `main` is a protected branch and the default `GITHUB_TOKEN` is blocked by branch protection (`GH006`); the PAT has bypass rights for this workflow's automated commit. |
| |
| ### Adding a New Action to the Allow List |
| |
| ```mermaid |
| graph TD; |
| manual["manual PR"]--new entry-->actions.yml |
| actions.yml--"<b>update</b> job"-->composite[".github/actions/for-dependabot-triggered-reviews/action.yml"] |
| actions.yml--"<b>update</b> job"-->approved["approved_patterns.yml"] |
| ``` |
| |
| A human-authored PR edits `actions.yml` directly. Once it merges to `main`, the **`update`** job (in `update.yml`) regenerates both `.github/actions/for-dependabot-triggered-reviews/action.yml` and `approved_patterns.yml` from the new entries, so contributors never have to hand-edit the generated files. |
| |
| To request addition of an action to the allow list: |
| |
| 1. **Fork** this repository |
| 2. **Add** an entry to `actions.yml` using the following format: |
| |
| ```yaml |
| repo/owner: |
| '<exact-commit-sha>': |
| tag: vX.Y.Z |
| ``` |
| |
| 3. **Create a PR** against the `main` branch |
| 4. **Include in your PR description**: |
| - Why this action is needed for your project |
| - Any alternatives you've considered |
| - Any security concerns you've identified |
| |
| 5. **Wait for review** by the infrastructure team |
| |
| > [!NOTE] |
| > Always pin actions to exact commit SHAs, never use tags or branch references. |
| |
| The infrastructure team will review your request and either approve, request changes, or provide feedback on alternatives. |
| |
| ### Updating version of already approved action |
| |
| ```mermaid |
| graph TD; |
| dependabot--"PR updates"-->composite[".github/actions/for-dependabot-triggered-reviews/action.yml"] |
| dependabot-.verified by.-verify["<b>verify</b> job"] |
| composite--"<b>update</b> job (on merge)"-->actions.yml |
| actions.yml--"<b>update</b> job"-->approved["approved_patterns.yml"] |
| ``` |
| |
| In most cases, new versions are automatically added through Dependabot: |
| - Dependabot opens PRs against `.github/actions/for-dependabot-triggered-reviews/action.yml` to update actions to the newest releases |
| - The **`verify`** job (in `verify_dependabot_action.yml`) runs on each such PR, rebuilds the action's compiled JavaScript in Docker, and diffs it against the published version (see [Automated Verification in CI](#automated-verification-in-ci)) |
| - Once a reviewer merges the PR, the **`update`** job (in `update.yml`) reflects the new commit SHAs back into `actions.yml`, regenerates `approved_patterns.yml`, and enforces the 800/1000-entry cap inline before pushing |
| - The previously approved version is marked with an `expires_at` date 3 months out, giving projects a grace period to update their workflows; see [Automatic Expiration of Old Versions](#automatic-expiration-of-old-versions) for how the cleanup runs |
| |
| Projects are encouraged to help review updates to actions they use. Please have a look at the diff and mention in your approval what you have checked and why you think the action is safe. |
| |
| #### Verifying Compiled JavaScript |
| |
| Many GitHub Actions ship pre-compiled JavaScript in their `dist/` directory. To verify that the published compiled JS matches a clean rebuild from source, use the verification script: |
| |
| ```bash |
| uv run utils/verify-action-build.py org/repo@commit_hash |
| ``` |
| |
| For example: |
| |
| ```bash |
| uv run utils/verify-action-build.py dorny/test-reporter@dc3a92680fcc15842eef52e8c4606ea7ce6bd3f3 |
| ``` |
| |
| The script will: |
| 1. Clone the action at the specified commit inside an isolated Docker container |
| 2. Save the original `dist/` files as published in the repository |
| 3. Rebuild the action from source, picking the right toolchain automatically — Node.js (`npm ci && npm run build`, `yarn`, or `pnpm`), Dart (`dart compile js` when a `pubspec.yaml` is present), or Deno (`deno task bundle` when a `deno.json`/`deno.jsonc` is present) |
| 4. Reformat both versions of the JavaScript for readable comparison |
| 5. Show a colored diff of any differences |
| |
| A clean result confirms that the compiled JS was built from the declared source. Any differences will be flagged for manual inspection. |
| |
| Non-minified compiled JS (e.g. Deno `deno task bundle` output, Dart `dart compile js` readable output) is handled differently: a clean rebuild for these tends to produce toolchain-version noise (esbuild/ncc/webpack boilerplate differences) rather than actionable diffs. The script keeps these files in place during the pre-rebuild deletion step and instead diffs them against the previously approved version of the action, so reviewers see real source changes rather than rebuild artifacts. The detection threshold mirrors the comparison heuristic — fewer than 10 lines or an average line length above 500 chars is treated as minified. |
| |
| Files that appear **only in the rebuild** are reported as informational rather than as a failure. The action does not publish them, so they never reach a consumer's runner and cannot be a supply-chain vector. In practice they are intermediate build output from a multi-stage build that upstream deliberately does not commit — for example `JetBrains/qodana-action` declares `main: scan/dist/index.js`, so the output directory resolves to the whole `scan/` sub-project and the rebuild's gitignored `scan/lib/*.js` (stage one of its `tsc` → `esbuild` build) lands inside the compared tree. The inverse remains a hard failure: JS present in the published tree but *absent* from the rebuild is unaccounted-for shipped code, as is a published tree with no compiled JS at all when the rebuild produced some — there is then nothing to reconcile the rebuild against. |
| |
| The **source diff vs approved** and the **Script analysis** check both cover more than the language the entrypoint is written in. A node action is free to shell out to a script committed beside it — `uraimo/run-on-arch-action` declares `main: src/run-on-arch.js`, which then `exec()`s `src/run-on-arch.sh` — so shell/interpreter scripts (`.sh`, `.bash`, `.ps1`, `.py`, `.rb`, `.pl`) and `Dockerfile*` are diffed alongside the JS/TS sources, and committed shell scripts are discovered from the repo tree rather than only from the files `action.yml` or a `Dockerfile` happen to name. Script analysis runs for every action type; for JavaScript actions its findings are reported in the summary but do not change the pass/fail verdict. |
| |
| #### Security Review Checklist |
| |
| When reviewing an action (new or updated), watch for these potential issues in the source diff between the approved and new versions: |
| |
| - **Credential exfiltration**: code that reads secrets, tokens, or environment variables (e.g. `GITHUB_TOKEN`, `AWS_*`, `ACTIONS_RUNTIME_TOKEN`) and sends them to external endpoints via `fetch`, `http`, `net`, or shell commands (`curl`, `wget`). |
| - **Arbitrary code execution**: use of `eval()`, `new Function()`, `child_process.exec/spawn` with unsanitised inputs, or downloading and running scripts from remote URLs at build or runtime. |
| - **Unexpected network calls**: outbound requests to domains unrelated to the action's stated purpose, especially in `post` or cleanup steps that run after the main action. |
| - **Workflow permission escalation**: actions that request or rely on elevated permissions (`contents: write`, `id-token: write`, `packages: write`) beyond what their functionality requires. |
| - **Supply-chain risks**: new or changed dependencies in `package.json` that are unpopular, recently published, or have been involved in known compromises; mismatches between `package-lock.json` and `package.json`. |
| - **Obfuscated code**: hex-encoded strings, base64 blobs, or intentionally unreadable code in source files (not in compiled `dist/`). |
| - **File-system tampering**: writing to locations outside the workspace (`$GITHUB_WORKSPACE`), modifying `$GITHUB_ENV`, `$GITHUB_PATH`, or `$GITHUB_OUTPUT` in unexpected ways to influence subsequent workflow steps. |
| - **Compiled JS mismatch**: any unexplained diff between the published `dist/` and a clean rebuild — this is the primary check the verification script performs. |
| - **Pre-compiled native binaries shipped in-tree**: actions that commit Go/Rust/C-style binaries (`main-linux-amd64`, `*.exe`, `*.dll`, `*.so`, `*.dylib`, `*.jar`, `*.wasm`, etc.) directly in the repo and exec them from a small launcher are running opaque executable code on the runner. The JS-rebuild check verifies the launcher but **cannot** reconcile the binaries with source on its own. `verify-action-build`'s **In-tree binary check** tries to close the gap automatically: each detected binary is verified first by the clean rebuild (binaries a bundler copies into the output directory — `.wasm`, `.node`, native libraries — are deleted before the rebuild along with the minified JS, so one that comes back byte-identical was regenerated from the lockfile-pinned dependency tree and needs no release provenance of its own; `1Password/load-secrets-action` ships `dist/core_bg.wasm` this way, copied by `ncc` out of `@1password/sdk-core`), then via `gh attestation verify --owner <org>` (the SLSA attestation transparency log populated by [`actions/attest-build-provenance`](https://github.com/actions/attest-build-provenance)), then by SHA256-comparing each binary against the release's `SHA256SUMS` asset. Binaries that pass any of the three are ✓; binaries that pass none are a hard reject. Push back on actions in this shape until upstream adds attestation or `SHA256SUMS` so the chain from release to artifact can be verified. |
| - **Runtime binary downloads without an in-source checksum**: some actions pull their tool binary at runtime via `tc.downloadTool` / `curl` / `fetch` and rely on the publishing pipeline (GitHub release immutability + Sigstore attestation) for integrity rather than an inline `sha256sum -c` / `cosign verify-blob`. The **Binary Download Verification** check fails these by default. A per-action escape hatch lives in `utils/verify_action_build/security.py` as the `TRUSTED_DOWNLOAD_PROVENANCE` dict — an entry asserts that the configured `release_repo` publishes immutable releases AND emits Sigstore attestations via `actions/attest-build-provenance`. Adding an entry is a security review decision and the rationale must link the upstream confirmation (e.g. a maintainer comment). The config alone is not enough: at scan time the verify pipeline GETs `releases/latest` of the configured `release_repo`, confirms `release.immutable` is true, downloads one small attested asset (`.sbom.json` preferred), and runs `gh attestation verify` against it. Only when both halves pass are the action's unverified-download findings reclassified as warnings; if the runtime check fails, failures stay failures and the reason is printed. Note the scope: the spot-check proves the *release repo's pipeline* attests and that its latest release is immutable — it does not machine-verify that the action downloads from that `release_repo`, nor that the *specific version* it fetches is itself immutable (only `releases/latest` is checked). That binding remains the reviewer's call, backed by the entry's `rationale`. |
| |
| For the full approval policy and requirements, see the [ASF GitHub Actions Policy](https://infra.apache.org/github-actions-policy.html). |
| |
| #### Batch-Reviewing Dependabot PRs |
| |
| To review all open dependabot PRs at once, run: |
| |
| ```bash |
| uv run utils/verify-action-build.py --check-dependabot-prs |
| ``` |
| |
| This will: |
| 1. List all open PRs from dependabot |
| 2. For each PR, extract the action reference from the diff |
| 3. Run the full build verification (rebuild in Docker, compare compiled JS) |
| 4. Show source changes between the previously approved version and the new one |
| 5. If verification passes, ask whether to approve and merge the PR |
| 6. On merge, add a review comment documenting what was verified |
| |
| #### Running Without the `gh` CLI |
| |
| If you prefer not to install the `gh` CLI, you can use `--no-gh` to make all GitHub API calls via Python `requests` instead. In this mode you must provide a GitHub token either via `--github-token` or the `GITHUB_TOKEN` environment variable: |
| |
| ```bash |
| # Using the flag: |
| uv run utils/verify-action-build.py --no-gh --github-token ghp_... org/repo@commit_hash |
| |
| # Or via environment variable: |
| export GITHUB_TOKEN=ghp_... |
| uv run utils/verify-action-build.py --no-gh --check-dependabot-prs |
| ``` |
| |
| The `--no-gh` mode supports all the same features as the default `gh`-based mode. |
| |
| #### Automated Verification in CI |
| |
| Two workflows in `.github/workflows/` run `verify-action-build` on PRs that touch the allow list, so the verification status is visible on every PR as a required-candidate status check: |
| |
| - **`verify` job in `verify_dependabot_action.yml`** — triggers on Dependabot PRs that modify `.github/actions/for-dependabot-triggered-reviews/action.yml`. Extracts the action reference from the PR, rebuilds the compiled JavaScript in Docker, and compares it against the published version. |
| - **`verify` job in `verify_manual_action.yml`** — triggers on human-authored PRs that modify `actions.yml` or `approved_patterns.yml` (i.e. manual allow-list additions / version bumps). Dependabot-authored PRs are skipped, since they are already covered by the workflow above. |
| - **`check_action_tags` job in `check_action_tags.yml`** — triggers when `actions.yml`, `approved_patterns.yml`, the generated Dependabot composite action, the update workflow, or gateway verification code changes. It verifies that configured action SHAs exist and, when a `tag` is recorded, that the SHA is reachable from that Git tag or branch. |
| |
| These workflows use regular `pull_request` triggers with read-only permissions and no PR comments — pass/fail is surfaced through the status check. They do not auto-approve or merge; a human reviewer must still approve. |
| |
| The script exits with code **1** (failure) when something is unexpectedly broken — for example, the action cannot be compiled, the rebuilt JavaScript is invalid, or required tools are missing. In all other cases it exits with code **0** and produces reviewable diffs: a large diff does not by itself cause an error (e.g. major version bumps will naturally have big diffs). It is always up to a human reviewer to inspect the output, assess the changes, and decide whether the update is safe to approve. |
| |
| To verify a specific PR locally (non-interactively), use: |
| |
| ```bash |
| uv run utils/verify-action-build.py --ci --from-pr 123 |
| ``` |
| |
| The `--ci` flag skips all interactive prompts (auto-selects the newest approved version for diffing, auto-accepts exclusions, disables paging). The `--from-pr` flag extracts the action reference from the given PR number. |
| |
| Additional flags: |
| - `--no-cache` — rebuild the Docker image from scratch without using the layer cache. |
| - `--show-build-steps` — display a summary of Docker build steps on successful builds (the summary is always shown on failure). |
| |
| > [!NOTE] |
| > **Prerequisites:** `docker` and `uv`. When using the default mode (without `--no-gh`), `gh` (GitHub CLI, authenticated via `gh auth login`) is also required. The build runs in a `node:20-slim` container so no local Node.js installation is needed. |
| |
| #### Dependabot Cooldown Period |
| |
| This repository uses a [Dependabot cooldown period](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#cooldown) of 0 days so that maintainers can review before Dependabot opens a PR on project repositories. |
| |
| > [!TIP] |
| > We recommend that ASF projects configure a cooldown in their own `dependabot.yml` to avoid being overwhelmed by update PRs and to catch up with approved actions here: |
| > ```yaml |
| > updates: |
| > - package-ecosystem: "github-actions" |
| > directory: "/" |
| > schedule: |
| > interval: "weekly" |
| > cooldown: |
| > default: 4 |
| > ``` |
| > Adjust the `default` value (in days) to match your project's review capacity. |
| |
| ### Manual Addition of Specific Versions |
| |
| If you need to add a specific version of an already approved action (especially an older one): |
| |
| 1. **Fork** this repository |
| 2. **Add** a new version entry to an existing action in `actions.yml`. Choose its metadata based on |
| why the version is needed. |
| |
| For the newest version: |
| |
| ```yaml |
| existing/action: |
| '<exact-commit-sha>': |
| tag: vX.Y.Z |
| ``` |
| |
| The current version must have neither `keep` nor `expires_at`, so that it is included in the |
| composite action watched by Dependabot. Each action must have at most one such version. When adding |
| a new current version manually, add `expires_at: <date>` to the previous current version to give |
| projects time to migrate. |
| |
| For an older version that is needed temporarily: |
| |
| ```yaml |
| existing/action: |
| '<exact-commit-sha>': |
| expires_at: 2025-01-01 |
| tag: vX.Y.Z |
| ``` |
| |
| Use `keep: true` only as an exceptional alternative when an older version must remain available |
| indefinitely: |
| |
| ```yaml |
| existing/action: |
| '<exact-commit-sha>': |
| # Explain why this version must remain available indefinitely. |
| keep: true |
| tag: vX.Y.Z |
| ``` |
| |
| A reference with `keep: true` is retained indefinitely and is not watched for updates by |
| Dependabot. To keep the action updated, it must also have a separate current version with neither |
| `keep` nor `expires_at`. Never set both `keep` and `expires_at` on the same reference. |
| |
| |
| 3. **Create a PR** against the `main` branch |
| 4. **Include in your PR description**: |
| - Specific reason why this version is required |
| - Any blockers preventing upgrade to newer versions |
| - Risk assessment for using an older version |
| - Expected timeline for migration to newer versions (if applicable) |
| |
| > [!WARNING] |
| > Older versions may contain security vulnerabilities or performance issues. Always evaluate if using the latest version is possible before requesting older versions. |
| |
| ### Automatic Expiration of Old Versions |
| |
| ```mermaid |
| graph TD; |
| entry["actions.yml entry<br/>with expires_at"]--"<b>remove_expired</b> job (daily, 02:04 UTC)"-->actions.yml |
| actions.yml--"<b>update</b> job"-->composite[".github/actions/for-dependabot-triggered-reviews/action.yml"] |
| actions.yml--"<b>update</b> job"-->approved["approved_patterns.yml"] |
| ``` |
| |
| Routine cleanup of superseded versions is automated: |
| |
| - Any entry in `actions.yml` with an `expires_at: YYYY-MM-DD` field is a candidate for removal. |
| - Dependabot-driven updates (see [Updating Version of Already Approved Action](#updating-version-of-already-approved-action)) set `expires_at` to **3 months out** on the previously approved version. For manually added older versions, set `expires_at` explicitly (see [Manual Addition of Specific Versions](#manual-addition-of-specific-versions)). |
| - The **`remove_expired`** job (in `remove_expired.yml`) runs daily at **02:04 UTC**. Every entry whose `expires_at` date has passed is deleted from `actions.yml`; the job then commits the change and lets the `update` job in `update.yml` regenerate `approved_patterns.yml` and the dependabot composite. |
| - Entries without `expires_at` (for example, `keep: true` wildcards and the current approved version) are never auto-removed — removal of those requires a manual PR. |
| |
| No human action is required for the routine case: projects get a 3-month grace window after a version bump, and the old entry disappears on its own afterwards. |
| |
| ### Removing a version manually |
| |
| Routine removal is already automated: set `expires_at` on the entry and the daily `remove_expired` job (in `remove_expired.yml`) will delete it once the date passes. Use the manual process below only when you need an immediate removal that can't wait for the entry to expire. |
| |
| > [!IMPORTANT] |
| > If a version or entire action needs to be removed immediately due to a security vulnerability: |
| |
| 1. **Fork** this repository |
| 2. **Remove** the relevant entry from `actions.yml` |
| 3. **Create a PR** against the `main` branch |
| 4. **Mark it as urgent** in the PR title (e.g., "URGENT: Remove vulnerable action X") |
| 5. **Include in your PR description**: |
| - The reason for removal |
| - Any CVE or security advisory ID if applicable |
| - Impact on projects currently using the action |
| - Recommended alternatives if available |
| |
| The infrastructure team will prioritize these removal requests and may take additional steps to notify affected projects if necessary. |
| |
| For 'regular' removals (not security responses), you can use `./utils/action-usage.sh someorg/theaction` to see if/how an action is still used anywhere in the ASF, and create a 'regular' PR removing it from `actions.yml` (or adding an expiration date) when it is no longer used. |
| |
| ## Auditing Repositories for Actions Security Tooling |
| |
| Recent security breaches have shown that GitHub Actions can fail silently, leaving repositories vulnerable without any visible indication. The `actions-audit.py` script helps ensure that all Apache repositories using GitHub Actions have a baseline set of security tooling in place. |
| |
| ### Why This Matters |
| |
| GitHub Actions workflows can introduce security risks in several ways: |
| - **Unpinned or unreviewed action versions** may contain malicious code or vulnerabilities |
| - **Missing static analysis** means workflow misconfigurations (secret exposure, injection vulnerabilities) go undetected |
| - **No dependabot** means action versions never get updated, accumulating known vulnerabilities over time |
| |
| The audit script checks each repository for four security configurations and can automatically open PRs to add any that are missing: |
| |
| | Check | What it does | |
| |-------|-------------| |
| | **Dependabot** | Keeps GitHub Actions dependencies up to date with a 4-day cooldown to avoid overwhelming reviewers | |
| | **CodeQL** | Runs static analysis on workflow files to detect security issues in Actions syntax | |
| | **Zizmor** | Specialized scanner for GitHub Actions anti-patterns: credential leaks, injection vulnerabilities, excessive permissions | |
| | **ASF Allowlist Check** | Ensures every action used is on the ASF Infrastructure approved allowlist | |
| |
| ### Prerequisites |
| |
| - **Python 3.11+** and [**uv**](https://docs.astral.sh/uv/) **>= 0.9.17** (dependencies are managed inline via PEP 723). Make sure your uv is up to date — depending on how you installed it, run `uv self update`, `pip install --upgrade uv`, `pipx upgrade uv`, or `brew upgrade uv` |
| - **`gh`** (GitHub CLI, authenticated via `gh auth login`) — or provide a `--github-token` with `repo` scope and use `--no-gh` |
| - **`zizmor`** ([install instructions](https://docs.zizmor.dev/installation/)) — required for PR creation mode; not needed for `--dry-run`. If missing, zizmor pre-checks are skipped with a warning |
| |
| ### Usage |
| |
| Always start with `--dry-run` to see what the script would do without making any changes: |
| |
| ```bash |
| # Audit all repos for a specific PMC (prefix before first '-' in repo name) |
| uv run utils/actions-audit.py --dry-run --pmc spark --max-num 10 |
| |
| # Audit multiple PMCs |
| uv run utils/actions-audit.py --dry-run --pmc kafka --pmc flink |
| |
| # Audit the first 50 repos (no PMC filter) |
| uv run utils/actions-audit.py --dry-run --max-num 50 |
| |
| # Increase GraphQL page size for fewer API round-trips |
| uv run utils/actions-audit.py --dry-run --max-num 200 --batch-size 100 |
| ``` |
| |
| When satisfied with the dry-run output, remove `--dry-run` to create PRs: |
| |
| ```bash |
| # Create PRs for spark repos missing security tooling |
| uv run utils/actions-audit.py --pmc spark --max-num 10 |
| ``` |
| |
| #### Options |
| |
| | Flag | Description | |
| |------|-------------| |
| | `--pmc PMC` | Filter by PMC prefix (repeatable). The prefix is the text before the first `-` in the repo name, e.g. `spark` matches `spark`, `spark-connect-go`, `spark-docker`. | |
| | `--dry-run` | Report findings without creating PRs or branches. | |
| | `--max-num N` | Maximum number of repositories to check (0 = unlimited, default). | |
| | `--batch-size N` | Number of repos to fetch per GraphQL request (default: 50, max: 100). | |
| | `--github-token TOKEN` | GitHub token. Defaults to `GH_TOKEN` or `GITHUB_TOKEN` environment variable. | |
| | `--no-gh` | Use Python `requests` instead of the `gh` CLI for all API calls. Requires `--github-token` or a token env var. | |
| |
| #### How PMC Filtering Works |
| |
| The `--pmc` flag matches repos by prefix: the text before the first hyphen in the repository name. For example, `--pmc spark` matches `apache/spark`, `apache/spark-connect-go`, and `apache/spark-docker`. If the repo name has no hyphen, the full name is used as the prefix. |
| |
| The script downloads the list of known PMCs from `whimsy.apache.org` on first run and caches it locally (`~/.cache/asf-actions-audit/pmc-list.json`) for 24 hours. If a `--pmc` value doesn't match any known PMC, a warning is printed but it is still used as a prefix filter. |
| |
| #### What the PRs Contain |
| |
| For each repository that is missing one or more checks, the script creates a single PR on a branch named `asf-actions-security-audit` containing only the missing files: |
| |
| - `.github/dependabot.yml` — created or updated to include the `github-actions` ecosystem with a 4-day cooldown |
| - `.github/workflows/codeql-analysis.yml` — CodeQL scanning for the `actions` language |
| - `.github/workflows/zizmor.yml` — Zizmor scanning with SARIF upload |
| - `.github/workflows/allowlist-check.yml` — ASF allowlist verification on workflow changes |
| |
| #### Zizmor Pre-Check |
| |
| Before creating a PR, the script runs `zizmor` against the repository's existing workflow files. If zizmor finds errors, the **CodeQL and Zizmor workflow files are added but commented out**, with instructions explaining: |
| - That zizmor found existing issues in the workflows |
| - How to auto-fix common issues (`zizmor --fix .github/workflows/`) |
| - That the PMC should uncomment the workflows and fix remaining issues in a follow-up PR |
| |
| This avoids creating PRs that would immediately fail CI due to pre-existing problems. |
| |
| #### Interactive Confirmation |
| |
| When not in `--dry-run` mode, the script prompts for confirmation before creating each PR: |
| |
| ``` |
| Create PR for apache/spark? |
| Will add: dependabot, codeql, zizmor, allowlist-check |
| Proceed? [yes/no/quit] (yes): |
| ``` |
| |
| - **yes** (default) — create the PR |
| - **no** — skip this repository and continue to the next |
| - **quit** — stop processing entirely and print the summary |
| |
| #### Idempotency |
| |
| The script is safe to re-run. Before creating a PR for a repository, it checks whether a PR with the branch name `asf-actions-security-audit` already exists — open, closed, or merged — and skips the repo if so. |
| |
| ## Snapshotting Queued and Running Actions Jobs |
| |
| When the ASF runners feel slow, the first question is always "who is using them right now, and |
| who is waiting?" The `actions-queue-status.py` script answers that across the whole organisation |
| without needing org-admin rights. |
| |
| ### Why This Matters |
| |
| The obvious endpoint — `GET /orgs/apache/actions/runners`, which reports each runner's `status` |
| and `busy` flag — requires `admin:org`, so only Infra can call it. Everyone else debugging a slow |
| queue is left guessing. Check-run state, on the other hand, is readable by anyone who can read the |
| repository, and a check run maps one-to-one onto a workflow job. That is enough to see which |
| repositories are consuming capacity and which are stuck behind it. |
| |
| There is no org-wide REST endpoint for queued jobs at all — the only alternatives are polling every |
| repository one at a time or running a `workflow_job` webhook listener. This script batches the |
| question into a handful of GraphQL requests instead. |
| |
| ### Prerequisites |
| |
| - **Python 3.11+** and [**uv**](https://docs.astral.sh/uv/) (dependencies are declared inline via PEP 723) |
| - **`gh`** (GitHub CLI, authenticated via `gh auth login`) — or pass `--github-token` with a token |
| that can read the org's repositories and use `--no-gh` |
| |
| ### Usage |
| |
| ```bash |
| # Whole org: discover every repo with workflows, then snapshot job state |
| uv run utils/actions-queue-status.py |
| |
| # Skip discovery by reading the repo list stored in this repository |
| uv run utils/actions-queue-status.py --repos-file utils/apache-actions-repos.txt --top 40 |
| |
| # Refresh that stored list (see "The Stored Repository List" below) |
| uv run utils/actions-queue-status.py --save-repos utils/apache-actions-repos.txt |
| |
| # Write both orderings to CSV: <path>-by-running.csv and <path>-by-queued.csv |
| uv run utils/actions-queue-status.py --csv /tmp/asf-ci.csv |
| |
| # A single project, sampling more of its open PRs |
| uv run utils/actions-queue-status.py --repos-file <(echo airflow) --prs 25 |
| |
| # Group the rows by PMC instead of by repository |
| uv run utils/actions-queue-status.py --by-pmc |
| ``` |
| |
| Output is two tables — repositories sorted by running jobs, and by queued jobs — plus a one-line |
| total. Each table closes with a `TOTAL` row carrying the org-wide queued and running counts and |
| the number of active repositories; when `--top` truncates the table, a dim `shown (top N)` row |
| above it subtotals the visible rows, so what the table leaves out is visible from the table |
| itself rather than only from the summary line printed above it. `--json` prints the same data as |
| JSON, with the same figures under `totals`, and each CSV ends with a matching `TOTAL` row. |
| |
| #### Options |
| |
| | Flag | Description | |
| |------|-------------| |
| | `--org ORG` | Organisation to sweep (default: `apache`). | |
| | `--batch-size N` | Repositories per GraphQL query (default: 20). | |
| | `--prs N` | Open PRs sampled per repository, most recently updated first (default: 3). | |
| | `--suites N` | Check suites read per commit (default: 5). | |
| | `--workers N` | Batched queries in flight (default: 3). | |
| | `--top N` | Rows shown per table (default: 25). | |
| | `--by-pmc` | Group rows by PMC — the repository name's prefix before the first hyphen. The `Repos` column reads active / total. | |
| | `--include-archived` | Include archived repositories. | |
| | `--repos-file PATH` | Skip discovery and read repository names from a file; `#` comment lines are ignored. | |
| | `--save-repos PATH` | Write the discovered repository list to a file, sorted and with a header. | |
| | `--csv PATH` | Write both orderings as CSV alongside `PATH`. | |
| | `--json` | Print JSON instead of tables. | |
| | `--github-token TOKEN` | GitHub token. Defaults to `GH_TOKEN` or `GITHUB_TOKEN`. | |
| | `--no-gh` | Use Python `requests` instead of the `gh` CLI. Requires a token. | |
| | `--no-rest-fallback` | Skip the exact REST re-count for repos with more open PRs than `--prs`. | |
| | `-v`, `--verbose` | Add per-batch, per-retry and per-repo diagnostics to the progress output. | |
| | `-q`, `--quiet` | Suppress progress and diagnostics; print only the result. Overrides `--verbose`. | |
| | `--no-color` | Disable colour and progress bars. `NO_COLOR` in the environment does the same. | |
| |
| #### Progress and Diagnostics |
| |
| A full sweep takes minutes, so it reports what it is doing while it does it. Everything below goes |
| to **stderr** — stdout carries only the `--json` payload, so piping stays safe. |
| |
| Each phase is announced with a banner and closed with its elapsed time and the GraphQL points it |
| spent, and while it runs it draws a live progress bar: |
| |
| ```text |
| ▸ Phase 2/3 Status 62 queries of up to 20 repos |
| Status ━━━━━━━━━━━━━╺━━━━━━━━━ 38/62 61% 4102 pts 0:01:12 eta 0:00:45 |
| ``` |
| |
| The points figure is colour-coded — green above 2,000, yellow down to the 200-point floor, red |
| below it — so an approaching budget stop is visible before it happens. |
| |
| Bars are drawn only on a terminal. When output is piped or redirected, the same progress degrades |
| to one line per step (every page during discovery, every tenth batch afterwards), which keeps log |
| files readable and greppable: |
| |
| ```text |
| Status 10/62 4102 pts |
| ``` |
| |
| The run ends with a `Run diagnostics` table — queries issued, retries, batches split, repositories |
| skipped, REST requests, points spent and wall time — so a slow or partial sweep can be explained |
| after the fact rather than guessed at. Counters that represent failures stay dim while they are |
| zero and turn yellow or red when they are not. |
| |
| `--verbose` adds the detail behind those counters: each discovery page and its cursor, every retry |
| with its backoff and the error that caused it, each batch split, and — for every repository |
| re-counted over REST — the exact counts next to what the GraphQL sample had reported: |
| |
| ```text |
| query attempt 1/4 failed, retrying in 4s: 502 Bad Gateway |
| batch of 20 failed, splitting in two: 502 Bad Gateway |
| airflow: 10 active runs → 37 running, 0 queued (GraphQL sampled 13 running, 0 queued) |
| ``` |
| |
| `--quiet` goes the other way and prints only the result tables; `--no-color` (or `NO_COLOR` in the |
| environment) drops both colour and bars. |
| |
| #### How Discovery Works |
| |
| Rather than guessing from `pushedAt` or probing each repository over REST, the discovery query |
| reads the workflows directory straight out of the git tree: |
| |
| ```graphql |
| workflows: object(expression: "HEAD:.github/workflows") { |
| ... on Tree { entries { name } } |
| } |
| ``` |
| |
| A repository counts as using Actions only when that tree exists and holds at least one `.yml` or |
| `.yaml` entry. Archived, disabled and empty repositories are skipped. |
| |
| Pages are read 50 repositories at a time. Each node costs a git-tree lookup and an open-PR count, |
| and at 100 the query times out server-side often enough to end a sweep — two consecutive full runs |
| died on `HTTP 502`, after 200 and 300 repositories, with every retry exhausted. The same paging at |
| 50 walked the whole organisation without a single retry. |
| |
| #### Grouping by PMC |
| |
| `--by-pmc` changes the unit of the report from repository to PMC. Both orderings, the `TOTAL` |
| footer, `--top`, `--csv` and `--json` work exactly as before; only the rows change: |
| |
| ```text |
| Sorted by RUNNING jobs |
| ┏━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ |
| ┃ PMC ┃ Queued ┃ Running ┃ Repos ┃ Repositories ┃ |
| ┡━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ |
| │ airflow │ 13 │ 133 │ 2 / 10 │ airflow, airflow-client-python │ |
| │ fineract │ 0 │ 96 │ 1 / 8 │ fineract │ |
| │ datafusion │ 21 │ 49 │ 2 / 12 │ datafusion, datafusion-comet │ |
| ├────────────┼────────┼─────────┼───────────┼────────────────────────────────┤ |
| │ TOTAL │ 330 │ 881 │ 60 / 1258 │ 49 PMCs │ |
| └────────────┴────────┴─────────┴───────────┴────────────────────────────────┘ |
| ``` |
| |
| The `Repos` column reads *active / total*: how many of the PMC's repositories have jobs right |
| now, out of every repository of theirs the sweep covered. Four busy repositories mean something |
| different for a PMC of four than for a PMC of forty-seven, and the denominator is what says |
| which one you are looking at. It counts the repositories the sweep actually ran over, so a |
| `--repos-file` run measures against that file rather than the whole organisation. |
| |
| A repository's PMC is the text before the first hyphen in its name, and the whole name when there |
| is no hyphen — so `spark`, `spark-connect-go` and `spark-docker` group under `spark`. That is the |
| same rule [`--pmc` uses in `actions-audit.py`](#how-pmc-filtering-works), so the two scripts agree |
| on what a PMC covers. |
| |
| It is a naming convention, not authoritative ownership, and it is not checked against the |
| committee list: an `incubator-` repository groups under `incubator` rather than under the |
| podling's eventual PMC. For per-repository detail, drop the flag. |
| |
| The `Source` column is spent on the repository count instead, since which API counted a row is a |
| per-repository fact that a PMC of several repositories can only blur. The CSV keeps it, reading |
| `mixed` where a PMC's repositories were counted different ways. |
| |
| #### The Stored Repository List |
| |
| Discovery is the slowest phase of a sweep — it walks every repository in the organisation before a |
| single job is counted — and its result changes slowly. So the current list is stored in this |
| repository at [`utils/apache-actions-repos.txt`](utils/apache-actions-repos.txt), and passing it |
| back skips discovery entirely: |
| |
| ```bash |
| uv run utils/actions-queue-status.py --repos-file utils/apache-actions-repos.txt |
| ``` |
| |
| The file is one repository name per line, sorted, with `#` comment lines that `--repos-file` |
| ignores. Its header records how many repositories it holds and when they were discovered. Sorting |
| is what keeps it reviewable: discovery returns repositories in push order, which reshuffles on |
| every run, so an unsorted file would diff as a thousand moved lines instead of the handful that |
| actually joined or left. |
| |
| **It must be refreshed periodically.** A stored list only ages in one direction — repositories are |
| created, archived, and adopt Actions after the list was written — and a stale list fails silently, |
| because the sweep reports totals across the repositories it was given without any way to know which |
| ones are missing. Regenerate it with `--save-repos`, pointed at the file itself: |
| |
| ```bash |
| uv run utils/actions-queue-status.py --save-repos utils/apache-actions-repos.txt |
| ``` |
| |
| That runs a full sweep and rewrites the file, header and all, so the refresh and the snapshot come |
| from the same pass. Commit the result; the diff shows exactly which repositories joined and left. |
| When accuracy matters more than speed — a report on organisation-wide capacity, rather than a look |
| at what is queued right now — run the sweep without `--repos-file` and let it discover afresh. |
| |
| #### Rate Limits |
| |
| A full sweep of the `apache` organisation is not free: GraphQL charges by node count, and the cost |
| climbs sharply with `--prs`. Measured over the ~1,250 `apache` repositories that use Actions, a |
| sweep costs roughly 250 points at `--prs 3`, 2,700 at `--prs 5` and 21,000 at `--prs 10`, against |
| a budget of 5,000 points per hour. |
| |
| Raising `--prs` is a worse trade than it looks. It buys coverage of *quiet* repositories -- the |
| open-PR distribution is long-tailed, so `--prs 3` already covers 46% of them outright and `--prs 5` |
| only reaches 57% -- and every repository it fails to cover costs about one cheap REST call instead. |
| The two budgets are separate pools of 5,000, and a full REST pass uses only ~2,100-2,800 calls, so |
| GraphQL points are the scarce resource, not REST calls. Keeping the cheap pass genuinely cheap is |
| therefore also what keeps the sweep accurate. Each run prints the open-PR distribution during |
| discovery, so the trade-off can be re-checked against the organisation as it is today. Three safeguards keep a |
| sweep inside the caller's hourly allowance: |
| |
| - every query asks for `rateLimit { cost remaining resetAt }`, and the sweep stops with a warning |
| once fewer than 200 points remain, reporting partial counts rather than dying; |
| - transient failures (502, and rate-limit rejections) are retried with 4s/16s/64s backoff, because |
| GitHub enforces a per-minute points cap as well as the hourly one; |
| - `--workers` defaults to 3 — higher concurrency reliably trips that per-minute cap mid-sweep. |
| |
| Note that the REST `/rate_limit` endpoint is **not** a reliable pre-flight check here: it can report |
| a full GraphQL budget (`5000/5000, used=0`) while the API is actively rejecting queries with |
| `RATE_LIMIT`. The `rateLimit` block returned inside each query is the trustworthy signal. |
| |
| #### Why Some Repositories Are Counted Over REST |
| |
| GraphQL caps `pullRequests(first:)` at 100, and a sweep samples far fewer than that, so a |
| repository with more open PRs than `--prs` is necessarily under-counted — `apache/airflow` reported |
| 0 running jobs from a 3-PR sample while REST found 176 in the same repository. |
| |
| So the sweep uses each API where it is strongest. Every repo's query also returns the two |
| `totalCount`s that reveal whether the sample was complete: `pullRequests(states: OPEN)` and |
| `checkSuites` per commit. A repository's GraphQL numbers stand only when **both** limits held -- |
| no more open PRs than `--prs`, and no commit carrying more check suites than `--suites`. Checking |
| only the PR count is not enough: `apache/skywalking-java` has a single open PR but thirteen active |
| runs on its head commit, and reading five of them reported 32 queued jobs where the true figure was |
| 203. Everything else is re-counted over REST, which has no org-wide endpoint but is exact per |
| repository: list the runs that are still active, then count their jobs. In practice that is a small minority of repositories, so the sweep keeps |
| GraphQL's batching for the bulk of the org and pays REST's per-repo cost only where it buys |
| accuracy. The `source` column records which API produced each row, and `--no-rest-fallback` turns |
| the second pass off. |
| |
| #### Known Limits |
| |
| - Neither API attributes a job to a runner -- no labels, no runner name, no self-hosted versus |
| GitHub-hosted split -- so a job held back by a `concurrency` group cannot be told apart from one |
| waiting for capacity. Runs blocked on maintainer approval are reported separately in the |
| `runs_awaiting_approval` column, since those are not capacity waits either. For true runner |
| state, Infra can use `GET /orgs/apache/actions/runners` (`admin:org`), whose objects carry |
| `status` and `busy`. |
| - GraphQL reaches workflow runs through check suites, which hang off commits, so the cheap pass |
| sees only the default branch head and open PR heads. A run started by a push to another branch, |
| by a tag, or by a schedule on a non-default branch is invisible to it, and no `totalCount` |
| reveals that. Such a repository is only counted if something else routes it to REST. In a paired |
| run against a full REST sweep the residual difference was 8 repositories out of ~160, all small, |
| and indistinguishable from ordinary churn over the twenty minutes separating the two passes. |
| - The snapshot is a moment, not an average. A busy organisation moves enough in twenty minutes to |
| change totals by a third, so compare runs taken close together or not at all. |
