This directory is the complete source of truth for the GitHub mirror used by the Apache Traffic Server Jenkins controller.
The mirror exists because cloning directly from github.com during Jenkins fanout had performance problems that could cause PR tests to timeout and fail. It also makes ATS CI a better citizen of github.com's shared resources. Instead of making each CI job perform an external clone, GitHub sends webhook deliveries to the controller, the controller updates local bare mirrors under /home/mirror, and Jenkins jobs clone from:
https://ci.trafficserver.apache.org/mirror/trafficserver.git https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git
The package is intentionally self-contained. If the controller is lost, this README plus the files in this directory are enough to rebuild the mirror. The installed controller copy lives under /opt/github-mirror, including the Docker Compose file and mirror-specific configuration.
GitHub | | push and pull_request webhooks v https://ci.trafficserver.apache.org/github-mirror-webhook | | ATS remap v 127.0.0.1:9419/github-mirror-webhook | | github-mirror compose service, signed webhook receiver v /home/mirror/trafficserver.git /home/mirror/trafficserver-ci.git ^ | read-only bind mount | 127.0.0.1:9417/mirror/ | | github-mirror-smart-http compose service, git-http-backend v https://ci.trafficserver.apache.org/mirror/ | | ATS remap, cache disabled v Jenkins controller and docker agents
The supported Jenkins serving path is smart Git HTTP behind ATS. The public URLs stay under https://ci.trafficserver.apache.org/mirror/, but ATS remaps that path to a controller-local Compose service on 127.0.0.1:9417. That service runs git-http-backend and mounts /home/mirror read-only.
Static dumb HTTP is not acceptable for the Jenkins fanout path. It cannot negotiate packs with the client, so many child jobs can repeatedly download large static pack files and probe missing loose objects. Smart HTTP uses git-upload-pack so each clone/fetch gets a negotiated pack.
The normal controller process model is:
/opt/github-mirror/docker-compose.yml github-mirror webhook receiver on 127.0.0.1:9419 github-mirror-smart-http smart Git HTTP on 127.0.0.1:9417 github-mirror-fallback manual/timer mirror refresh helper /etc/systemd/system/github-mirror.service boot hook for the two long-running Compose services /etc/systemd/system/github-mirror-fallback.timer safety-net timer that runs docker-compose run --rm github-mirror-fallback
git-daemon on port 9418 is kept only as a diagnostic or emergency fallback. Do not use git:// URLs as the normal Jenkins configuration.
The webhook service only accepts signed GitHub payloads for:
apache/trafficserverapache/trafficserver-ciapache/trafficserver mirrors heads, tags, and pull request refs. The PR refs are required because Jenkins PR jobs receive GITHUB_PR_HEAD_SHA and then run a GitSCM checkout from the mirror.
apache/trafficserver-ci mirrors heads and tags.
These steps assume Ubuntu and a controller that serves ci.trafficserver.apache.org through ATS.
Ask ASF Infra to add the following GitHub webhooks. Generate the shared secret with github-mirror/bin/generate-webhook-secret.sh and share only the secret value, not the GITHUB_WEBHOOK_SECRET= prefix.
Hello ASF Infra, The Apache Traffic Server project would like GitHub webhooks added for our Jenkins Git mirror on ci.trafficserver.apache.org. Payload URL: https://ci.trafficserver.apache.org/github-mirror-webhook Content type: application/json Secret: We will generate a webhook secret with: github-mirror/bin/generate-webhook-secret.sh We will share it with ASF Infra out of band and install it only on the Jenkins controller in: /opt/github-mirror/config/github-mirror-webhook.env Repositories and events: apache/trafficserver: - ping - push - pull_request apache/trafficserver-ci: - ping - push Rationale: Our Jenkins jobs run on a fleet of docker hosts behind the controller. The jobs currently clone repeatedly from GitHub. We are moving those checkouts to a local read-only smart HTTP mirror on the controller. The webhook keeps branch and pull request refs current before Jenkins fans work out to the docker hosts. Thanks.
Clone or copy trafficserver-ci onto the controller.
git clone https://github.com/apache/trafficserver-ci.git /tmp/trafficserver-ci cd /tmp/trafficserver-ci
Install the mirror package.
sudo github-mirror/bin/install-controller.sh
The installer:
git, git-daemon-sysvinit, docker.io, docker-compose, python3, rsync, and util-linux;/opt/github-mirror;/opt/github-mirror/config across reinstalls;/opt/github-mirror/config/github-mirror-webhook.env if needed;/opt/github-mirror/.env with the host mirror UID/GID for Compose;/home/mirror;/home/mirror/trafficserver.git;/home/mirror/trafficserver-ci.git;http.uploadpack=true and http.receivepack=false;/etc/default/git-daemon to /opt/github-mirror/config/git-daemon.default;If the webhook secret is still CHANGE_ME, the installer leaves the Compose stack stopped and prints the command to start it after the secret is set.
Install the GitHub webhook secret.
/opt/github-mirror/bin/generate-webhook-secret.sh sudo editor /opt/github-mirror/config/github-mirror-webhook.env sudo chmod 0600 /opt/github-mirror/config/github-mirror-webhook.env
Paste the generated env line into the file:
GITHUB_WEBHOOK_SECRET=<generated secret shared with ASF Infra>
If the old controller is gone and the previous secret is unavailable, generate a new secret and ask ASF Infra to update both GitHub webhooks.
Start the Compose stack.
sudo systemctl enable --now github-mirror.service sudo systemctl enable --now github-mirror-fallback.timer cd /opt/github-mirror sudo docker-compose ps
Common service operations:
cd /opt/github-mirror sudo docker-compose restart github-mirror sudo docker-compose restart github-mirror-smart-http sudo docker-compose run --rm github-mirror-fallback
Configure ATS remaps.
Add /opt/github-mirror/ats/remap-snippet.config before the generic ci.trafficserver.apache.org Jenkins remap in:
/opt/ats/etc/trafficserver/remap.config
Add or update the /mirror/ remap with /opt/github-mirror/ats/mirror-smart-http-remap-snippet.config. The important target is:
https://ci.trafficserver.apache.org/mirror/ -> http://localhost:9417/mirror/
Keep proxy.config.http.cache.http=0. Keep hdr_rw_git.config unless testing proves it interferes with smart Git POSTs. Do not change the docs httpd/container remaps.
Reload ATS:
sudo /opt/ats/bin/traffic_ctl config reload
Verify the smart HTTP service.
cd /opt/github-mirror sudo docker-compose config sudo docker-compose exec github-mirror-smart-http httpd -t git ls-remote http://127.0.0.1:9417/mirror/trafficserver.git refs/heads/master git ls-remote http://127.0.0.1:9417/mirror/trafficserver-ci.git refs/heads/main
Configure Jenkins top-level jobs.
For GitHub PR and branch jobs, set GITHUB_URL to the ATS mirror URL:
https://ci.trafficserver.apache.org/mirror/trafficserver.git
For the GitHub PR top-level job, set quietPeriod to 0. The repo-managed top-level PR pipelines wait up to two minutes for the mirrored PR head and merge refs before starting child jobs.
Verify the public HTTPS mirror and at least one docker host.
/opt/github-mirror/bin/check-mirror.sh --pr <open-pr-number> CONTROLLER=- \ /opt/github-mirror/bin/check-docker-access.sh \ --pr <open-pr-number> docker12
To verify the exact PR head Jenkins is about to build:
GITHUB_PR_HEAD_SHA=<sha-from-jenkins-or-github> \ /opt/github-mirror/bin/check-mirror.sh --pr <open-pr-number>
The webhook is the primary update path. Every delivery is validated with X-Hub-Signature-256 before it can mutate a mirror. ping deliveries validate the endpoint without changing repositories. push deliveries update heads and tags. pull_request deliveries for apache/trafficserver update only that PR's refs/pull/<number>/head and refs/pull/<number>/merge refs.
Every mirror update runs through update-mirror.sh, takes a per-repository flock, and finishes with git update-server-info. The fallback systemd timer is only a safety net for missed deliveries. PR correctness comes from the webhook plus the Jenkins readiness gate: the top-level PR jobs wait for the mirrored PR head to match GITHUB_PR_HEAD_SHA and for the merge ref to exist before fanout starts.
Initialize or reconfigure the mirrors:
sudo /opt/github-mirror/bin/init-mirrors.sh
Recreate mirrors from scratch:
sudo /opt/github-mirror/bin/init-mirrors.sh --force
Refresh both mirrors manually:
cd /opt/github-mirror sudo docker-compose run --rm github-mirror-fallback
Refresh one ATS PR:
sudo -u gitdaemon \ /opt/github-mirror/bin/update-mirror.sh trafficserver --pr 12345
Check local and public refs:
/opt/github-mirror/bin/check-mirror.sh --pr 12345
Check from docker agents:
/opt/github-mirror/bin/check-docker-access.sh --pr 12345 docker1 docker12
Inspect services:
sudo systemctl status github-mirror.service systemctl list-timers github-mirror-fallback.timer cd /opt/github-mirror sudo docker-compose ps sudo docker-compose logs --tail=100 github-mirror sudo docker-compose logs --tail=100 github-mirror-smart-http sudo tail -n 100 /var/log/github-mirror-smart-http/access_log
Use git-daemon only as a diagnostic fallback:
git ls-remote git://ci.trafficserver.apache.org/trafficserver.git refs/heads/master
All mirror-specific application and configuration files live under:
/opt/github-mirror
That means the simplest mirror backup is:
sudo rsync -a /opt/github-mirror/ backup-host:/secure/backups/github-mirror/
The backup includes /opt/github-mirror/config/github-mirror-webhook.env, so store it in a private, access-controlled location.
The helper script creates a timestamped path-preserving backup. It includes Jenkins job XML files by default because Jenkins stores GITHUB_URL and quietPeriod outside /opt/github-mirror:
sudo /opt/github-mirror/bin/backup-controller-config.sh /secure/backup/location
To include the live ATS config files as well:
sudo /opt/github-mirror/bin/backup-controller-config.sh --include-ats \ /secure/backup/location
Use --no-jenkins when you only want the mirror package and OS integration stubs.
To restore a helper-script backup onto a rebuilt controller, inspect MANIFEST.txt, then run:
cd /secure/backup/location/<backup-name> sudo rsync -a rootfs/ / sudo systemctl daemon-reload sudo /opt/ats/bin/traffic_ctl config reload sudo systemctl restart github-mirror.service
After ASF Infra adds the webhook, use the GitHub UI to send a ping delivery. The response should be HTTP 200.
View webhook service logs and ATS access logs:
cd /opt/github-mirror sudo docker-compose logs -f github-mirror sudo tail -f /opt/ats/var/log/trafficserver/access.log
Local signed ping test:
secret=$(sudo awk -F= '/^GITHUB_WEBHOOK_SECRET=/ { print $2 }' \ /opt/github-mirror/config/github-mirror-webhook.env) body='{"repository":{"full_name":"apache/trafficserver"}}' sig=$(SECRET="$secret" BODY="$body" python3 - <<'PY' import hashlib import hmac import os print( "sha256=" + hmac.new( os.environ["SECRET"].encode(), os.environ["BODY"].encode(), hashlib.sha256, ).hexdigest() ) PY ) curl -i \ -H "X-GitHub-Event: ping" \ -H "X-Hub-Signature-256: ${sig}" \ --data "${body}" \ http://127.0.0.1:9419/github-mirror-webhook
To test the full public ATS remap path, use the same signed request against the public endpoint:
curl -i \ -H "X-GitHub-Event: ping" \ -H "X-Hub-Signature-256: ${sig}" \ --data "${body}" \ https://ci.trafficserver.apache.org/github-mirror-webhook
A bad secret or unsigned payload should return HTTP 401 and must not update any repository:
curl -i \ -H "X-GitHub-Event: ping" \ -H "X-Hub-Signature-256: sha256=bad" \ --data "${body}" \ https://ci.trafficserver.apache.org/github-mirror-webhook
Anonymous push attempts must fail:
GIT_TERMINAL_PROMPT=0 \ git push https://ci.trafficserver.apache.org/mirror/trafficserver.git \ HEAD:refs/heads/github-mirror-push-test
The expected result is rejection because the bare repositories have http.receivepack=false and the service does not allow receive-pack.
Jenkins should clone from these URLs:
https://ci.trafficserver.apache.org/mirror/trafficserver.git https://ci.trafficserver.apache.org/mirror/trafficserver-ci.git
For GitHub PR jobs, configure the top-level job's GITHUB_URL parameter to:
https://ci.trafficserver.apache.org/mirror/trafficserver.git
The repo-managed PR pipeline scripts fetch:
refs/pull/<number>/head;refs/pull/<number>/merge.They also use CloneOption(honorRefspec: true, shallow: true, depth: 1000, noTags: true, timeout: 20) so Jenkins does not fan out a wildcard PR ref fetch to every child job.
The child jobs intentionally combine narrow refspecs with shallow, no-tags checkouts. The refspec controls which refs Jenkins asks the mirror for; the shallow checkout controls how much reachable commit history Git transfers for those refs. noTags: true keeps Jenkins from pulling extra tag-reachable history that the builds do not need.
PR jobs use depth: 1000 because they still run Jenkins' local PreBuildMerge. That depth must be high enough for Git to find the merge base between the PR head and the target branch. If the depth is too low, checkout should fail during the local merge with shallow-history or missing-ancestor errors. Raise the depth before disabling shallow clone globally.
The repo-managed top-level PR jobs wait up to two minutes for the mirrored PR head to match GITHUB_PR_HEAD_SHA and for the PR merge ref to exist before starting child jobs. Set the Jenkins PR top-level job quiet period to 0.
For branch jobs, configure the top-level branch jobs' GITHUB_URL parameter to the same ATS mirror URL. Child jobs will receive that value from the fanout job. Branch jobs use shallow, no-tags checkouts with depth: 1000. Normal branch tip builds should have enough history. A manually requested old SHA outside the shallow window should fail fast instead of falling back to a large full-history fetch.
The simplest rollback is to bypass the mirror in Jenkins and clone directly from GitHub again.
Point the Jenkins PR and branch top-level job parameters back at GitHub:
https://github.com/apache/trafficserver.git https://github.com/apache/trafficserver-ci.git
Re-run or restart the affected Jenkins jobs.
If the mirror should not keep updating while GitHub URLs are in use, stop the Compose stack and fallback timer.
sudo systemctl stop github-mirror.service sudo systemctl stop github-mirror-fallback.timer
Rollback does not require deleting /opt/github-mirror or /home/mirror.
Missing PR ref:
sudo -u gitdaemon \ /opt/github-mirror/bin/update-mirror.sh trafficserver --pr <number> git --git-dir=/home/mirror/trafficserver.git show-ref refs/pull/<number>/head git --git-dir=/home/mirror/trafficserver.git show-ref refs/pull/<number>/merge
Webhook returns 401:
Confirm ASF Infra and the controller have the same secret.
Confirm the env file is readable by Docker Compose and not world-readable:
sudo ls -l /opt/github-mirror/config/github-mirror-webhook.env cd /opt/github-mirror sudo docker-compose config
Jenkins cannot clone from HTTPS:
Verify ATS remap order.
Verify /mirror/ points to http://localhost:9417/mirror/.
Verify the Compose services are healthy.
If the smart HTTP service logs say detected dubious ownership, rebuild the current image so Git trusts the bind-mounted mirror repositories.
Verify the public URL:
sudo systemctl status github-mirror.service cd /opt/github-mirror sudo docker-compose build github-mirror-smart-http sudo docker-compose restart github-mirror-smart-http sudo docker-compose exec github-mirror-smart-http httpd -t git ls-remote https://ci.trafficserver.apache.org/mirror/trafficserver.git refs/heads/master
Jenkins fetches look like dumb HTTP:
Confirm ATS is using the smart HTTP remap, not another /mirror/ backend.
Confirm logs include git-upload-pack:
sudo tail -n 100 /var/log/github-mirror-smart-http/access_log
Jenkins fetches fail with HTTP 502 after about 60 seconds:
Rebuild and restart the current smart HTTP image. The supported config gives git-upload-pack more time to generate large packs during CI fanout.
cd /opt/github-mirror sudo docker-compose build github-mirror-smart-http sudo docker-compose restart github-mirror-smart-http
Docker hosts cannot reach the mirror:
/opt/github-mirror/bin/check-docker-access.sh docker12
Webhook service will not start:
sudo systemctl status github-mirror.service cd /opt/github-mirror sudo docker-compose logs --tail=100 github-mirror
The service intentionally refuses to start when GITHUB_WEBHOOK_SECRET is unset or still set to CHANGE_ME.
The installer copies this repo-managed package and all mirror-specific config to:
/opt/github-mirror/
The key files under that directory are:
/opt/github-mirror/docker-compose.yml /opt/github-mirror/.env /opt/github-mirror/ats/remap-snippet.config /opt/github-mirror/ats/mirror-smart-http-remap-snippet.config /opt/github-mirror/bin/backup-controller-config.sh /opt/github-mirror/bin/generate-webhook-secret.sh /opt/github-mirror/bin/github-mirror-webhook.py /opt/github-mirror/bin/init-mirrors.sh /opt/github-mirror/bin/update-mirror.sh /opt/github-mirror/config/github-mirror-webhook.env /opt/github-mirror/config/github-mirror-webhook.env.example /opt/github-mirror/config/git-daemon.default /opt/github-mirror/httpd/Dockerfile /opt/github-mirror/httpd/mirror.conf /opt/github-mirror/systemd/github-mirror.service /opt/github-mirror/systemd/github-mirror-fallback.service /opt/github-mirror/systemd/github-mirror-fallback.timer /opt/github-mirror/webhook/Dockerfile
The installer creates these small OS integration files:
/etc/default/git-daemon -> /opt/github-mirror/config/git-daemon.default /etc/systemd/system/github-mirror.service /etc/systemd/system/github-mirror-fallback.service /etc/systemd/system/github-mirror-fallback.timer
The local bare mirrors live under:
/home/mirror/trafficserver.git /home/mirror/trafficserver-ci.git
The smart HTTP container writes host-mounted logs here:
/var/log/github-mirror-smart-http/
ATS needs the webhook and mirror remap entries in:
/opt/ats/etc/trafficserver/remap.config
Use these repo snippets as the source of truth for those remaps:
/opt/github-mirror/ats/remap-snippet.config /opt/github-mirror/ats/mirror-smart-http-remap-snippet.config
The mirror remap also references the existing ATS header rewrite file:
/opt/ats/etc/trafficserver/hdr_rw_git.config
Jenkins stores the GITHUB_URL and quietPeriod settings in job XML under:
/opt/jenkins/home/jobs/
Check the GitHub PR top-level job and branch top-level job configs there after a rebuild. Steady-state mirror updates do not require a cron file.