fix(release): produce a clean source-only tarball and gate it in verify-rc (#829)
The 0.1.0 rc1 source zip drew a binding -1: it bundled 181 .pyc files in
61 __pycache__ dirs (zipped from a working tree, not exported from the
tag) and carried 138 dangling symlinks because .gitattributes
export-ignored .agents/ while the committed .claude/skills / .kiro/skills
relays and the specs still referenced into it; the same stripping broke
skill/spec link integrity (.github templates, projects/_template/.gitignore).
Packaging (.gitattributes):
- Ship .agents/ so every relay symlink resolves to skills/ (0 dangling).
- Strip only real CI/dev under .github/ (workflows, dependabot.yml); keep
ISSUE_TEMPLATE/ and PULL_REQUEST_TEMPLATE.md, which shipped skills link.
- Anchor root-only dotfiles with `/` so projects/_template/.gitignore ships.
Verification hardening (release-verify-rc):
- Step 6 always surfaces and hard-fails on .pyc / __pycache__.
- New Step 7 "source-tree integrity": no dangling symlinks, and run
symlink-lint + skill-and-tool-validate + spec-validate against the
unpacked tarball; hard-fail on broken internal references. Renumber the
following steps and add failure-mode rows.
- Reinforce in release-build.md: always `git archive <tag>`, never `zip -r`.
Verified against a real git archive of the fixed tree: 0 .pyc, 0 dangling
symlinks, all three validators exit 0, no VCS metadata.
Co-authored-by: Tester <t@example.com>
diff --git a/.gitattributes b/.gitattributes
index 09a8bf8..b03ee42 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -20,16 +20,37 @@
# apache-magpie-<version>-source.zip) honours `export-ignore`, so the
# paths below are kept out of the signed source .zip that the [VOTE]
# votes on. Keep this list to VCS/CI/editor metadata — never source.
+#
+# IMPORTANT — do NOT export-ignore a directory that shipped files
+# still reference (by markdown link, `test -f`, or symlink). The
+# committed agent-view symlink dirs (`.claude/skills/`, `.github/skills/`,
+# `.kiro/skills/`) relay through `.agents/skills/*` → `skills/*`, and
+# specs test those paths, so `.agents/` MUST ship or every relay dangles
+# in the archive. Likewise `.github/ISSUE_TEMPLATE/` and
+# `.github/PULL_REQUEST_TEMPLATE.md` are linked by skills, and
+# `projects/_template/.gitignore` is example content — so only the true
+# CI/dev entries under `.github/` are stripped, and `.gitignore` is
+# anchored to the repo root. `release-verify-rc` re-checks the unpacked
+# tarball (symlink-lint + validators + no-.pyc) so a regression here
+# fails the RC before the [VOTE].
-.gitattributes export-ignore
-.gitignore export-ignore
-.github/ export-ignore
+# Root-only VCS / dev-tool metadata (anchored so identically-named files
+# deeper in the tree — e.g. projects/_template/.gitignore — still ship).
+/.gitattributes export-ignore
+/.gitignore export-ignore
+/.pre-commit-config.yaml export-ignore
+/.lychee.toml export-ignore
+/.lycheecache export-ignore
+/.markdownlint.json export-ignore
+/.typos.toml export-ignore
+/.zizmor.yml export-ignore
+/.apache-magpie.session-state.json export-ignore
+
+# Editor metadata.
.idea/ export-ignore
-.agents/ export-ignore
-.pre-commit-config.yaml export-ignore
-.lychee.toml export-ignore
-.lycheecache export-ignore
-.markdownlint.json export-ignore
-.typos.toml export-ignore
-.zizmor.yml export-ignore
-.apache-magpie.session-state.json export-ignore
+
+# CI / bot config only. The rest of `.github/` (ISSUE_TEMPLATE,
+# PULL_REQUEST_TEMPLATE.md, and the skills relay) is referenced by
+# shipped skills and resolves via `.agents/`, so it stays in the release.
+.github/workflows/ export-ignore
+.github/dependabot.yml export-ignore
diff --git a/projects/magpie/release-build.md b/projects/magpie/release-build.md
index 08a1420..9854aea 100644
--- a/projects/magpie/release-build.md
+++ b/projects/magpie/release-build.md
@@ -31,7 +31,11 @@
## Build invocation
The canonical source artefact is a deterministic `git archive` of the
-tagged tree — no VCS metadata, no build output:
+tagged tree — no VCS metadata, no build output. **Never `zip -r` a
+working directory**: that captures `__pycache__/*.pyc` and other test
+cruft and produces a non-reproducible tarball (this was the rc1 `-1`).
+`git archive` only ever includes tracked files from the tag, and honours
+the `export-ignore` rules in [`.gitattributes`](../../.gitattributes):
```bash
# From the release tag <version>-rcN:
diff --git a/skills/release-verify-rc/SKILL.md b/skills/release-verify-rc/SKILL.md
index 64d59e8..a1914bb 100644
--- a/skills/release-verify-rc/SKILL.md
+++ b/skills/release-verify-rc/SKILL.md
@@ -9,7 +9,9 @@
Read-only pre-flight verification of a staged release candidate (RC)
for `<upstream>`. Checks artefact integrity (GPG signatures and
checksums), Apache RAT licence headers, NOTICE/LICENSE completeness,
- prohibited-binary absence, and version-string consistency. Emits a
+ prohibited-binary absence (including `.pyc` / `__pycache__`),
+ source-tree integrity (no dangling symlinks or broken internal
+ references), and version-string consistency. Emits a
structured PASS / PASS-WITH-WARNINGS / FAIL report. Makes no state
change; a `--post-to <planning-issue>` flag proposes a comment for
explicit RM confirmation before any posting.
@@ -431,9 +433,14 @@
scan command:
```bash
-# Example: find compiled Java class files and native shared libraries
-find <unpacked-dir> -type f \( -name "*.class" -o -name "*.jar" \
- -o -name "*.so" -o -name "*.dylib" -o -name "*.dll" -o -name "*.exe" \)
+# Compiled Java class files, native shared libraries, AND compiled
+# Python bytecode. `.pyc` / `__pycache__` must NEVER appear in a
+# source release — their presence proves the artefact was zipped from
+# a working tree that had run tests rather than exported clean from
+# the tag (build via `git archive <tag>`, never `zip -r`).
+find <unpacked-dir> \( -type f \( -name "*.class" -o -name "*.jar" \
+ -o -name "*.so" -o -name "*.dylib" -o -name "*.dll" -o -name "*.exe" \
+ -o -name "*.pyc" \) -o -type d -name "__pycache__" \) -print
```
Emit the bare `find` with no `grep` post-filtering: the recipe must
@@ -472,11 +479,78 @@
}
```
-`status` is `"FAIL"` if `prohibited_found` is non-empty.
+`status` is `"FAIL"` if `prohibited_found` is non-empty. Any `.pyc`
+file or `__pycache__` directory found is a hard `FAIL` (never an
+`EXPECTED-BINARY`): it is both a prohibited binary and proof the
+tarball was not exported clean from the tag.
---
-## Step 7 — Version string consistency
+## Step 7 — Source-tree integrity (dangling symlinks + broken references)
+
+The rc1 `-1` came from this class of defect, not from signatures or
+RAT: committed agent-view symlinks (`.claude/skills/*`, `.kiro/skills/*`,
+`.github/skills/*`) relayed into a directory (`.agents/`) that the
+release stripped, so **every relay dangled**; and shipped files linked
+to other stripped paths (`.github/` templates, `projects/_template/.gitignore`,
+`.claude/skills/magpie-*/SKILL.md`), so the framework's own validators
+failed. This step runs the same checks the framework applies to its own
+tree, but **against the unpacked tarball**, so a packaging regression
+(an `export-ignore` that strips a still-referenced path) fails the RC
+before the `[VOTE]` rather than during it.
+
+Emit the paste-ready recipe (run from the unpacked dir; adapt tool
+paths to the project — for Magpie the tools ship in-tree under
+`tools/`):
+
+```bash
+cd <unpacked-dir>
+
+# 1. Dangling symlinks — every symlink must resolve inside the tarball.
+find . -type l ! -exec test -e {} \; -print # any output = FAIL
+
+# 2. Internal reference / link integrity — run the project's own
+# validators against the unpacked source (Magpie ships these):
+uv run --project tools/symlink-lint symlink-lint .
+uv run --project tools/skill-and-tool-validator skill-and-tool-validate
+uv run --project tools/spec-validator spec-validate # if the project ships specs
+```
+
+Classify:
+
+- `PASS` — no dangling symlinks and every validator exits 0.
+- `FAIL` — any dangling symlink, or any validator reports a broken
+ internal link / missing referenced file. This is a hard `FAIL`:
+ a release whose own files reference content that was stripped from
+ the artefact is incomplete.
+- `SKIP` — the project ships no symlinks and no in-tree validators
+ (state this explicitly; do not silently pass).
+
+Do **not** post-filter the `find`; surface every dangling link so the
+voter sees the full set. When a validator is not shippable in the
+tarball, run it from a checkout of the *same tag* against the unpacked
+dir instead, and note that in the report.
+
+Return ONLY valid JSON with this structure:
+
+```json
+{
+ "step": "source-tree-integrity",
+ "status": "PASS" | "FAIL" | "SKIP",
+ "dangling_symlinks": ["<path>"],
+ "validator_failures": [
+ {"validator": "<name>", "detail": "<broken link / missing target>"}
+ ],
+ "paste_recipe": "<multi-line shell commands>"
+}
+```
+
+`status` is `"FAIL"` if `dangling_symlinks` is non-empty or any
+validator failed.
+
+---
+
+## Step 8 — Version string consistency
Read each file listed in `version_manifest_files` from
`release-management-config.md` (e.g. `setup.cfg`,
@@ -510,7 +584,7 @@
---
-## Step 8 — Hand-back verification report
+## Step 9 — Hand-back verification report
Aggregate the per-step results into a final report.
@@ -604,7 +678,10 @@
| Step 5 FAIL — NOTICE or LICENSE absent | Source artefact build skipped packaging | RM fixes build process and cuts a new RC |
| Step 5 WARN — material diff | Licence or attribution changed vs previous release | RM reviews diff; if intentional, document in planning issue |
| Step 6 FAIL — prohibited binary | Binary sneaked into source artefact | RM removes binary, updates `.gitattributes` or build excludes, cuts new RC |
-| Step 7 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC |
+| Step 6 FAIL — `.pyc` / `__pycache__` present | Tarball zipped from a working tree that ran tests, not exported clean from the tag | RM rebuilds via `git archive <tag>` (never `zip -r`), cuts new RC |
+| Step 7 FAIL — dangling symlink | A committed symlink's target was stripped by `export-ignore` (or is otherwise absent) | RM fixes `.gitattributes` to ship the target (or drops the symlink), cuts new RC |
+| Step 7 FAIL — broken internal reference | A shipped file links to a path stripped from the artefact | RM stops stripping the referenced path, or repoints the reference at shipped content, cuts new RC |
+| Step 8 FAIL — version mismatch | Version bump missed one manifest file | RM fixes the manifest and cuts a new RC |
---
