Defame1297
acd2f1d422
A review of PR #85's last two commits (1164f3a,4d018af) found the new release-gate script fails open in four separate ways, and the new drift check for the duplicated Vale styles only ever detects drift after a human already hand-edited both copies out of sync. check-release-needed.sh: - The `-e` existence filter dropped a RELEASE_PATHS entry from the diff pathspec once it was deleted from the tree, so deleting a path exposed via .pre-commit-hooks.yaml since the last tag passed the gate clean — exactly the breakage the gate exists to catch. git diff reports deletions fine without an existence check; the filter is gone. - `git diff ... 2>/dev/null || true` turned any git failure (a shallow clone missing the tag's objects, a corrupted ref) into an empty, falsely-clean diff. The diff result is no longer swallowed: a failure now hard-fails with the underlying git error visible. - RELEASE_PATHS was a hand-maintained array duplicating .pre-commit-hooks.yaml's entry: paths with only a comment holding them in sync, and was already over-broad (it swept in validate.sh / validate-provenance.sh, which no hook entry references). It's now parsed straight from .pre-commit-hooks.yaml's entry: lines at runtime, so it can't drift from the manifest and only tracks what a hook actually exposes. - `git describe --tags --abbrev=0` accepted any tag reachable from HEAD as the diff baseline, not just release tags. Added `--match 'v[0-9]*.[0-9]*.[0-9]*'` so an incidental checkpoint tag can't shift the baseline and mask a real release-relevant change. check-vale-style-sync.sh still only detects drift between skill-audit's and agent-audit's duplicated vale-wrap.sh/styles/Kyberforge copies (both copies must exist independently per the plugin's no-cross-skill- path packaging rule — a symlink would break at install time). Added scripts/sync-vale-styles.sh to regenerate skill-audit's copy from agent-audit's canonical one on demand, and pointed the sync check's failure message at it, so fixing drift is one command instead of a hand diff across two files. Also recorded, rather than silently left unfixed: check-release-needed.sh only fires on a local `git push` through pre-commit's pre-push hook — a PR merged via Gitea's merge button, or CI invoking `pre-commit run --hook-stage pre-push` directly, never sets PRE_COMMIT_REMOTE_BRANCH and skips the gate entirely. Closing that needs a server-side CI job this repo doesn't have yet; documented as a known limitation in ADR-0014 rather than papered over. Separately, LESSONS.md's "a clean check can mean nothing ran" entry was marked **Graduated** without ever being promoted per the repo's own graduation rule (3+ instances → a standing doc, marked `[graduated → target file]`). Actually promoted it into core/instructions/testing.md and fixed the marker. tests/test-check-release-needed.sh gained 4 regression tests, one per check-release-needed.sh fix above, each verified to fail against the pre-fix script and pass against the current one. Verification: bash tests/run-tests.sh (11 scripts + 125 bats, all passing), pre-commit run --all-files, and pre-commit run --all-files --hook-stage pre-push all clean. ADR: 0014
762 B
762 B
Testing conventions
- Prefer integration tests over mocks. Mocks mask production divergence; real systems catch real failures.
- Automate everything automatable. Manual testing only for nuanced UI/UX or agent interaction behaviour requiring human judgment.
- Test observable end-state, not implementation internals. Tests must survive refactoring.
- No test is better than a wrong test. A passing mock that masks a real failure is actively harmful.
- A clean result can mean nothing ran. Before trusting a new check, prove it fails against a deliberately-bad fixture, then run it against the real target. Where a check can scan zero inputs, assert on the input count, not just the exit code — a zero-file run and a real clean pass look identical otherwise.