Two pre-existing failures, both red at HEAD before ADR-0020 work began,
both invisible in an ordinary local run.
RUN_TESTS_STRICT leaked from the environment into test-run-tests.sh's
fixture children. The meta-test is itself a suite the runner discovers,
so under the gate's own invocation the variable propagated outer runner
-> batch_run -> the fixture's copy of run-tests.sh, flipping it strict.
Case 10c (a deliberate control asserting a skip is tolerated WITHOUT
strict) then failed. Six further cases were silently running strict too
and asserting against the wrong stream — case 9 was matching the stderr
strict block rather than the stdout skip list it was written to check.
run_fake now spawns via 'env -u RUN_TESTS_STRICT', so fixture strictness
is a property of the case, never of how the file was launched. No
assertion weakened; run-tests.sh itself is untouched.
pretty-format-json --autofix was re-sorting apm's output on the way into
every commit. .claude/settings.json is apm-owned (ADR-0018/0019) and its
exclude list named fifteen generated manifests but not this file, so
since 2e395a4 it has been committed in a key order apm would never write
— permanent drift on a file with an empty git diff. Content was always
byte-identical; only JSON key order differed. The exclude ships in the
same commit as the corrected file because otherwise the hook re-breaks
it during staging.
apm.lock.yaml: generated_at churn, plus lint's exec_status corrected from
'deployed' to 'gated_pending_approval' — executables.allow grants only
kyberforge#1.5.0, so lint's hooks/bin are genuinely gated.
New coverage: an ambient RUN_TESTS_STRICT must not reach a fixture that
did not ask for it, and under --strict the skip report goes to stderr
only with the stdout list suppressed. Neither was pinned.
Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.
run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.
run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.
BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.
The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.
Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
run_bats() was an if with no else, so a missing or non-executable run-bats.sh
made all 166 bats tests disappear with no diagnostic while the run printed a
green summary and exited 0. --bats-only became a total no-op. This is the same
defect the PR fixed one level down, left open in the dispatcher that pre-push
actually invokes -- and nothing tested run-tests.sh at all.
run-bats.sh's aggregation was asserted by nothing. Three separate mutations to
its failure-detection line all survived the existing suite, because real bats
emits both a nonzero exit and "not ok" lines, so each signal masked the other.
The new cases produce each signal without the other; all three mutants now die.
Also in this pass:
- zero discovered .bats files exited 0, so a widened path exclusion retired the
suite silently. Replaced with a file-count floor: a collapse to one or two
files is the same failure as a collapse to zero
- an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]]
is arithmetic-true and the || echo 1 fallback only covered a missing file.
The repro is deterministic: the stub truncates its own status file, then
kill -9s its parent so the real exit-code write never happens
Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way
split against a single summary line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
