7607522 fixed the symptom in the wrong place. It made `test-run-tests.sh`'s
`run_fake()` spawn fixtures via `env -u RUN_TESTS_STRICT`, which stops that one
suite inheriting strictness — and leaves every future suite to defend itself the
same way. The variable's only job is done the moment `run-tests.sh` latches it
into the `STRICT` shell local, so it is unset there now and the leak is gone for
every child. The `env -u` stays as this suite's own defence in depth rather than
as the fix.
Two corrections to that commit's account of the bug, both overstated and both
cheap to have checked:
- The blast radius was two assertions, cases 10c and 10g, not six. Nothing else
in the repo reads `RUN_TESTS_STRICT`.
- The pre-push gate was never red. It invokes `bash tests/run-tests.sh --strict`,
and the flag sets a shell local that is never exported, so the flag spelling
never leaked at all. Only the env-var spelling did.
That asymmetry between the two documented spellings is the real finding, and
nothing asserted against it. Case 10b compared the parent's verdict, which is the
half that already matched; the halves that differed were the environments the two
spellings handed every dispatched suite. New case 10i asks a child directly —
`${VAR+set}`, so an exported empty value still counts as a leak — and asserts the
two observations equal each other rather than a hardcoded expectation, so they
cannot drift apart in a direction the case did not anticipate.
Two pre-existing failures, both red at HEAD before ADR-0020 work began,
both invisible in an ordinary local run.
RUN_TESTS_STRICT leaked from the environment into test-run-tests.sh's
fixture children. The meta-test is itself a suite the runner discovers,
so under the gate's own invocation the variable propagated outer runner
-> batch_run -> the fixture's copy of run-tests.sh, flipping it strict.
Case 10c (a deliberate control asserting a skip is tolerated WITHOUT
strict) then failed. Six further cases were silently running strict too
and asserting against the wrong stream — case 9 was matching the stderr
strict block rather than the stdout skip list it was written to check.
run_fake now spawns via 'env -u RUN_TESTS_STRICT', so fixture strictness
is a property of the case, never of how the file was launched. No
assertion weakened; run-tests.sh itself is untouched.
pretty-format-json --autofix was re-sorting apm's output on the way into
every commit. .claude/settings.json is apm-owned (ADR-0018/0019) and its
exclude list named fifteen generated manifests but not this file, so
since 2e395a4 it has been committed in a key order apm would never write
— permanent drift on a file with an empty git diff. Content was always
byte-identical; only JSON key order differed. The exclude ships in the
same commit as the corrected file because otherwise the hook re-breaks
it during staging.
apm.lock.yaml: generated_at churn, plus lint's exec_status corrected from
'deployed' to 'gated_pending_approval' — executables.allow grants only
kyberforge#1.5.0, so lint's hooks/bin are genuinely gated.
New coverage: an ambient RUN_TESTS_STRICT must not reach a fixture that
did not ask for it, and under --strict the skip report goes to stderr
only with the stdout list suppressed. Neither was pinned.
Three gaps left by the previous round, all the same shape: a gate reporting
success having verified less than it appears to.
run_bats() hard-failed on a missing or non-executable runner but never checked
that the runner produced anything. An empty, executable run-bats.sh exits 0,
and the dispatcher printed a green summary with 166 bats tests silently absent.
It now requires an "N tests, M failures" line with a non-zero count.
run-tests.sh's skip listing is swallowed by pre-commit on a pass, so on a
machine without vale three suites exited 77 and the pre-push gate went green
having run 14 of 17. The hook now invokes it as --strict, where a skip fails
and the error names each suite and the reason it skipped. An ad-hoc local run
still skips gracefully -- at pre-push a skip means a documented dependency is
missing, which is a setup error, not a legitimate state. Deliberately not wired
to the vale downgrade's env var: one flag must not disarm two gates.
BATS_FILE_FLOOR is replaced by an expectation derived from git ls-files. A
floor of 8 against a real count of 10 let two files and eleven tests disappear
green, and the number needed an edit whenever a plugin was added. The derived
set needs no number, and catches an addition as well as a removal -- a .bats
file staged into the index and deleted from disk is now demanded back.
The vale opt-out announced its downgrade to nobody: pre-commit prints nothing
for a passing hook, so the summary line AGENTS.md tells the reader to check was
unreachable in exactly the situation it exists for. The hook is now verbose.
Also corrects the PROBES_CHECKED guard, whose commit message described a state
that cannot occur -- the .vale.ini loop errs first. Its two reachable triggers,
a gutted probe heredoc and a probe row naming a missing directory, had no test;
they do now, each asserting the guard is the sole cause.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
run_bats() was an if with no else, so a missing or non-executable run-bats.sh
made all 166 bats tests disappear with no diagnostic while the run printed a
green summary and exited 0. --bats-only became a total no-op. This is the same
defect the PR fixed one level down, left open in the dispatcher that pre-push
actually invokes -- and nothing tested run-tests.sh at all.
run-bats.sh's aggregation was asserted by nothing. Three separate mutations to
its failure-detection line all survived the existing suite, because real bats
emits both a nonzero exit and "not ok" lines, so each signal masked the other.
The new cases produce each signal without the other; all three mutants now die.
Also in this pass:
- zero discovered .bats files exited 0, so a widened path exclusion retired the
suite silently. Replaced with a file-count floor: a collapse to one or two
files is the same failure as a collapse to zero
- an existing-but-empty status file counted as a pass, because [[ "" -eq 0 ]]
is arithmetic-true and the || echo 1 fallback only covered a missing file.
The repro is deterministic: the stub truncates its own status file, then
kill -9s its parent so the real exit-code write never happens
Adds tests/test-run-tests.sh, 9 cases pinning the exit 0 / 1 / 77 three-way
split against a single summary line.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
