docs: log two lessons from PR #95's fourth review round

The first is new: four fixes specified by the orchestrating reviewer were wrong,
each in a way that would have shipped a guard catching nothing. A finding arrives
with evidence and gets scrutinised; the fix beside it arrives with the same
authority and gets implemented. Worth naming because the failure is silent — the
guard looks right and the suite stays green.

The second is flagged as a graduation candidate rather than promoted: it overlaps
two 2026-08-09 entries on assertions that pin something adjacent to what they
name. Per the graduation rule, promotion to core/instructions/testing.md is the
human's call, not mine.

Refs: #95

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7GvKuJfy2WrdBmUttV4DT
This commit is contained in:
2026-08-14 07:09:19 +00:00
parent 013b913bd4
commit 55d956b298

View File

@@ -165,3 +165,13 @@ Dropping the `--config` argument from `.pre-commit-hooks.yaml` was the right fix
## 2026-08-09 — A documented impossibility is a claim, not a constraint
`vale-wrap.sh` flattens multi-line YAML `description:` scalars so Vale's `text.frontmatter.description` scope keeps matching. Its last-resort branch rewrote ASCII `'` to U+2019, justified at the emission site and in review as "the single combination no YAML scalar can carry verbatim" — an accepted-by-design residual, documented and test-covered, which is exactly why nobody retested it. The claim was false: a `|-` literal block with one indented content line carries `'`, `"`, `\` and `: ` verbatim, keeps the scope alive, and the wrapper's own header docstring already said literal blocks were unaffected. The cost of the unexamined claim was a silent underlint on 12 of 54 in-scope files — any rule whose token contained an apostrophe simply never fired, and the covering test (case 20) pinned only "the scope stays alive", so it passed either way. Fix: when a residual is accepted because something is "impossible", write down the specific claim in a falsifiable form and test *that*, not the workaround built on top of it. The tell here was that the residual and its justification were documented in the same breath by the same author — documentation records a belief, and a belief adjacent to a workaround is the one most worth attacking. Related: an assertion written to cover an accepted residual tends to assert the residual's *presence* rather than the behaviour it costs; case 20b asserted the scope survived flattening, never that a rule matching the rewritten characters still fired.
## 2026-08-14 — A fix handed down with authority is the least-reviewed code in the change
Across one review round, four fixes specified by the orchestrating reviewer were wrong, and every one would have shipped a guard that looked correct and caught nothing — the same defect class the guard was written to close. `nproc([[:space:]]|$)` does not match `$(nproc)`, the only spelling that occurs in real code. `grep -E ... | grep -Evq ...` under `set -o pipefail` returns 141 because `-q` exits on first match and SIGPIPEs the upstream, and 141 as an `if` condition reads as "no findings" — worse, it is *size-dependent*, so on the real 4-line `.vale.ini` the broken form behaves correctly and only fails once the input grows. `FUNCNAME` and `BASH_ARGC` were proposed as never-empty shell arrays to exempt from an unguarded-expansion scan; both are empty in reachable states (outside a function; `BASH_ARGC` measured 1 at top level and 0 inside a function), so exempting them suppresses a real bash 3.2 abort. `sed 's/#.*//'` as a comment-stripper truncates at the `#` in `${var#prefix}` — a form this repo actually uses at `check-manifests.sh:58` — reintroducing the exact blind spot being fixed. Each was caught only because the implementing agent re-derived the fix and measured, rather than applying what it was told; each had survived being written down confidently in a numbered finding with a reproduction attached. The asymmetry is the point: a finding arrives with evidence and gets scrutinised, while the fix beside it arrives with the same authority and gets implemented. Fix: state a proposed fix as a hypothesis with its own falsifiable check, and require the implementer to verify the fix mechanism independently of the defect reproduction — the two are different claims. The tell is a fix whose correctness depends on a regex boundary, a shell exit-status rule, or an "always/never" property of a builtin: measure it at the size, scope, and spelling it will actually meet, because the small case and the shipped case can disagree.
## 2026-08-14 — Every assertion needs a revert it provably fails against [graduation candidate]
Mutation testing a review round's own fixes found repeatedly that a passing test was pinning nothing. Deleting `sync_dir`'s stale-directory wipe, its check-mode stale branch, or three of five `MIRROR_DIRS` entries each left the suite at 18/18 green; so did replacing the hooks trailing-newline normalisation with plain `cp`. A pair of concurrency assertions written to guard a reentrancy defect caught it 0 times in 10 runs against the deliberately broken script — and one of them was structurally incapable of ever catching it, because the broken code wrote to the system temp dir while the assertion inspected `$TMPDIR`. A fixture-leak fix ran green with and without the fix, verified only by external observation. Two manifest fixtures passed with the canonicalisation they claimed to cover deleted, rescued by an unrelated name-matching axis. In each case the test named the right behaviour in its description and asserted something adjacent to it. The cheap discipline that finds all of these: for every assertion, construct the revert it is supposed to catch and confirm it fails — and when an assertion survives every revert you can think of, that is not reassurance, it is the finding (one test only revealed itself as decoration once a sixth, differently-targeted revert was built for it). Fix: treat "which revert does this fail against?" as a required answer at the time an assertion is written, and record it where the assertion lives, since a test's own description is exactly the artifact that made the gap invisible.
Graduation candidate: this overlaps 2026-08-09's "an assertion written to cover an accepted residual tends to assert the residual's presence rather than the behaviour it costs" and the same date's "assert on the expected members, so a derivation whose input vanished fails loudly instead of quietly covering less." Three entries circling one pattern — human review for promotion to `core/instructions/testing.md`.