docs: reconcile the ADRs, gates and audit log with the shipped behaviour
Why A six-agent review of the two preceding commits found their code sound -- the differential claim holds, the published hook contract is byte-unchanged -- but their prose drifted from it in three ways: statements of fact the code contradicts, markers in a convention this repo does not use, and figures that went stale when the merge changed what they counted. Implementation Notes ADR-0025's edge-path table is rewritten around one stated doctrine: exit 0 is audited and clean, exit 1 is audited with findings or a target present but unreadable, exit 2 is that nothing was audited. Its old row 1 promised "one generic matches-neither message" for three different inputs; there are three distinct messages, and the missing-path case exited 1 until the preceding commit fixed it. Rows are added for the preflight and CDPATH changes, because a table claiming to enumerate every entry-point behaviour change reproduces its own "an earlier revision of this ADR said they were behaviour-neutral" failure if it omits any. ADR-0025 also gains a Consequences supersession record in ADR-0016's form: partially-superseded entries for 0008, 0014, 0020 and 0021, and explicit "is not superseded" entries with reasoning for the rest. Twelve ADRs are amended and it previously listed none. ADR-0008 moves from an amendment note to partially superseded. Its contract genuinely narrowed -- an agent .md outside an agents/ directory was audited before the merge and is refused now -- and ADR-0020 already recorded that the merge "reopens ADR-0008". Its detector description said "a path under .apm/agents/", the phrasing ADR-0025 rejects as wider than the script and circular; the shipped rule is a .md whose immediate parent is named agents/, at any scope. ADR-0020's amendment claimed the boundary resolver is sourced by validate-provenance.sh. It is not, and never was; only validate.sh sources it, once per mode branch. Three Home-column entries pointed at reference filenames the merge renamed, one of which now resolves to two files because its row covers skills and agents. Five ADRs opened with "Skill renamed per ADR-0025", a form this repo does not use, in the same commit that used the conventional "Amended by ADR-0025" twice. They are normalized. "Renamed" was also wrong: the BREAKING-CHANGE trailer says the skills were removed and their flows merged. SIMPLIFICATION-AUDIT.md had 2026-09-15 notes attached to headlines that were never updated, against its own convention of correcting in place with strikethrough. Every figure here was re-derived at HEAD by command, and several differed from the review's own numbers, so the notes record the basis rather than the result alone. LESSONS.md asserted the two review-time suite failures were the SIGPIPE race. The commit that fixed that race explicitly declined to claim it -- the suite was running while agents edited live config files -- so the hedge is restored. Impact No code, test or configuration change; documentation only. Suites stay 20/20 strict with 0 skipped and 374/374 bats. No gate parses ADR or gates.md content, so nothing here is load-bearing for a hook. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YR2CjVumUbEGWcMikcoXBD
This commit is contained in:
@@ -367,6 +367,16 @@ into the single `factory-audit/scripts/lib-boundary-resolver.sh`, sourced by tha
|
||||
The two remaining copies must still stay byte-identical — a plugin script cannot source the root
|
||||
one, which is the constraint that forces a copy to exist at all.
|
||||
|
||||
`tests/test-adr0020-contract.sh` pins that arrangement, and one of its assertions was green on a
|
||||
defect it named. "`validate.sh` sources the resolver in **both mode branches**" was implemented as a
|
||||
file-wide `grep -Ec … -ge 2`, which cannot see a branch at all: delete the `agent)` arm's source line
|
||||
and duplicate the `skill)` arm's, and the file-wide count is still 2 and the assertion still passes,
|
||||
with the agent path running no resolver or some other one. It is now a **per-arm structural check** —
|
||||
each arm of `validate.sh`'s `case "$MODE" in` block must carry exactly one `source` line inside its
|
||||
own body, and the file must carry exactly those two — with a mutation self-test that performs that
|
||||
exact count-preserving edit on a copy and requires the check to fail on it. The suite went 25 → 28
|
||||
cases.
|
||||
|
||||
### `python3` and PyYAML are hard requirements
|
||||
|
||||
Both, and neither is a best-effort accelerator.
|
||||
@@ -426,6 +436,58 @@ the hook definitions filter on `SKILL.md`
|
||||
"extend" that hook's `files:` pattern to cover agents** on the assumption that the script already
|
||||
knows the difference; doing so silently enforces a gate ADR-0020 declines to set.
|
||||
|
||||
### The three exit tiers of `factory-audit`'s validators
|
||||
|
||||
`validate.sh` and `validate-provenance.sh` use the exit code to say **whether an audit happened**,
|
||||
not only what it found. The skill's flow references tell the auditor to surface a non-zero run's
|
||||
output verbatim as real findings, so the distinction is load-bearing rather than cosmetic:
|
||||
|
||||
| Exit | Means |
|
||||
|---|---|
|
||||
| **0** | audited, and clean |
|
||||
| **1** | audited, and there are findings — or the target is present but cannot be opened |
|
||||
| **2** | **nothing was audited.** The checks never ran |
|
||||
|
||||
Exit 2 covers: no arguments at all; a target that does not exist, with its own "does not exist"
|
||||
message; a target matching neither the skill nor the agent shape; a missing or unreadable `lib-*.sh`
|
||||
beside the entry script; and a missing `python3` or PyYAML. Every one of those used to exit 1, so
|
||||
an install problem or a typo'd path arrived at the auditor as findings about the artifact. A missing
|
||||
target was the sharpest case: only the directory branch stats the target, so a typo'd `*.agent.md`
|
||||
path was classified on name alone, handed to `python3`, and came back as a FAIL about a file that was
|
||||
never there.
|
||||
|
||||
**A dangling symlink or a symlink loop stays exit 1, deliberately.** Both are false to `-e` and true
|
||||
to `-L`, and the existence guard tests both: something *is* at that path, it just cannot be opened,
|
||||
and "exists but unreadable" is a real finding the check suite reports as a FAIL naming the file.
|
||||
Catching them in the existence guard would replace that FAIL with a false "does not exist".
|
||||
|
||||
`--help` is exit 0 and now works with a library missing. `validate-provenance.sh` sourced both
|
||||
provenance libraries at the top of the file, so a partial install turned the one command that
|
||||
explains how to use the script into an exit 2; the libraries are now loaded only when a mode's half
|
||||
of the usage is actually printed, and when one is gone that half says so instead of the whole command
|
||||
failing. It also separates an **empty** positional from **no** positional — `validate-provenance.sh
|
||||
""`, an unquoted variable that expanded to nothing, used to be reported as "only flags were given",
|
||||
which sent the reader after a flag they never typed. Both are exit 2, with different messages.
|
||||
|
||||
**All three entry scripts are CDPATH-safe.** `cd` **prints** the directory whenever CDPATH supplied
|
||||
it, so a bare `cd` on a path starting with neither `/` nor `.` can both emit an extra line into
|
||||
whatever captures it *and* resolve to an unrelated tree. Every such site now does
|
||||
`CDPATH='' cd -- …`: `SCRIPT_DIR` and the target's parent-directory name in `validate.sh` and
|
||||
`validate-provenance.sh`, and **both** sites in `vale-wrap.sh` — its default `--config` resolution,
|
||||
and the directory-mirror walk below. The two bare `cd`s left in `vale-wrap.sh` take absolute paths
|
||||
(`mktemp -d` output, and the mirror root derived from it), which CDPATH is never consulted for.
|
||||
|
||||
The second of `vale-wrap.sh`'s two sites is not an asset lookup: its directory mirroring runs
|
||||
`cd "$arg" && find -L . … -print0` on a *path argument*, which is relative whenever the caller passed
|
||||
a relative one. With an exported CDPATH holding a same-named directory, the `cd` emitted the decoy's
|
||||
path into the `-print0` stream and the mirror was built from the decoy's files. It is cleared the
|
||||
same way now. Only the `cd` changed: the walk still mirrors the whole tree, prunes `.git`, and
|
||||
follows symlinks with `find -L` because vale does.
|
||||
|
||||
**That one was latent, not a live gate defect.** Neither published Vale hook reaches it: both filter
|
||||
on `files:` patterns that match single markdown files, and only the `-d "$arg"` branch mirrors a
|
||||
directory. The exposed caller is the hand-invoked `vale-wrap.sh <dir>`.
|
||||
|
||||
## Current retrofit status
|
||||
|
||||
The ADR-0020 gates ship hot, with no baseline file — a shrinking baseline was considered and
|
||||
@@ -614,6 +676,22 @@ existed.
|
||||
every path it matches must be in that hook's own artifact class. A hook narrowed to zero files never
|
||||
runs, and pre-commit reports no error.
|
||||
|
||||
**Case 34** asks, statically and with no Vale binary, whether the shipped `.vale.ini` could load a
|
||||
style at all. Four assertions: every `[glob]` section declares a **non-empty** `BasedOnStyles`; every
|
||||
style any section names resolves to a real directory under `StylesPath`; that `StylesPath` is **not
|
||||
absolute**; and at least one `[glob]` section exists, so the check cannot pass vacuously on a config
|
||||
with nothing in it. A section whose `BasedOnStyles` is empty is the silent case — Vale lints every
|
||||
file that glob matches with no rule loaded, prints `0 errors` and exits 0. The absolute-path clause
|
||||
is the one that is not obvious: an absolute `StylesPath` passes on the machine that wrote it and
|
||||
hard-fails for every external consumer of `.pre-commit-hooks.yaml`, which is the only reason those
|
||||
styles ship at all. Part B is a mutation self-test against the same function Part A calls — it empties
|
||||
each section's `BasedOnStyles` in a copy of the assets, and absolutizes `StylesPath` in another
|
||||
pointed at that copy's own real `styles/` directory, and requires each to fail by name.
|
||||
|
||||
**It is not case 0 again.** Case 0 reads `BasedOnStyles` lines file-wide and tolerates an absolute
|
||||
`StylesPath`; case 34 reads them **per `[glob]` section** and rejects one. Case 0 checks the config
|
||||
loads; case 34 checks that loading it arms anything.
|
||||
|
||||
### What Vale owns, and what stays LLM judgment
|
||||
|
||||
Six rule files, six distinct rules:
|
||||
@@ -830,8 +908,8 @@ hook — is deleted with the second Vale copy (ADR-0025). The six glob probes su
|
||||
|
||||
`test-vale-wrap.sh` without Vale skips only its Vale-dependent cases, not the whole suite. The cases
|
||||
that are plain greps and awk over the config and the two hook manifests still run: case 0, 16, 26,
|
||||
27, the static halves of 28, 31 Parts A and B, 32 and 33. A static failure exits 1, because a real
|
||||
defect is not a setup error. Only an all-static-pass run exits 77.
|
||||
27, the static halves of 28, 31 Parts A and B, 32, 33 and 34. A static failure exits 1, because a
|
||||
real defect is not a setup error. Only an all-static-pass run exits 77.
|
||||
|
||||
### Mentioning banned phrasing without tripping the rule
|
||||
|
||||
@@ -866,11 +944,47 @@ what to install. (It was three until `test-check-vale-style-sync.sh` was deleted
|
||||
`echo "$OUT" | grep -q PATTERN`. Under `set -o pipefail` the pipe form fails depending on timing:
|
||||
`grep -q` exits on its first match, `echo` takes SIGPIPE on its next write, and pipefail reports that
|
||||
as the pipeline failing, so output that matched reads as "no match". It showed up as a push gate that
|
||||
failed about once in 670 runs, on a different suite each time. `tests/test-no-pipefail-early-exit-grep.sh`
|
||||
scans every tracked shell file that sets pipefail and fails on the pipe form. It does this for `echo`
|
||||
or `printf` piped into `grep` with `-q`, `-m`, `-l`, `-L`, `--quiet` or `--silent`. It checks its own
|
||||
scanner against fixtures before trusting a clean result. Pipes from other commands are out of scope.
|
||||
In practice they either absorb the writer's exit status with `|| true` or write only once, at exit.
|
||||
failed about once in 670 runs, on a different suite each time.
|
||||
|
||||
`tests/test-no-pipefail-early-exit-grep.sh` is the static guard. **What it scans** is the tracked
|
||||
`*.sh`, `*.bats` and `*.bash` files, minus itself — `git ls-files -- '*.sh' '*.bats' '*.bash'` — and,
|
||||
within each one that sets pipefail, it fails on the pipe form. Every tracked shell file in this tree
|
||||
carries one of those three extensions today, so "every tracked shell file" is true in effect, but it
|
||||
is a property of the tree and not of the scan: a shell script tracked under any other name is not
|
||||
reached. It checks its own scanner against fixtures before trusting a clean result.
|
||||
|
||||
**What the pattern models**, after a review widened it on four axes and narrowed it on one:
|
||||
|
||||
- **The reader** is `grep`, `egrep` or `fgrep` — behind a path prefix (`/bin/grep`), a `command`
|
||||
prefix, or env-var assignments (`LC_ALL=C grep`) — taking `-q`, `-m`, `-l`, `-L`, `--quiet`,
|
||||
`--silent`, `--max-count` or `--files-with`. (The last two match the regex and were missing from
|
||||
this list.)
|
||||
- **Intermediate stages are seen.** `echo x | filter | grep -q y` is a site; only the two-stage form
|
||||
used to be.
|
||||
- **Both line continuations are joined** before matching: a trailing backslash, and a trailing `|`,
|
||||
which is equally legal in a pipeline. The hit is reported at the line the command starts on.
|
||||
- **`pipefail` may sit anywhere in a `set` line**, so `set -o errexit -o pipefail` arms the file. It
|
||||
previously had to follow the *first* `-o`, and a file-level miss skips every site in that file
|
||||
rather than one — which is why that test is deliberately loose.
|
||||
- **A bare `&` ends a segment**, so `echo ok && other | grep -q x` — whose writer is `other`, not the
|
||||
`echo` — is not a site. `&` followed by a digit is kept, so `2>&1` does not end one.
|
||||
|
||||
The widening turned up **five live vulnerable sites in `tests/test-apm-current-hook.sh`** that the
|
||||
narrower scanner never saw while the suite reported green; all five are converted, and the scanner
|
||||
now reports zero over the tree. Its fixtures went from 4 vulnerable spellings to **12**, plus
|
||||
near-miss negatives it must leave alone, and the suite from 5 cases to **7**.
|
||||
|
||||
**Remit: early-exiting GREP readers only.** `head`, `sed -n 1p` and a bare `read` exit early too, and
|
||||
an `echo` or `printf` feeding any of them is the same race. Those are guarded by **convention** —
|
||||
absorb the writer's status with `|| true`, or take the verdict from a here-string — and deliberately
|
||||
not by this test: most legitimate uses of them in this tree are already absorbed, and the scanner
|
||||
cannot see absorption from the pipeline text alone, so flagging them would be noise. Two live
|
||||
`grep … | head -1` sites (`tests/test-vale-wrap.sh:620` and `:1046`) were fixed by hand with that
|
||||
idiom. Pipes from a non-builtin writer (`run_wrap … | grep -q`) are out of scope for the same reason:
|
||||
in practice they either absorb the writer's exit status with `|| true` or write only once, at exit.
|
||||
|
||||
**Known limitation: heredoc bodies are scanned as code.** A `cat <<'EOF'` body containing a
|
||||
vulnerable-looking line would be reported as a real site. There are none in the tree today.
|
||||
|
||||
`tests/run-bats.sh` derives the set of `.bats` files it expects from `git ls-files`, so a `.bats`
|
||||
file deleted from the worktree but still tracked in the index fails the run rather than silently
|
||||
|
||||
Reference in New Issue
Block a user