refactor(skills): retrofit the corpus to the ADR-0020 context contract (#129)
Retrofits all 39 skills to ADR-0020's description/body context contract, then fixes what six rounds of independent review found in that retrofit — including four ways the hot gate itself failed open. Closes #99, #107, #108, #110, #111, #114, #115, #120. ## The retrofit (waves 1-5) | | Start | Now | |---|---|---| | Description FAILs (>400 chars) | 26 | **0** | | Body FAILs (>900 words, body-only) | 9 | **0** | | Dangling routing targets | 2 | **0** | | `Kyberforge.CompositionNote` | 10 | **0** | | Preload tax | 21,005 chars | **~10,500** | Under the 12,000-char success criterion. Per-wave detail is on #99. ## The review fixes **The gate failed open four ways, three of them found after the retrofit shipped.** An unrecognised follower token made a dangling target vanish. A skill directory with no `SKILL.md` resolved as a valid target, so a commit could be green locally and red in a fresh clone — three existing fixtures were relying on that, one of which made the install-leak A/B pass vacuously. Then the free-standing `/name` sweep turned out to be gated on the sentence carrying a boundary marker, so route notation in any other sentence was invisible — not an ERROR, not a SUGGESTION, not an INFO — which left the documented "`/name` always blocks" promise false from a second direction. All four fixed and pinned. **Two checks were silently not running.** `validate-provenance.sh` checks 7-8 were dead across nine skills. Waking them exposed a deeper problem: they assume `Research doc:` names a source index, but 30 of 121 entries point at topic content documents, so every new check-7 INFO was a false positive and check 8 was saved from a false-FAIL flood only by an *unannounced* skip. Checks 7/8 are now scoped to source indexes and every skip announces itself (#121). **The retrofit's own anti-goal, four times.** ADR-0020 warns that a blunt gate gets satisfied by deleting content rather than relocating it. `diagnose` and `skill-audit` relocated prose and then read it unconditionally; `prototype` and `vale-config` deleted rules outright that survived nowhere. All four addressed. ## Verification - `bash tests/run-tests.sh --strict` — 24 suites, 0 skipped, 0 failed - `bash tests/run-bats.sh` — 325 tests, 0 failures - `pre-commit run --all-files` — 17/17 - `pre-commit run --hook-stage pre-push --all-files` — 16/16, with `apm marketplace check` and `apm pack --check-clean` run against the remote, not skipped - `scripts/skill-size-check.sh` over all 39 skills — rc 0, 0 ERROR/FAIL, SUGGESTION-only - Preload tax measured at **10,498 chars**, max description 390 — both inside budget - Every new test proven non-vacuous by a deliberate mutation of the behaviour it covers **Per-commit sync, stated accurately:** the ten commits from the latest review round each pass `check-plugin-content-sync` in isolation, verified by checking each out in a detached worktree with a clean between. The earlier gitea window (`dfacf05..bedbd1d`, nine commits) does **not** — its mirror was regenerated in one batch at `bbc7300`. An earlier revision of this description claimed the property held for every commit; it does not, and a bisect through that window lands on a red commit. **Squash-merge** to collapse it, or accept that this range is not bisectable. ## Version bump Six plugins and the catalog take a **patch**, not a minor. The branch is **89 commits — 40 `fix` / 30 `refactor` / 12 `docs` / 5 `chore` / 2 `test` — zero `feat`, zero `!`, zero `BREAKING CHANGE`** — and adds no skill, agent, command or hook. (Two earlier revisions of this section cited a stale histogram, most recently 78 commits; the figures above are measured at HEAD.) Both rules this repo ships (`forge/references/version-bump.md`, landing in this PR, and `git-commits/references/conventional-commits-spec.md`) make that a patch, and the catalog set is unchanged at 7 entries. Not settled by that: four published files were removed from the installed tree, three moved, and `caveman` gained `disable-model-invocation`, retiring its old triggers. Under a strict reading those are major-class and currently ship under `refactor:` with no marker. Whether the deployed skill surface is a public contract is written down nowhere — worth deciding, but it outlives this PR. ## Deliberately not in scope #112 (cherry-pick ownership, now resolved in favour of `git-commits`), #113 (`rtk git` normalisation), #116 (research fan-out), #101 (audit-skill merge), #122 (non-spec skill-root files), #123 (no PRD producer) stay open. #117 is the one worth reading: the contract's remedy is to move prose into `references/`, which is exactly where neither the size gate nor Vale looks — and the blind spot is wider than #117 currently records, since there is no root `.vale.ini` at all, so every ADR, `CONTEXT.md` and `README.md` is unlinted too. That blind spot let this branch carry two `level: error` `Kyberforge.SentenceOpenerThereIs` violations into `references/` files it created — `provider-adapter-author/references/provider-matrix.md:31` and `agent-audit/references/finding-criteria.md:95`. Both are reworded in `afadaae`, confirmed by routing each file through the audit's own `vale-wrap.sh` (1 error each before, 0 after). Five further occurrences sit in `references/` files already on `main`; those are the pre-existing corpus and stay with #117, which is the real fix. Also unfixed and not this PR's: `apm install` appends a duplicate `SessionStart` entry to `.claude/settings.json`, so a fresh clone cannot get pre-push green without an edit AGENTS.md warns against. Reproduces identically on `main`. Co-authored-by: Defame1297 <gitea@rkdr.net> Reviewed-on: https://git.dev.rkdr.net/Defame1297/holocron/pulls/129 Co-authored-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net> Co-committed-by: Claude Code AI - Gitea MCP <claude@noreply.git.dev.rkdr.net>
This commit was merged in pull request #129.
This commit is contained in:
35
plugins/bin/.apm/skills/diagnose/README.md
Normal file
35
plugins/bin/.apm/skills/diagnose/README.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# diagnose
|
||||
|
||||
A six-phase discipline for hard bugs and performance regressions: feedback loop → reproduce → hypothesise → instrument → fix with a regression test → clean up.
|
||||
|
||||
## What it does
|
||||
|
||||
Imposes an order of operations on debugging so the agent cannot skip to guessing. The load-bearing phase is the first one: build a fast, deterministic, agent-runnable pass/fail signal for the bug. Everything downstream — bisection, hypothesis testing, instrumentation — just consumes that signal, so the skill refuses to advance to Phase 2 without one, and says so explicitly rather than hypothesising blind.
|
||||
|
||||
The remaining phases each carry a constraint worth knowing about: hypotheses are generated 3–5 at a time and must be falsifiable, so the first plausible idea cannot anchor the whole investigation; every debug log is tagged with a unique prefix (`[DEBUG-a4f2]`) so cleanup is a single grep; the regression test is written before the fix and only at a seam that exercises the real bug pattern; and the run closes by asking what would have prevented the bug, handing off to `improve-codebase-architecture` when the answer is architectural.
|
||||
|
||||
Performance regressions take a branch of their own inside Phase 4 — baseline measurement and bisection, not logs.
|
||||
|
||||
## Conditional reading
|
||||
|
||||
Neither reference file is read on every run; `SKILL.md` names the condition for each.
|
||||
|
||||
- `references/feedback-loops.md` is read when Phase 1 has no signal yet, or when the loop you have is slow or intermittent.
|
||||
- `references/regression-seams.md` is read when Phase 5 leaves you unsure whether the available seam is deep enough — or whether one exists at all.
|
||||
|
||||
## Usage
|
||||
|
||||
```text
|
||||
/diagnose
|
||||
```
|
||||
|
||||
Describe the bug or the regression. For filing and triaging a reported bug rather than diagnosing it, use `triage`; for test-first feature work, use `tdd`.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | The six phases and their gates — what must be true before each one ends |
|
||||
| `references/feedback-loops.md` | Loaded when Phase 1 has no loop or the loop is too weak: ten ways to construct one ordered by cost, how to sharpen an existing loop, handling intermittent bugs, and what to ask the user for when the bug resists reproduction |
|
||||
| `references/regression-seams.md` | Loaded when Phase 5 is unsure about the seam: what makes a seam correct, the four shapes of a too-shallow seam, and what to do when no correct seam exists |
|
||||
| `assets/hitl-loop.template.sh` | Copy-and-edit bash template for the last-resort human-in-the-loop feedback loop, cited by `references/feedback-loops.md`. Provides `step` and `capture` helpers and prints captured values as `KEY=VALUE` for the agent to parse |
|
||||
@@ -1,6 +1,9 @@
|
||||
---
|
||||
name: diagnose
|
||||
description: Disciplined diagnosis loop for hard bugs and performance regressions. Reproduce → minimise → hypothesise → instrument → fix → regression-test. Use when user says "diagnose this" / "debug this", reports a bug, says something is broken/throwing/failing, or describes a performance regression.
|
||||
description: >
|
||||
Use when the user says "diagnose this" or "debug this", reports something
|
||||
broken, throwing, or failing, or says something got slow. Not filing or
|
||||
triaging a reported bug -> `triage`. Not test-first feature work -> `tdd`.
|
||||
---
|
||||
|
||||
# Diagnose
|
||||
@@ -15,40 +18,11 @@ When exploring the codebase, use the project's domain glossary to get a clear me
|
||||
|
||||
Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**
|
||||
|
||||
### Ways to construct one — try them in roughly this order
|
||||
**If you do not yet have such a signal, read `references/feedback-loops.md`** — ten ways to build one ordered by cost, and what to ask the user for when the bug resists reproduction entirely.
|
||||
|
||||
1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
|
||||
2. **Curl / HTTP script** against a running dev server.
|
||||
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
|
||||
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
|
||||
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
|
||||
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
|
||||
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
|
||||
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
|
||||
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
|
||||
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
|
||||
**If you do have one, it is probably not sharp enough yet.** Make it faster and more deterministic, and make it assert on the exact symptom rather than "didn't crash" — a 30-second flaky loop is barely better than no loop. If it stays slow or intermittent after that, read that file's "Iterate on the loop itself" and "Intermittent bugs" sections.
|
||||
|
||||
Build the right feedback loop, and the bug is 90% fixed.
|
||||
|
||||
### Iterate on the loop itself
|
||||
|
||||
Treat the loop as a product. Once you have _a_ loop, ask:
|
||||
|
||||
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
|
||||
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
|
||||
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
|
||||
|
||||
A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.
|
||||
|
||||
### Non-deterministic bugs
|
||||
|
||||
The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
|
||||
|
||||
### When you genuinely cannot build a loop
|
||||
|
||||
Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.
|
||||
|
||||
Do not proceed to Phase 2 until you have a loop you believe in.
|
||||
Do not proceed to Phase 2 until you have a loop you believe in. If you cannot build one, stop and say so explicitly, listing what you tried — never hypothesise without a signal.
|
||||
|
||||
## Phase 2 — Reproduce
|
||||
|
||||
@@ -57,7 +31,7 @@ Run the loop. Watch the bug appear.
|
||||
Confirm:
|
||||
|
||||
- [ ] The loop produces the failure mode the **user** described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
|
||||
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
|
||||
- [ ] The failure is reproducible across multiple runs. If it is intermittent, `references/feedback-loops.md` defines the rate high enough to debug against — go back to Phase 1 and raise it.
|
||||
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.
|
||||
|
||||
Do not proceed until you reproduce the bug.
|
||||
@@ -90,19 +64,17 @@ Tool preference:
|
||||
|
||||
## Phase 5 — Fix + regression test
|
||||
|
||||
Write the regression test **before the fix** — but only if there is a **correct seam** for it.
|
||||
Write the regression test **before the fix** — but only at a **correct seam**: one where the test exercises the real bug pattern as it occurs at the call site. If the available seam looks too shallow, or you cannot tell whether it is, read `references/regression-seams.md`.
|
||||
|
||||
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.
|
||||
**If no correct seam exists, that itself is the finding.** Note it and carry it into Phase 6 — the architecture is preventing the bug from being locked down.
|
||||
|
||||
**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.
|
||||
At a correct seam:
|
||||
|
||||
If a correct seam exists:
|
||||
|
||||
1. Turn the minimised repro into a failing test at that seam.
|
||||
1. Turn the Phase 1 loop into a failing test at that seam, narrowed to the symptom captured in Phase 2.
|
||||
2. Watch it fail.
|
||||
3. Apply the fix.
|
||||
4. Watch it pass.
|
||||
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.
|
||||
5. Re-run the Phase 1 feedback loop against the original, un-narrowed scenario.
|
||||
|
||||
## Phase 6 — Cleanup + post-mortem
|
||||
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Constructing and sharpening a feedback loop
|
||||
|
||||
A feedback loop is a fast, deterministic, agent-runnable pass/fail signal for the bug. Build the right one and the bug is 90% fixed. This file covers the whole arc: building a loop, sharpening one you already have, and escalating when the bug resists reproduction.
|
||||
|
||||
## Ways to construct one — try them in roughly this order
|
||||
|
||||
1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
|
||||
2. **Curl / HTTP script** against a running dev server.
|
||||
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
|
||||
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
|
||||
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
|
||||
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
|
||||
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
|
||||
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
|
||||
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
|
||||
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `assets/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
|
||||
|
||||
## Iterate on the loop itself
|
||||
|
||||
Treat the loop as a product. Once you have _a_ loop, ask:
|
||||
|
||||
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
|
||||
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
|
||||
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
|
||||
|
||||
A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.
|
||||
|
||||
## Intermittent bugs — raise the reproduction rate
|
||||
|
||||
If the loop only sometimes fails, the goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
|
||||
|
||||
## When you genuinely cannot build a loop
|
||||
|
||||
Stop and say so explicitly. List what you tried. Ask the user for:
|
||||
|
||||
- access to whatever environment reproduces it,
|
||||
- a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or
|
||||
- permission to add temporary production instrumentation.
|
||||
|
||||
Do **not** proceed to hypothesise without a loop. A hypothesis you cannot falsify against a signal is a guess, and the fix that follows it is unverifiable.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Judging a regression-test seam
|
||||
|
||||
Read this when Phase 5 leaves you unsure whether the seam available for the regression test is the correct one — either because the obvious seam looks shallow, or because there appears to be no seam at all.
|
||||
|
||||
## What makes a seam correct
|
||||
|
||||
A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site: the same entry point, the same participants, the same ordering, and the same state the real caller holds when it goes wrong.
|
||||
|
||||
## Seams that are too shallow
|
||||
|
||||
- A single-caller test when the bug only appears with multiple callers.
|
||||
- A unit test that cannot replicate the chain of calls that triggered the bug.
|
||||
- A test that reproduces the symptom by construction — asserting on a value the test itself set — rather than by driving the code path that produces it.
|
||||
- A test that mocks out the collaborator the bug actually lives in.
|
||||
|
||||
A regression test at a shallow seam gives false confidence. It passes forever, including after a change reintroduces the bug at the real call site, and it will be read by the next maintainer as proof the bug is locked down.
|
||||
|
||||
## When there is no correct seam
|
||||
|
||||
Do not force one, and do not settle for a shallow seam to have something green. Instead:
|
||||
|
||||
1. Apply the fix and verify it against the Phase 1 loop directly.
|
||||
2. Write down which seams you considered and why each was too shallow.
|
||||
3. Carry that into Phase 6's "what would have prevented this bug" question. A missing seam is an architecture finding — tangled callers, hidden coupling, or a module with no testable boundary — and the handoff is the `improve-codebase-architecture` skill, with those specifics attached.
|
||||
Reference in New Issue
Block a user