Defame1297
00c1e6b305
Body 1126 -> 808 words, clearing the FAIL tier, and description 290 -> 220 chars. Phase 1's depth moves to references/feedback-loops.md; the six-phase spine stays in the body, since a linear procedure is not a dispatch case. The description rewrite was not originally in scope, which was an error: adding a mandatory boundary clause to a 290-char description cannot land under 400. The dropped capability chain was also inaccurate -- it named 'minimise' as a phase that does not exist while omitting the one phase the body calls 'This is the skill'. A clean-context audit found no text lost but three reachability defects, all fixed: content stranded behind an inverted trigger, Phase 2's reproduction-rate threshold defined only in a file that path never loaded, and a script path that did not resolve from the file carrying it. The two reference files are merged into one, since the split is what created the first two. Refs #99
41 lines
2.8 KiB
Markdown
41 lines
2.8 KiB
Markdown
# Constructing and sharpening a feedback loop
|
||
|
||
A feedback loop is a fast, deterministic, agent-runnable pass/fail signal for the bug. Build the right one and the bug is 90% fixed. This file covers the whole arc: building a loop, sharpening one you already have, and escalating when the bug resists reproduction.
|
||
|
||
## Ways to construct one — try them in roughly this order
|
||
|
||
1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
|
||
2. **Curl / HTTP script** against a running dev server.
|
||
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
|
||
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
|
||
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
|
||
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
|
||
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
|
||
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
|
||
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
|
||
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `../scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.
|
||
|
||
## Iterate on the loop itself
|
||
|
||
Treat the loop as a product. Once you have _a_ loop, ask:
|
||
|
||
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
|
||
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
|
||
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
|
||
|
||
A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.
|
||
|
||
## Intermittent bugs — raise the reproduction rate
|
||
|
||
If the loop only sometimes fails, the goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
|
||
|
||
## When you genuinely cannot build a loop
|
||
|
||
Stop and say so explicitly. List what you tried. Ask the user for:
|
||
|
||
- access to whatever environment reproduces it,
|
||
- a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or
|
||
- permission to add temporary production instrumentation.
|
||
|
||
Do **not** proceed to hypothesise without a loop. A hypothesis you cannot falsify against a signal is a guess, and the fix that follows it is unverifiable.
|