Defame1297
03abcffb77
diagnose read its feedback-loops reference unconditionally, so every invocation paid for guidance most runs never used; the read is conditional again and the per-invocation cost drops from 1,278 to 831 words. Its HITL template moves to assets/ because it is copied out, not read as reference. prototype's two branch flows move into references/ for the same reason — only one branch is ever taken. research could not search the codebase it was asked to research without Grep and Glob. caveman's description had grown into a paragraph where one sentence carries the trigger. Four references pointed at things that do not exist: a to-prd skill, a /setup-matt-pocock-skills command, two cross-skill ../ links that only resolve in the source tree, and two places calling this project's Gitea host GitHub. Addresses #114.
2.8 KiB
Constructing and sharpening a feedback loop
A feedback loop is a fast, deterministic, agent-runnable pass/fail signal for the bug. Build the right one and the bug is 90% fixed. This file covers the whole arc: building a loop, sharpening one you already have, and escalating when the bug resists reproduction.
Ways to construct one — try them in roughly this order
- Failing test at whatever seam reaches the bug — unit, integration, e2e.
- Curl / HTTP script against a running dev server.
- CLI invocation with a fixture input, diffing stdout against a known-good snapshot.
- Headless browser script (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
- Replay a captured trace. Save a real network request / payload / event log to disk; replay it through the code path in isolation.
- Throwaway harness. Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
- Property / fuzz loop. If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
- Bisection harness. If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can
git bisect runit. - Differential loop. Run the same input through old-version vs new-version (or two configs) and diff outputs.
- HITL bash script. Last resort. If a human must click, drive them with
assets/hitl-loop.template.shso the loop is still structured. Captured output feeds back to you.
Iterate on the loop itself
Treat the loop as a product. Once you have a loop, ask:
- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)
A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.
Intermittent bugs — raise the reproduction rate
If the loop only sometimes fails, the goal is not a clean repro but a higher reproduction rate. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.
When you genuinely cannot build a loop
Stop and say so explicitly. List what you tried. Ask the user for:
- access to whatever environment reproduces it,
- a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or
- permission to add temporary production instrumentation.
Do not proceed to hypothesise without a loop. A hypothesis you cannot falsify against a signal is a guess, and the fix that follows it is unverifiable.