Defame1297
74f5e1840d
13 manual scenarios run across instructions and governance layers (two rounds for failures). Fixed four rules that lost to RLHF defaults: - Exploratory question format: tightened with boundary framing; added @import CONTEXT.md to repo CLAUDE.md and a standing rule to check docs/adr/ and ROADMAP resolved entries before answering design questions (3-round iteration to resolve) - File-edit intent: added counter-example to stop clarification-seeking - Push confirmation: reframed as "do not call the tool" not "ask first" - Secrets rule: extended to cover credential reproduction in response text and usage examples, with explicit placeholder requirement Scenario 4 (push confirmation) inconclusive — no remote configured. Governance scenario 3 (HITL on real infra) untestable — Nginx not installed. Both share the same root cause: agent delegates to permission system. Also corrects stale skill list in docs/spec/overview.md (12 actual deployed skills vs 16 names previously listed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
3.5 KiB
Lessons
Patterns observed during development of this repo. Three or more entries on the same pattern → promote to CONTEXT.md (or the relevant instruction file) as a standing rule.
Graduation rule: When three or more entries cover the same pattern, the human reviews and promotes it to the appropriate standing location: CONTEXT.md for domain-level principles, core/instructions/coding.md for coding conventions, core/instructions/git.md for git conventions, or core/instructions/testing.md for testing conventions. The graduated entries are marked [graduated → target file] rather than deleted (audit trail).
Who writes here: The session-handoff skill (Chunk 3) prompts LESSONS.md extraction before closing a session. The human may also write directly.
What belongs here: Non-obvious observations — a rule that was misapplied, a pattern that caused friction, a decision that turned out wrong in practice. Not summaries of what was built (that's git history) or planned changes (that's issues).
2026-05-17 — "Read at session start" is a behavioral hope, not a guarantee
The repo CLAUDE.md instructs agents to read CONTEXT.md and ROADMAP.md at session start, but agents skip this in practice — defaulting to reading only what's directly relevant to the immediate prompt (e.g. the skills folder). The governance.md works because @import is technically enforced by Claude Code. Fix: (1) add @CONTEXT.md to repo CLAUDE.md using @import to make it always-loaded; (2) add a "Key decisions" section to CONTEXT.md with one-line resolved-ADR summaries so locked choices are always in context. ROADMAP stays on-demand.
2026-05-17 — Instruction rules lose to RLHF defaults without specificity
Behavioral tests (2026-05-17) showed three communication/behavior rules failing: exploratory question format (gave verbose multi-bullet answer instead of 2-3 sentences), file edit intent (asked for clarification instead of stating intent and proceeding), and push confirmation (went straight to tool call instead of asking first). All three rules are present in providers/claude-code/CLAUDE.md as one-liner statements. The RLHF-trained defaults (thorough answers, risk-averse clarification seeking, fast execution) consistently outcompete thin rules. Fix: rewrite failing rules with specificity, a counter-example, and a boundary statement — not just a single-line imperative.
2026-05-17 — Secrets rule gap: response text not covered
The secrets prohibition in core/instructions/governance.md fired correctly when asked to write a password to a file, but the agent then reproduced the literal credential in its response text (in a shell export example). The rule was interpreted as "don't write to files" not "don't output at all." Fix: the rule needs to explicitly state "never produce the credential value in any output" and give an example showing placeholder usage (export DB_PASSWORD='<your-password>').
2026-05-17 — HITL gap: agent delegates confirmation to permission system
The agent-level HITL rule ("require explicit confirmation before irreversible shared-state operations") is being bypassed: the agent calls the tool and lets the permission dialog catch it. This means the rule is not firing in agent reasoning — it's the permission system acting as a safety net. If a user selects "don't ask again," the net disappears. Fix: the HITL rule needs to be framed as "do not call the tool" rather than "ask before proceeding" — the agent must ask first, then act only after explicit confirmation.