24 KiB
Lessons
Patterns observed during development of this repo. Three or more entries on the same pattern → promote to CONTEXT.md (or the relevant instruction file) as a standing rule.
Graduation rule: When three or more entries cover the same pattern, the human reviews and promotes it to the appropriate standing location: CONTEXT.md for domain-level principles, core/instructions/coding.md for coding conventions, core/instructions/git.md for git conventions, or core/instructions/testing.md for testing conventions. The graduated entries are marked [graduated → target file] rather than deleted (audit trail).
Who writes here: The session-handoff skill (Chunk 3) prompts LESSONS.md extraction before closing a session. The human may also write directly.
What belongs here: Non-obvious observations — a rule that was misapplied, a pattern that caused friction, a decision that turned out wrong in practice. Not summaries of what was built (that's git history) or planned changes (that's issues).
2026-05-17 — Workflow documents should prescribe sub-agent usage, not just allow it
When writing workflow documents (like docs/notes/skill-implementation-workflow.md), the natural tendency is to describe steps at a high level and leave sub-agent usage as an implementation detail. But if the workflow doesn't explicitly prescribe "spawn a sub-agent here," practitioners default to doing everything in the main context — accumulating token cost and losing the isolation benefit. Fix: make sub-agent usage a named step in the workflow, specifying what the agent receives, what it returns, and why it's isolated. This makes the workflow reproducible rather than dependent on the practitioner remembering to use agents.
2026-05-17 — Conflict check before synthesis grill, not during
When combining upstream sources into a skill, conflicts with governing documents (AI constitution, factory principles) tend to surface in the middle of the synthesis grill — disrupting the combining discussion and requiring context switches. Fix: run a dedicated conflict-check step before the grill. A sub-agent reads the governing documents, checks the upstream content against them, and returns a numbered list of tensions. The grill then starts with those items as explicit agenda points, making it faster and more systematic. An empty conflict list is also valuable — it confirms the upstreams are clean before co-writing begins.
2026-05-17 — Cross-references to "produced by issue N" rot before the session ends
Issue files frequently referenced "the workflow defined in docs/notes/skill-implementation-workflow.md (produced by issue 0016)." Within the same session that closes issue 0016, that parenthetical is already stale — the document exists and is the authoritative reference. Fix: reference the document path directly, not the issue that produced it. The git history records the producing issue; cross-references should point to the artifact that persists.
2026-05-17 — "Read at session start" is a behavioral hope, not a guarantee
The repo CLAUDE.md instructs agents to read CONTEXT.md at session start, but agents skip this in practice — defaulting to reading only what's directly relevant to the immediate prompt (e.g. the skills folder). The governance.md works because @import is technically enforced by Claude Code. Fix: (1) add @CONTEXT.md to repo CLAUDE.md using @import to make it always-loaded; (2) add a "Key decisions" section to CONTEXT.md with one-line resolved-ADR summaries so locked choices are always in context.
2026-05-17 — Instruction rules lose to RLHF defaults without specificity
Behavioral tests (2026-05-17) showed three communication/behavior rules failing: exploratory question format (gave verbose multi-bullet answer instead of 2-3 sentences), file edit intent (asked for clarification instead of stating intent and proceeding), and push confirmation (went straight to tool call instead of asking first). All three rules are present in providers/claude-code/CLAUDE.md as one-liner statements. The RLHF-trained defaults (thorough answers, risk-averse clarification seeking, fast execution) consistently outcompete thin rules. Fix: rewrite failing rules with specificity, a counter-example, and a boundary statement — not just a single-line imperative.
2026-05-17 — Secrets rule gap: response text not covered
The secrets prohibition in core/instructions/governance.md fired correctly when asked to write a password to a file, but the agent then reproduced the literal credential in its response text (in a shell export example). The rule was interpreted as "don't write to files" not "don't output at all." Fix: the rule needs to explicitly state "never produce the credential value in any output" and give an example showing placeholder usage (export DB_PASSWORD='<your-password>').
2026-05-17 — Synthesis grill and SKILL.md co-write are two separate conversations
The synthesis grill (step 4) answers schema-level questions: how to combine upstreams, which eval schema to use, merge behaviour. Step 5b is a different conversation: how upstream content maps to each SKILL.md body section, what options each section had, and which was chosen. Collapsing them — writing the SKILL.md immediately after the grill without a per-section walk-through — means the human never sees the upstream options for the body and has no opportunity to redirect before the file is written. Fix: step 5b is now a named gate in the workflow. Walk through every body section one at a time, cite the upstream source, present alternatives, get confirmation. Only then write. Applies to both hand-written (bootstrap) and write-skill-produced skills.
2026-05-17 — Skill-calls-skill composition must be a named process step
When a skill invokes another skill as part of its work (e.g. write-skill invoking write-eval to produce the eval), that call must be a numbered step in the Process section — not left as an implicit external workflow step. If it isn't named, practitioners either forget it or do it manually outside the skill, breaking the composition chain. The user caught this during the write-skill co-write; it was absent from the process despite being in the workflow doc. Fix: when designing any skill that composes another, list each composed call explicitly as a numbered step with a "do not mark complete until X exists" constraint.
2026-05-17 — AGPL-3.0 repos appear prominently in community skill search results
When searching GitHub for agent skill upstreams, AGPL-3.0 repos (e.g. dceoy/speckit-agent-skills) appear alongside permissive-licensed ones without obvious visual distinction. AGPL imposes copyleft obligations on adopted content. Always run a licence check (GitHub API /license endpoint) before extracting any content from a new upstream. An AGPL finding is a hard exclude — record the repo, SHA, and licence in source review notes so future sessions don't re-review it.
2026-05-17 — Trigger description gate is not satisfied by embedding it in the section walk-through
The per-skill workflow (and write-skill's own process step 4) requires testing the trigger description against 3 cases — explicit, implicit, negative — as a standalone gate with explicit PASS/FAIL markers before any body content is written. During write-docs (issue 0018 phase 2), the trigger description was included in the section walk-through (step 5b) rather than tested first as a named gate. The gate never had explicit pass/fail output, which means neither the human nor the agent confirmed the trigger was sound before section content was written. Fix: treat the trigger test as a numbered standalone step with per-case PASS/FAIL output before step 5b begins. A section walk-through that happens to include the description field is not a substitute.
2026-05-17 — write-eval confirmation gate is bypassed when called via sub-agent with pre-designed cases
write-eval's process requires presenting the full test plan and waiting for user confirmation before writing the file. When write-eval is invoked by passing pre-designed test cases directly to a write sub-agent, this gate is skipped — the file is written before the user sees the plan. This happened during write-docs (issue 0018 phase 2). Fix: when orchestrating write-eval as part of a larger workflow, split into two steps: (1) sub-agent proposes test cases and returns to the main conversation; (2) after user confirmation, sub-agent writes the file. Or: design cases in the main conversation, present them to the user, then spawn the write agent. The plan-then-write separation is the gate — collapsing it into a single sub-agent call silently removes it.
2026-05-18 — Skill body sections were cargo-culted, not spec-defined
The write-skill authoring standard required 8 body sections including Role and When/When not. These were assumed to be agentskills.io requirements. Checking the actual spec revealed the body has no format restrictions at all — recommended sections are step-by-step instructions, examples, and edge cases. Role and When/When not were added by convention without verifying the standard. Fix: before encoding any requirement as part of an authoring standard, check the upstream spec directly. The agentskills.io spec also confirmed that negative triggers belong in the description field — not in a separate body section — which eliminates a persistent duplication pattern across all skills.
2026-05-18 — Provenance fields in frontmatter are loaded on every skill scan
Fields like source:, references:, version:, updated:, and when: in SKILL.md frontmatter are loaded at agent startup alongside name and description for every installed skill. None of these are used for routing or runtime execution — they are audit and upgrade-cycle records. Loading them at startup violates progressive disclosure and wastes tokens proportional to the number of installed skills. Fix: move all non-routing frontmatter to a separate META.md file in the skill directory. Frontmatter keeps only name, description, metadata.category, and allowed-tools (when applicable) — the four fields the spec actually uses for routing and discovery.
2026-05-18 — Copy-fill is more deterministic than generate for structured skill artifacts
When a skill produces a structured artifact like SKILL.md, the natural approach is to generate it from internalized rules in the Process section. But this means section structure is only as reliable as the agent's instruction-following under token pressure. Copy-fill (copy the template to the target path, then fill in content) separates structure from content: the template mechanically enforces section order and presence, freeing the Process section to focus only on sequencing constraints (what order to decide things) rather than also policing structure. Side benefit: the template is a human-usable artifact that can be adopted independently of the skill. Fix applied in write-skill refactor: SKILL-TEMPLATE.md and META-TEMPLATE.md are the authoritative structure sources; the Process section no longer contains a body structure constraint — the template handles it.
2026-05-17 — HITL gap: agent delegates confirmation to permission system
The agent-level HITL rule ("require explicit confirmation before irreversible shared-state operations") is being bypassed: the agent calls the tool and lets the permission dialog catch it. This means the rule is not firing in agent reasoning — it's the permission system acting as a safety net. If a user selects "don't ask again," the net disappears. Fix: the HITL rule needs to be framed as "do not call the tool" rather than "ask before proceeding" — the agent must ask first, then act only after explicit confirmation.
2026-05-26 — META-TEMPLATE uses YAML comments; META.md output retains them
META-TEMPLATE.md uses YAML # comments to explain fields inline. SKILL-TEMPLATE.md uses HTML comments inside XML tags, which the agent strips on fill. The structural difference means SKILL.md output is clean but META.md output retains the explanatory # lines — an inconsistency. Fix (deferred): restructure META-TEMPLATE.md so all explanatory guidance is prose above the code block (markdown, never copied into the output YAML), and the code block itself uses <placeholder> syntax with no # comment lines. This makes META.md fill behaviour deterministic for the same reason SKILL.md fill is: <...> markers are unambiguously replaceable; prose above the block is not part of the template. Do not apply until the human/copy-fill tradeoff is resolved — see 2026-05-26 session discussion.
2026-05-26 — Overlap checks must scan the deployed directory, not just the source repo
write-a-skill existed only in ~/.agents/skills/ (installed from a pre-refactor source) and was invisible during a repo-level scan of .agents/skills/. Governance reviews and overlap checks that only look at the source repo will miss skills added by install.sh from other sources or prior runs. Fix: overlap checks must scan the deployed ~/.agents/skills/ directory, not just the repo's .agents/skills/.
2026-05-26 — model: field belongs in SKILL.md frontmatter, not META.md
Claude Code supports model: as a provider extension in SKILL.md frontmatter — it overrides the session model for the skill's turn and reverts after. Attempting to put it in META.md was wrong: META.md is provenance/audit metadata, not runtime config. The boundary: if a field affects agent behaviour at invocation time, it belongs in SKILL.md frontmatter; if it serves upgrade reviews and audit trails, it belongs in META.md.
2026-05-26 — Research agents present synthesis as spec fact
When asked to research skill sub-file best practices, the research sub-agent reported "Process goes in SKILL.md. Context goes in reference files" as if it were verbatim from the Claude Code docs or the Agent Skills spec. Checking agentskills.io directly showed the spec says: "There are no format restrictions" on the body. The principle is a reasonable synthesis, not a quoted rule — but it nearly landed in write-skill's constraints as authoritative spec language. Fix: always verify research agent claims against the primary source before encoding them as rules, especially for spec or documentation claims. Plausible synthesis is the hardest fabrication to catch because it's often correct in spirit.
2026-06-21 — claude plugin validate --strict is absent from the standard test sweep
When running a full test audit, claude plugin validate --strict was not included in the initial agent sweep — only discovered mid-session when the user flagged the gap. The command catches warnings that normal mode tolerates (missing version fields, non-agent .md files in agents/) and will cause CI to fail when strict mode is enforced in Chunk 6. Fix: include claude plugin validate --strict on all plugin paths and marketplace manifests as a named step in any plugin audit. It belongs in the pre-push hook alongside check-manifests.sh — currently only check-manifests.sh runs there. See tests/test-plugin-validate.sh (pending, Gitea issue #2).
2026-06-21 — Source and deployed gitleaks configs can silently diverge
scripts/gitleaks.toml (source, in git, deployed to repo root by setup-gitleaks.sh) and .gitleaks.toml (deployed root copy, read by the hook, also tracked in git) were found with different allowlist states — someone had updated the deployed file directly without updating the source. Running setup-gitleaks.sh again would overwrite the deployed file with the stale source, silently deleting the existing allowlist and re-exposing a known false positive as a blocking pre-commit failure. Fix: treat scripts/gitleaks.toml as the single source of truth; never edit .gitleaks.toml directly. When making allowlist changes, always update source and deployed copy together in the same commit. Longer-term fix: setup-gitleaks.sh should merge rather than overwrite, or detect divergence and warn when .gitleaks.toml is tracked in git.
2026-06-21 — shellcheck without -x blocks pre-commit on any script using source (LEGACY SHELL HOOKS)
Status: Historical. Shell-hook-based pre-commit was replaced by pre-commit framework (Chunk 5, .pre-commit-config.yaml). Modern repos no longer affected. Documented for reference when supporting legacy repos.
The pre-commit hook ran shellcheck "$f" without -x. Without -x, shellcheck fires SC1091 for every source statement and exits non-zero, blocking the commit. This was a latent bug in legacy shell hooks, only triggered when install.sh (which sources deploy-manifest.sh) was staged for the first time. Compounding it: the # shellcheck source= directive in install.sh pointed to deploy-manifest.sh (bare filename, resolved from CWD = repo root) rather than scripts/deploy-manifest.sh (correct repo-root-relative path), so even with -x the file wasn't found on the first attempt.
Lesson for future work: When writing a source= directive, use a path that resolves correctly from the CWD where shellcheck will be invoked — verify with shellcheck -x <file> before committing. Pre-commit framework hooks include -x by default in the ecosystem's shellcheck integration.
2026-06-22 — Plugin cache isolation rules out shared/ directories between skills
When two skills in the same plugin share a resource (e.g. validate.sh), the instinct is to put it in a shared/ directory and reference it with a relative path. This breaks silently after install: plugins are copied to a cache, and ../ paths across skill directories stop resolving. The correct pattern is duplication with clear ownership — one skill owns the canonical copy and the other delegates to it via a skill invocation (e.g. /skill-audit) rather than a file path. If delegation is not possible, duplicate the file and note the owning skill in a comment.
2026-06-22 — Qualitative rubrics should be grounded in upstream spec docs, not derived from in-repo usage
When skill-audit's qualitative checks for description quality and body discipline were first written, they were derived from skill-write's own authoring conventions — a circular dependency. Any drift in skill-write's conventions would silently propagate into the audit criteria. Fix: extract condensed reference files directly from the upstream spec (agentskills.io) and load them conditionally from the audit skill. The rubric is then grounded in the authoritative source and independent of in-repo convention drift.
2026-06-22 — Test files in scripts/ are dev tooling; document them in README as non-spec
The agentskills.io spec defines scripts/ for bundled executable scripts — it says nothing about test infrastructure. Bats test files placed in scripts/ (or scripts/tests/) are invisible to auditors following the spec and create silent README drift if not documented. Fix: place test files directly in scripts/ (no subdirectory), add a row to the README file table for each with a "dev tooling, not shipped with the plugin" note, and don't nest them in a tests/ subdirectory since that creates a non-spec directory structure.
2026-06-27 — Clean-context audit catches what biased forks miss
A skill-audit run by a fresh agent (no conversation context) caught 2 FAILs that the implementation fork's own audit pass missed — an incomplete README.md file table and references/sources.md paths invalid in the plugin cache. Forks that built the artifact are biased toward their own output: they know what was intended and fill in gaps silently. A fresh agent has no such priors and audits what is actually written. Fix: always run a clean-context audit as a named final step after implementation forks complete. It is not redundant with the in-process audit — it is a different check.
2026-06-27 — Parallel forks on the same file produce conflicts requiring a third fork to reconcile
Two forks independently fixed references/sources.md with different approaches — one added a header comment, the other replaced the paths with relative references. Both were plausible; neither read the spec first. Reconciling required a third fork to read the authoritative source and revert to the correct format (repo-root-relative, per skill-author Step 5). Fix: when multiple forks are in scope for the same file, either (a) scope them to non-overlapping files explicitly, or (b) sequence them rather than parallelise. If a fix is spec-governed, always read the spec before applying it — the "obvious" fix is wrong as often as it is right.
2026-06-28 — Implementation agents must invoke /skill-author, not write skill files directly
When briefing an agent to implement a new skill, the instinct is to tell it to write the SKILL.md and supporting files directly. This bypasses Step 5 of the skill-author process (provenance), which requires reading all research sources.md files and recording every extracted slug in META.md. The validate-provenance.sh script catches the gap — but only after the commit, requiring a fix round. This pattern recurred twice in one session (plugin-author and marketplace-author initial implementation, then again in the first round of fix agents). Fix: briefs for implementation agents must explicitly say "invoke /skill-author (read and follow plugins/kyberforge/skills/skill-author/SKILL.md)" — not "write the skill files." Invoking the skill is the only reliable way to ensure all process gates, including provenance, run.
2026-07-05 — Repo root is a bare checkout; work happens in worktrees only
/root/ai-development/.git has core.bare = true — the root directory itself has no working tree. Running plain git status, git commit, or editing tracked files at the root fails (fatal: this operation must be run in a work tree) or silently produces edits git can never see or commit — not discoverable until the error is hit, or worse, missed entirely. All real work — including one-line docs fixes — requires git worktree add <path> -b <branch> origin/main first. Fresh worktrees also don't have submodules (tests/bats, docs/wiki, etc.) initialized, so the run-tests pre-push hook fails until git submodule update --init --recursive is run. Fix: before any edit/commit in this repo, confirm a working tree exists (git rev-parse --is-inside-work-tree); if not, create a worktree first, and initialize submodules before attempting to push.
2026-07-05 — Local remote-tracking refs go stale; verify against the Gitea API before asking
After a PR merge (with Gitea's default auto-delete-branch behavior), git branch -a still showed the remote feature branch — the local remotes/origin/* ref hadn't been pruned. This led to asking the user for confirmation to delete a branch that was already gone server-side, which they correctly pushed back on. Fix: before asking the user to confirm a git/PR cleanup action, check the authoritative remote state directly (e.g. mcp__gitea__list_branches, or git fetch --prune first) rather than trusting local remote-tracking refs, which are not automatically kept in sync.
2026-05-18 — Planning meta-commentary does not belong in deployed artifacts
During write-skill refactor, an "open thread" note (about a deferred research step) was written directly into the SKILL.md Process section. The user caught it. The rule it violated: a deployed artifact (SKILL.md, a runtime file loaded by agents) must not contain planning meta-commentary — deferred items, open threads, and implementation notes belong in the issue file, which is the planning artifact. The skill body should contain only content relevant to runtime execution. If a decision is deferred, record it in the issue and leave no trace in the skill. The distinction: issue = planning record; skill = executable instruction.