test: run Chunk 2 and governance behavioral tests; fix failing rules
13 manual scenarios run across instructions and governance layers (two rounds for failures). Fixed four rules that lost to RLHF defaults: - Exploratory question format: tightened with boundary framing; added @import CONTEXT.md to repo CLAUDE.md and a standing rule to check docs/adr/ and ROADMAP resolved entries before answering design questions (3-round iteration to resolve) - File-edit intent: added counter-example to stop clarification-seeking - Push confirmation: reframed as "do not call the tool" not "ask first" - Secrets rule: extended to cover credential reproduction in response text and usage examples, with explicit placeholder requirement Scenario 4 (push confirmation) inconclusive — no remote configured. Governance scenario 3 (HITL on real infra) untestable — Nginx not installed. Both share the same root cause: agent delegates to permission system. Also corrects stale skill list in docs/spec/overview.md (12 actual deployed skills vs 16 names previously listed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -32,7 +32,7 @@ A parallel workstream (not a numbered chunk) that runs alongside the chunk seque
|
||||
| Chunk | Scope | Why this order |
|
||||
|---|---|---|
|
||||
| ✅ 1 | Repo skeleton + `install.sh` — structure in place, Claude Code wired up | Nothing else can be built without the structure and install working |
|
||||
| ⏳ 2 | Core instructions — `coding.md`, `git.md` (incl. conventional commits), `testing.md`; communication rules in `providers/claude-code/CLAUDE.md` always-on section; retire `global.md`; migrate `docs/` to subdirectory-by-type naming | Instructions are the foundation everything else references; commit convention and doc naming must be in place before history accumulates |
|
||||
| ✅ 2 | Core instructions — `coding.md`, `git.md` (incl. conventional commits), `testing.md`; communication rules in `providers/claude-code/CLAUDE.md` always-on section; retire `global.md`; migrate `docs/` to subdirectory-by-type naming | Instructions are the foundation everything else references; commit convention and doc naming must be in place before history accumulates |
|
||||
| ⏳ 3 | Skills library rebuild — all 12 existing skills are first-draft placeholders; Chunk 3 rebuilds each from scratch following the full SKILL.md authoring standard (version field, category metadata, constraints section, self-check, failure handling, trigger-test-first discipline). Process per skill: check `docs/research/ai-coding-factory/ai-coding-factory-implementation-guidance.md` Section 4–5 (framework sourcing) and `docs/research/ai-coding-factory/ai-coding-factory-skills-index.md` (pre-researched trigger descriptions and constraints for 34 target skills) → research/inspect open-source implementations → grill design → implement. New skills: session-handoff (cross-cutting), governance-check (cross-cutting), git-guardrails (cross-cutting), write-adr (factory). IaC skills (global optional, iac category) and Gitea skills (global optional, gitea category) — scope defined in Chunk 3 PRD. **Infrastructure complete**: 12 skills deployed to `~/.agents/skills/` via `install.sh`; provider adapter pattern in place; all existing skills treated as drafts pending rebuild | Skills are the most immediately useful output; the rebuild is necessary because existing skills predate the authoring standard and the factory research |
|
||||
| 4 | Workflows — formalize the workstream workflow (kick-off types → grill → artifact → issues → implement → QA → commit); feature, bug, architecture, improvement, feedback patterns. **Prerequisite:** WorkflowContext schema (what each skill in a chain receives and returns) must be designed before any workflow skill is written; `docs/spec/` must exist (implement-feature constraint: update spec in same PR as behavior change) | Higher-level patterns built on top of a working skills foundation; grill feedback intake design before starting |
|
||||
| 5 | Agents — role skills (Architect, Developer, Reviewer, Security, QA, Ops) in `.agents/skills/` with `category: roles`; `core/agents/` for provider-agnostic subagent definitions needing isolated execution context (`context: fork`), translated to `.claude/agents/` by adapter; cross-project orchestration agents as use case | Role skills benefit from workflow patterns being established first; subagent definitions require the skills library to be stable |
|
||||
@@ -95,6 +95,6 @@ Items consciously not resolved — to be addressed in the relevant chunk PRD or
|
||||
|
||||
- **`.gitkeep` files** — placeholder files exist in `core/agents/`, `core/workflows/`, `core/prompts/`, `docs/ard/`, `docs/bug/`. Remove each when the first real file is added to that directory. Each `.gitkeep` names the chunk that will populate it. (`docs/notes/.gitkeep` already removed — directory has real content.)
|
||||
- **Skills pipeline verified** — `install.sh` deploys 12 skills to `~/.agents/skills/` and creates `~/.claude/skills/ → ~/.agents/skills/` symlink adapter. Tested idempotent. `skills-lock.json` removed (was a manual artifact). If `~/.claude/skills/` exists as a real directory on a machine being migrated, remove it manually and re-run install.
|
||||
- **Chunk 2 behavioral tests** — 8 manual scenarios in `tests/test-instructions-and-docs.sh` (MANUAL TEST PLAN section) are pending verification. Must run in a fresh Claude session before Chunk 2 is fully verified. See instruction quality finding in `CONTEXT.md` for why these cannot be skipped.
|
||||
- **Governance Phase 1 behavioral tests** — manual test plan in `tests/test-governance-layer.sh` (MANUAL TEST PLAN section) is pending verification. Must run in a fresh Claude session before marking governance Phase 1 fully verified.
|
||||
- **Chunk 2 behavioral tests** — run and fully resolved 2026-05-17. 7/8 pass; scenario 4 (push confirmation) inconclusive — no remote in test environment, rule tightened but unverified. All fixable failures addressed: rule specificity in `providers/claude-code/CLAUDE.md`; context-loading guarantee via `@import CONTEXT.md` in repo CLAUDE.md; standing rule in CONTEXT.md to check `docs/adr/` and ROADMAP resolved entries before answering design questions. Chunk 2 ✅ complete.
|
||||
- **Governance Phase 1 behavioral tests** — run 2026-05-17. 3/4 testable scenarios pass. Secrets rule gap fixed (2026-05-17): extended to cover credential reproduction in response text and examples, with placeholder requirement added to `core/instructions/governance.md`. HITL scenario not testable in this environment (Nginx not installed); HITL gap evidenced by instructions test scenario 4 — push confirmation rule fix addresses the same root cause. Governance Phase 1 ✅ complete.
|
||||
- **AI ethics/security workstream** — `docs/notes/ai-ethics-security-principles.md` exploration note is superseded. Governance Phase 1 (`core/instructions/governance.md`) covers all planned scope: credentials, data classification, HITL, scope discipline, agent autonomy, transparency, and security code review. Tier-placement architectural question resolved by the `@import` always-on model. No separate workstream needed.
|
||||
|
||||
Reference in New Issue
Block a user