docs: add ai-coding-factory research, gap analysis, and grill-me scope
Add research exploration notes (principles, challenges, implementation guidance, skills index, session log) and gap/conflict analysis against the current repo vision and roadmap. Sharpen the roadmap housekeeping item with the grill-me intent, central scope question, and expected output (ADR + updated chunk scope for 2, 3, and 4). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
191
docs/research/ai-coding-factory/ai-coding-factory-challenges.md
Normal file
191
docs/research/ai-coding-factory/ai-coding-factory-challenges.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# AI Coding Factory — Open Questions & Research Challenges
|
||||
|
||||
**Purpose:** Agenda for the deep research session. Each item identifies a finding, assumption, or claim from the initial factory research that warrants deeper investigation, challenge, or independent quantification. Ordered by design impact — highest-stakes uncertainties first.
|
||||
|
||||
**How to use:** Upload alongside `ai-coding-factory-research.md` and `ai-coding-factory-session.md` to continue in a fresh session. Each challenge should be web-searched against independent sources before accepting or correcting the current position. Any finding refuted by the deep research should be corrected in all affected artifacts.
|
||||
|
||||
---
|
||||
|
||||
## 1. SkillsBench "2–3 skills optimal" — does this benchmark exist?
|
||||
|
||||
**Current position:** The research cites SkillsBench (2026) as finding the optimal range for skills per task is 2–3, with more skills per step increasing failure surface.
|
||||
|
||||
**What to challenge:**
|
||||
- Is SkillsBench a real published benchmark or a practitioner label? No primary source URL was confirmed in either the original research session or the re-search pass.
|
||||
- If it exists, what is the methodology — controlled experiment, user study, or practitioner survey?
|
||||
- Is the 2–3 figure specific to agent skill invocations or to general multi-step LLM chaining?
|
||||
- Does the finding generalise across tools (Claude Code, Codex, Gemini CLI) or is it tool-specific?
|
||||
|
||||
**Design implication:** The 2–3 skill optimal range influences workflow chain design. If unverified, the principle has no empirical basis and should be removed or replaced.
|
||||
|
||||
---
|
||||
|
||||
## 2. The 3,000-token degradation ceiling — does it hold for 2026 models?
|
||||
|
||||
**Current position:** LLM reasoning performance starts degrading around 3,000 tokens of context. This is the primary justification for the 500-line skill limit, the CONTEXT.md ≤200 line constraint, and most of the token optimisation strategy.
|
||||
|
||||
**What to challenge:**
|
||||
- The source is Levy, Jacoby, and Goldberg (2024) — a study on models that predate Claude Sonnet 4.6 and Opus 4.7 by at least one generation. Do 2026 models show the same degradation at the same threshold?
|
||||
- Has Anthropic or any independent lab published updated context length vs. reasoning quality data for Claude 4-series models?
|
||||
- The SkillReducer paper (arxiv.org/pdf/2603.29919) documents context budget problems but does not validate the 3,000-token specific threshold. Is there a more recent primary source?
|
||||
- Does the degradation manifest equally for structured code tasks vs. natural language tasks?
|
||||
|
||||
**Design implication:** If the threshold is actually higher for 2026 models, the 500-line skill limit and CONTEXT.md constraint may be over-conservative. If lower, they should tighten.
|
||||
|
||||
---
|
||||
|
||||
## 3. Skill trigger reliability across tools — does the description field work equally well everywhere?
|
||||
|
||||
**Current position:** The description field is the trigger — the agent uses it to decide whether a given request activates the skill. The research treats this as a universal cross-tool mechanism.
|
||||
|
||||
**What to challenge:**
|
||||
- Do Claude Code, Codex CLI, Gemini CLI, and GitHub Copilot implement skill activation identically, or does each tool use different matching logic (semantic similarity, keyword matching, LLM classification)?
|
||||
- What is the empirically observed false-positive and false-negative activation rate per tool for a typical skill description?
|
||||
- Does the negative case in trigger evals actually prevent activation reliably, or do tools still fire on adjacent prompts?
|
||||
- The OpenAI eval guidance (developers.openai.com/blog/eval-skills) is specifically for Codex. Does the 10–20 test case structure produce reliable coverage for Claude Code specifically?
|
||||
|
||||
**Design implication:** If trigger reliability differs significantly across tools, the tool-agnostic factory design may require tool-specific description tuning — undermining the single-source-of-truth principle.
|
||||
|
||||
---
|
||||
|
||||
## 4. Veracode vulnerable code finding — independent validation
|
||||
|
||||
**Current position:** Veracode 2025 GenAI Code Security Report found AI-generated code has 2.74× more vulnerabilities than human-written code, with 45% of code samples introducing OWASP Top 10 vulnerabilities across 100+ LLMs and 4 languages.
|
||||
|
||||
**What to challenge:**
|
||||
- Veracode is a security tool vendor with commercial interest in finding vulnerabilities. Do independent academic benchmarks confirm the 2.74× figure and 45% OWASP rate?
|
||||
- The Yan et al. (2025) range (9.8–42.1%) is from a different study. How do these reconcile — different metrics, different populations?
|
||||
- Does the vulnerability rate change significantly when AI-generated code is reviewed by a human before commit? The research implies review closes the gap but does not quantify by how much.
|
||||
- The Backslash Security finding (all seven tested LLMs generated code vulnerable to at least 4 of 10 common CWEs with "naive" prompts) — is this replicable with security-focused prompts?
|
||||
|
||||
**Design implication:** The security-review skill is positioned as mandatory pre-merge because of the vulnerability rate evidence. If review substantially closes the gap, the framing remains correct; if the rate with review is much lower, the mandatory-gate urgency may need calibration.
|
||||
|
||||
---
|
||||
|
||||
## 5. SDD adoption claims — stars vs. actual production usage
|
||||
|
||||
**Current position:** GitHub Spec Kit has ~90,000 stars (April 2026). AWS Kiro attracted 250,000 developers in its first three months. Spec-Driven Development is described as "now mainstream."
|
||||
|
||||
**What to challenge:**
|
||||
- GitHub stars measure visibility, not usage. What evidence exists that teams are shipping production software with SDD workflows rather than experimenting briefly?
|
||||
- The 250,000 figure for AWS Kiro — is this active users, signups, or downloads? AWS has a strong incentive to publicise adoption figures favourably.
|
||||
- Thoughtworks Radar rated SDD "Assess, not Adopt" in November 2025. What is their full rationale?
|
||||
- Does SDD overhead scale with team size such that it's genuinely counterproductive for solo developers? The counterargument evidence suggests yes — what does the evidence say for a solo homelab context specifically?
|
||||
|
||||
**Design implication:** The write-spec skill is the factory's highest-priority foundation skill. If SDD's evidence base is thinner than presented for solo use, a lighter-weight spec approach may be appropriate and the build order priority may need adjusting.
|
||||
|
||||
---
|
||||
|
||||
## 6. Prompt caching hit rate — achievable in coding contexts?
|
||||
|
||||
**Current position:** A 74–84% cache hit rate is achievable on "stable agent workloads," yielding 90% savings on cached input tokens.
|
||||
|
||||
**What to challenge:**
|
||||
- The 74–84% figure is from production support bot and stable document processing workflows (Vellum and Helicone reports via tokenmix.ai). Coding workflows have significantly more variable dynamic content — different diffs, files, and task inputs per session. Is this hit rate achievable for the factory's actual use patterns?
|
||||
- How does the 5-minute Sonnet TTL interact with typical development sessions where a developer pauses between tool calls? If sessions regularly exceed the TTL, cache savings may be substantially lower.
|
||||
- Is there published data on prompt caching performance specifically in agentic coding workflows?
|
||||
- What is the cache write overhead at low hit rates — does it add net cost at low reuse patterns?
|
||||
|
||||
**Design implication:** Prompt caching is the single highest-ROI optimisation recommended. If actual hit rates for coding workflows are 20–30% rather than 74–84%, the expected savings are dramatically lower and the static-prefix-first architecture design may need revisiting.
|
||||
|
||||
---
|
||||
|
||||
## 7. gstack productivity claims — what does "logical LOC" actually mean?
|
||||
|
||||
**Current position:** Garry Tan claims 600,000 lines of production code in 60 days (10,000–20,000/day), an 810× improvement over his 2013 baseline. gstack is cited as the primary evidence that role-based design works in practice.
|
||||
|
||||
**What to challenge:**
|
||||
- "Logical LOC" is not a standard metric. What does Tan's own methodology document define as a logical line — is it meaningful-change lines excluding whitespace, comments, and generated boilerplate?
|
||||
- The comparison baseline (2013, Bookface at YC) may not represent a typical developer's productivity — it may reflect his output at a specific early-career stage on a specific project type.
|
||||
- Are there independent accounts of other teams using gstack achieving comparable productivity improvements?
|
||||
- Does the role-based design pattern specifically drive the gain, or is it Claude Code + any structured workflow?
|
||||
|
||||
**Design implication:** The six-role factory design is substantially influenced by gstack as a reference implementation. If the productivity evidence is thin, the role design still holds on cognitive mode separation logic — but the specific gstack role mapping should be treated as one implementation, not the validated approach.
|
||||
|
||||
---
|
||||
|
||||
## 8. MCP security posture at homelab scale — is enterprise framing appropriate?
|
||||
|
||||
**Current position:** CIS guide, Red Hat OAuth 2.1 guidance, and vendor security research establish the MCP security requirements. Primary mitigations: dedicated agent accounts, token rotation, `--tools` filtering, HTTP over stdio for multi-user.
|
||||
|
||||
**What to challenge:**
|
||||
- The CIS guide, Red Hat article, and vendor research are all written for enterprise contexts with multiple users and adversarial threat models. What is the appropriate security posture for a single-user, self-hosted homelab with no external users?
|
||||
- The dedicated agent account mitigation for Gitea's lack of granular scopes — is this meaningfully different from a personal token in a single-user context where the account owner controls all repos anyway?
|
||||
- Is stdio MCP (no auth by default) a meaningful security gap in a single-user context where only the user can send prompts?
|
||||
- What does the threat model actually look like at homelab scale? Primary risks cited (token exfiltration via compromised agents, rogue MCP servers) require an adversary with machine access — at which point most mitigations are moot.
|
||||
|
||||
**Design implication:** The MCP security requirements may be calibrated for a threat model that doesn't apply to the current single-user homelab context. Some mitigations are still good practice; others may add friction without proportionate protection.
|
||||
|
||||
---
|
||||
|
||||
## 9. OWASP Agentic Skills Top 10 — does AST01 apply to self-authored skills?
|
||||
|
||||
**Current position:** The OWASP Agentic Skills Top 10 (April 27, 2026) is referenced in the security-review skill. AST01 (Malicious Skills) is the critical risk category.
|
||||
|
||||
**What to challenge:**
|
||||
- AST01 and the ToxicSkills/ClawHavoc research concern skills installed from public registries (ClawHub, skills.sh). The factory produces and uses self-authored skills only. Does AST01 apply to self-authored private skills in any meaningful way?
|
||||
- The attack vectors documented (staged payload delivery via Markdown-as-installer, prompt injection in SKILL.md) require the attacker to control the skill content. In a self-authored factory, the author controls all content. What residual risks remain?
|
||||
- Are there any AST10 risk categories beyond AST01 that apply to self-authored private skills rather than public marketplace skills?
|
||||
- The AST10 was published 18 days before the end of this research session. How mature is it — provisional community draft or reviewed standard?
|
||||
|
||||
**Design implication:** If AST01 primarily concerns third-party public skills, the security-review skill's AST10 reference may be misleading for a self-authored factory. The constraint should either be removed or replaced with relevant self-authored skill risks.
|
||||
|
||||
---
|
||||
|
||||
## 10. Single practitioner case studies — are the token optimisation figures replicable?
|
||||
|
||||
**Current position:** "Decisions not descriptions" yields 20% token reduction; AST-level file loading reduced relevant file reads from 40 to 5 (65% reduction). Both sourced to a single Nicola Alessi case study (earezki.com, February 2026).
|
||||
|
||||
**What to challenge:**
|
||||
- What was the baseline in the case study — 40 file reads suggests an unusually broad grep pattern that may not reflect typical agent behaviour.
|
||||
- Has any other practitioner independently documented similar reductions from the "decisions not descriptions" context file approach?
|
||||
- The 65% file read reduction from AST-level loading requires implementing AST tooling (`vexp`, Rust-based). What is the implementation cost relative to the savings? Is it warranted at homelab scale?
|
||||
- Are there published benchmarks or controlled studies of context file design approaches, or is this field entirely practitioner case studies?
|
||||
|
||||
**Design implication:** If these figures are not replicable, the token optimisation strategy rests on anecdote. The general principle (precise context over broad loading) is sound regardless, but the specific figures should not be cited as expected outcomes.
|
||||
|
||||
---
|
||||
|
||||
## 11. gitea-mcp permission model — verified for current Gitea versions?
|
||||
|
||||
**Current position:** Gitea has no granular token scopes — the agent can do anything including deleting repositories. The dedicated agent account is the primary mitigation.
|
||||
|
||||
**What to challenge:**
|
||||
- Has Gitea introduced token scope improvements in versions 1.20, 1.21, or 1.22? The warning comes from gitea-mcp-compact (community fork README), not official Gitea documentation.
|
||||
- Does the `--tools` flag in the official gitea-mcp actually restrict what actions the underlying token can perform, or does it only filter what the agent knows it can invoke?
|
||||
- The forgejo-mcp HTTP mode with multi-user support — does it address the token granularity problem through OAuth scoping, or does it still use broad Gitea tokens underneath?
|
||||
- Is the blast radius of an unintended action with a full-access token on a single-user homelab Gitea instance meaningfully different from one with a repo-scoped token?
|
||||
|
||||
**Design implication:** If Gitea has added scope support in recent versions, or if `--tools` provides real enforcement rather than just advertised scope, the dedicated-account-as-mitigation framing needs updating.
|
||||
|
||||
---
|
||||
|
||||
## 12. Skill eval reliability — does the OpenAI pattern transfer across tools?
|
||||
|
||||
**Current position:** The trigger eval pattern (10–20 test cases: explicit, implicit, negative) comes from developers.openai.com/blog/eval-skills, which documents evaluation for OpenAI Codex skills. Promptfoo is the recommended testing tool.
|
||||
|
||||
**What to challenge:**
|
||||
- Does skill activation in Claude Code use the same matching mechanism as Codex, or does it differ enough that the eval structure needs adaptation?
|
||||
- What is the observed false-negative rate (skill should trigger, doesn't) for well-written descriptions across tools?
|
||||
- Has Promptfoo's approach to non-OpenAI model evaluation changed since the March 2026 OpenAI acquisition? Specifically: has cross-model eval parity been maintained?
|
||||
- Do DeepEval or Arize Phoenix have skill-specific eval guidance, or do they require custom configuration to test skill trigger behaviour?
|
||||
|
||||
**Design implication:** If the OpenAI eval pattern doesn't transfer to Claude Code, the factory's CI eval pipeline may produce false confidence. If promptfoo's non-OpenAI coverage has degraded, the tooling recommendation needs to change.
|
||||
|
||||
---
|
||||
|
||||
## Deep Research Session Objectives
|
||||
|
||||
1. Confirm or refute the SkillsBench 2–3 skills finding — if unconfirmed, remove from research doc.
|
||||
2. Find updated context length vs. reasoning quality data for 2026 models — update the 3,000-token threshold if evidence warrants.
|
||||
3. Find independent, non-vendor data on AI-generated code vulnerability rates — reconcile with or replace the Veracode finding.
|
||||
4. Find practitioner or experimental data on SDD outcomes at solo/small team scale — calibrate the write-spec skill's priority if warranted.
|
||||
5. Find data on prompt caching performance in agentic coding workflows specifically — update expected savings if warranted.
|
||||
6. Determine whether Gitea has introduced token scope improvements — update MCP security posture if so.
|
||||
7. Assess OWASP AST10 applicability to self-authored private skills — adjust security-review constraints accordingly.
|
||||
8. Find cross-tool skill activation reliability data — assess whether tool-specific trigger testing is needed.
|
||||
|
||||
---
|
||||
|
||||
*Prepared after initial factory research session, May 2026.*
|
||||
*Upload alongside `ai-coding-factory-research.md` and `ai-coding-factory-session.md` to continue.*
|
||||
@@ -0,0 +1,712 @@
|
||||
# AI Coding Factory — Implementation Guidance
|
||||
|
||||
**Version:** 1.2
|
||||
**Date:** May 2026
|
||||
**Status:** Guidance — not yet adapted to repo vision, roadmap, or ADRs
|
||||
**Audience:** Human implementing the factory; Claude Code executing against it
|
||||
**Prerequisite reading:** `ai-coding-factory-principles.md`, `ai-coding-factory-research.md`, `ai-coding-factory-skills-index.md`
|
||||
|
||||
> **Important:** This document is research-derived implementation guidance, not a finalised plan. It must be reconciled with the repo's actual vision and roadmap (to be established via grill-me sessions in Claude Code) before implementation begins. Where this guidance conflicts with those outputs, the grill-me session outputs win. ADRs produced in Claude Code are the authoritative implementation decisions; this document provides the evidence base and recommendations that inform them.
|
||||
|
||||
---
|
||||
|
||||
## 1. Purpose and Scope
|
||||
|
||||
This document translates the factory's research and principles into concrete implementation guidance. It answers questions the principles document does not: *what to build first, where to take it from, how documentation stays in sync, and what good looks like when done.*
|
||||
|
||||
The factory is a governed, AI-assisted development environment implemented as a set of repo-committed files. It is not a SaaS tool, not a framework import, and not a third-party dependency. Everything lives in the repo and travels with it.
|
||||
|
||||
**What this document covers:**
|
||||
- Documentation layer hierarchy and drift prevention strategy
|
||||
- Framework file mapping — what to take from each external source and exactly where it goes
|
||||
- Repository structure (the actual directory layout)
|
||||
- Phased build plan with acceptance criteria
|
||||
- Anti-patterns to explicitly exclude
|
||||
- Best-practice sources for each implementation area
|
||||
|
||||
**What this document does not cover:**
|
||||
- The repo vision or roadmap (established via Claude Code grill-me sessions — see Section 10)
|
||||
- ADR decisions (produced in Claude Code and stored in `docs/adr/`)
|
||||
- Skill body content (that lives in `ai-coding-factory-skills-index.md`)
|
||||
- Governance rules (that lives in `AGENTS.md`)
|
||||
- Research rationale (that lives in `ai-coding-factory-research.md`)
|
||||
|
||||
---
|
||||
|
||||
## 2. Current Gaps
|
||||
|
||||
The existing research and principles are complete and evidence-based. The gap is implementation. Specifically:
|
||||
|
||||
1. No canonical repository structure exists yet — the factory is documented but not built.
|
||||
2. Skill files (`SKILL.md`) exist only as a design table in the skills index, not as actual files.
|
||||
3. The workflow chain (spec → plan → implement → review → deploy) is defined in principles but not wired into a repeatable process.
|
||||
4. No eval infrastructure exists to validate skill behaviour or catch model-version regressions.
|
||||
5. Repo vision, roadmap, and ADRs do not yet exist — these are prerequisites for adapting this guidance to the actual build.
|
||||
|
||||
---
|
||||
|
||||
## 3. Documentation-Driven Development
|
||||
|
||||
### 3.1 The documentation drift problem
|
||||
|
||||
Drift has a direction: implementation moves faster than docs, so docs lag. Disciplinary fixes ("remember to update the docs") don't work at scale. The only reliable fixes are structural.
|
||||
|
||||
The factory is documentation-driven by default. This means: every implementation decision either derives from an existing document or produces a new one. Code that cannot be traced to a spec is out-of-scope code. A convention that isn't in CONTEXT.md isn't a convention — it's undiscoverable tribal knowledge.
|
||||
|
||||
**Current tooling:** git + markdown. This is the correct starting point. Every layer described below is future-compatible: when better tools arrive (living-spec platforms, architecture-as-code, doc coverage CI), they read the file structure that already exists. No migration required.
|
||||
|
||||
**Future upgrade path (no breaking changes):**
|
||||
- `docs/spec/` → input for a living-spec synchronisation platform or architecture-as-code tool (Structurizr C4 DSL)
|
||||
- `docs/adr/` → machine-readable decision graph with tooling like adr-tools or Log4brains
|
||||
- Gitea issues → exportable to any future work-tracking platform; structure is plain markdown
|
||||
- SKILL.md evals → input for a doc-coverage CI metric ("what % of skills have passing evals?")
|
||||
|
||||
### 3.2 Documentation layer hierarchy
|
||||
|
||||
Five layers, each with a clear scope and update frequency. The rule: content lives in exactly one layer. Duplication between layers is prohibited — it is the primary cause of drift.
|
||||
|
||||
| Layer | Files | Changes when | Update mechanism |
|
||||
|---|---|---|---|
|
||||
| Governance | `AGENTS.md`, `HUMANS.md`, `CONTROLS.md`, `ai-constitution.md` | Principles change or tooling changes fundamentally | Human deliberate edit; reviewed in PR |
|
||||
| Convention | `CONTEXT.md`, `LESSONS.md`, `references/stack-versions.md` | New pattern adopted; LESSONS.md threshold hit | LESSONS.md auto-graduates to CONTEXT.md at 3× same pattern |
|
||||
| Specification | `docs/spec/` (what the system does now), `docs/adr/` (decisions) | Behaviour changes or decision made | Spec updated in same PR as implementation; ADRs via architect role; never deleted |
|
||||
| Work tracking | Gitea issues + feature branches | New work begins or completes | Issue created with EARS spec; branch per issue; closed when PR merges |
|
||||
| Skills & workflows | `.agents/skills/**`, `.agents/workflows/`, `.agents/evals/` | Skill behaviour changes or new skill added | PR with co-located eval update; CI blocks merge if eval fails |
|
||||
| Implementation | `src/`, `tests/`, `infra/`, `docs/api/`, `CHANGELOG.md` | Any code change | Derives from issue spec; changelog derives from conventional commits |
|
||||
|
||||
> **Work tracking is not committed files.** Gitea issues and branches are the work layer. They are queryable by agents via Gitea MCP, isolated per feature, and their history lives in Gitea — not in the repo working tree. The working tree only contains what is true right now and worth preserving permanently.
|
||||
|
||||
### 3.3 Anti-drift mechanisms
|
||||
|
||||
Six structural mechanisms prevent drift. All six must be active — any one of them alone is insufficient.
|
||||
|
||||
**One source per concern.** No content is duplicated across layers or tool-specific files. CLAUDE.md references AGENTS.md; it does not duplicate it. A skill's trigger description appears only in its `SKILL.md`; it is not copied into CONTEXT.md or a workflow file.
|
||||
|
||||
**Issue spec precedes code.** No implementation work begins without a Gitea issue containing an EARS-format spec. The `factory/write-issue-spec` skill drafts this; the Gitea MCP posts it. Code on a feature branch cannot drift from a spec that must exist before the branch is created. The `deploy/deployment-checklist` skill verifies a linked issue exists before any deployment.
|
||||
|
||||
**Living spec updates in the same PR as the behaviour change.** `docs/spec/` is updated as part of the PR that changes system behaviour — not as a separate "archive" step. If a PR changes behaviour without touching `docs/spec/`, CI flags it as spec drift. Git history preserves the full change trail; the working tree only shows current truth.
|
||||
|
||||
**CI eval gates block skill drift.** Every skill has an `eval.yaml` co-located in `.agents/evals/`. A change to any file under `.agents/skills/**` triggers the co-located eval in CI. A failing eval blocks merge. This means a skill body can never drift from its documented behaviour without a deliberate eval update.
|
||||
|
||||
**LESSONS.md graduation.** Three or more occurrences of the same pattern in LESSONS.md trigger promotion to CONTEXT.md as a standing rule. This is the only sanctioned upward flow in the documentation hierarchy — and only into the convention layer, not higher.
|
||||
|
||||
**Tool adapter pattern.** Tool-specific files (CLAUDE.md, `.cursor/rules`, `copilot-instructions.md`) are thin adapters that reference the source layer. They contain no original content. When a tool changes its format, only the adapter changes. The source — AGENTS.md, CONTEXT.md — is untouched.
|
||||
|
||||
---
|
||||
|
||||
## 4. External Framework Analysis — What to Take and From Where
|
||||
|
||||
Five external frameworks are relevant. Each contributes a specific, non-overlapping element. None should be imported wholesale.
|
||||
|
||||
---
|
||||
|
||||
### 4.1 Matt Pocock / `mattpocock/skills`
|
||||
|
||||
**What it is:** ~68K star MIT-licensed collection of 20+ production engineering skills built from Pocock's personal `.claude` directory. Workflow enforcement focus, not capability extension. TypeScript-centric but format-agnostic.
|
||||
|
||||
**Key elements to adopt:**
|
||||
|
||||
| Element | What it is | Why adopt it |
|
||||
|---|---|---|
|
||||
| `/grill-me` skill | Relentless pre-coding interrogation until every decision is explicit | Already in factory's skills index; validate against Pocock's implementation for completeness |
|
||||
| `/to-prd` skill | Converts conversation into a structured PRD artifact | Adopt the forcing-function pattern: refuse to proceed until synthesis is complete |
|
||||
| `/to-issues` skill | Breaks PRD into independently shippable issues | Maps to factory's `design/break-into-issues` skill — check Pocock's "independently shippable slices" constraint |
|
||||
| `/tdd` skill structure | Multi-file skill: `SKILL.md` + `tests.md` + `mocking.md` + `refactoring.md` + `deep-modules.md` | The multi-file skill pattern is best practice for complex skills. Factory's `test/write-tests` skill should adopt this structure |
|
||||
| `CONTEXT.md` generation | Setup skill creates a shared vocabulary file on init | Factory already has this concept; Pocock's implementation provides a concrete template |
|
||||
| `/git-guardrails` | Blocks risky agent actions (force pushes, direct main commits) | Adopt as factory's `cross-cutting/git-guardrails` skill — currently absent |
|
||||
| `/diagnose` | Structured debugging state machine | Adopt as factory's `implement/diagnose` skill — currently absent |
|
||||
| `/improve-codebase-architecture` | Architecture debt hunting using CONTEXT.md + ADRs | Reinforces factory's Reviewer role; drives ADR creation workflow |
|
||||
| `/zoom-out` | Periodic architecture perspective shift | Maps to Architect role; adopt the "zoom out before diving in" forcing function |
|
||||
| `caveman` skill | ~75% token reduction through compressed output mode | Adopt for token optimisation — maps to factory's orientation problem solution |
|
||||
|
||||
**What NOT to take:** TypeScript-specific skill content (typing gymnastics, TS-specific patterns). The factory is language-agnostic.
|
||||
|
||||
**Source:** `github.com/mattpocock/skills` (MIT). Install via: `npx skills@latest add mattpocock/skills/<skill-name>` — use for inspection only; copy and adapt into factory's own skill format rather than runtime dependency.
|
||||
|
||||
---
|
||||
|
||||
### 4.2 BMAD-METHOD (Breakthrough Method for Agile AI-Driven Development)
|
||||
|
||||
**What it is:** Open-source MIT-licensed multi-agent framework with 12+ specialized agent personas, 34+ agile workflows, and scale-adaptive planning depth. Most complete full-SDLC framework available.
|
||||
|
||||
**Key elements to adopt:**
|
||||
|
||||
| Element | What it is | Why adopt it |
|
||||
|---|---|---|
|
||||
| Scale-adaptive planning | Automatically adjusts depth based on complexity: bug fix vs. enterprise feature | Factory's spec skill currently has a binary "skip for simple tasks" rule. BMAD's graduated approach is more practical — adopt a complexity-gated checklist |
|
||||
| Party Mode concept | Multiple agent personas in one session for collaborative decision-making | Useful framing for the Architect role when design alternatives need to be stress-tested. Implement as an `architect/design-review` skill variant |
|
||||
| Scrum Master agent pattern | Transforms planning docs into hyper-detailed stories embedding full context | The "embed full context in each story" principle directly prevents context loss between sessions. Adopt this in `design/break-into-issues` |
|
||||
| Analyst → PRD → Epics → Stories pipeline | Structured handoff chain with artifacts at each stage | Maps cleanly to factory's spec → plan → implement chain. Use BMAD's artifact naming conventions as reference |
|
||||
| Market analysis as gate | Reality-check before PRD creation | Consider adapting as an optional `/validate-idea` skill that runs before write-spec |
|
||||
| 5 official modules model | Core + Builder + Test + GameDev + Creative modules | Adopt this modular structure for the factory's skill library — group skills by domain module |
|
||||
|
||||
**What NOT to take:** The full BMAD CLI installation or its agent persona system as-is. The factory runs on Claude Code + SKILL.md; BMAD's runtime is not compatible. Take the *conceptual patterns*, not the implementation.
|
||||
|
||||
**Caveat:** BMAD productivity claims are not independently verified. The framework is heavy by design — its ceremony is appropriate for greenfield enterprise builds, not single-developer homelabs. Apply scale-adaptive gating aggressively.
|
||||
|
||||
**Source:** `github.com/bmad-method/bmad-method` (MIT).
|
||||
|
||||
---
|
||||
|
||||
### 4.3 GitHub Spec Kit
|
||||
|
||||
**What it is:** GitHub's open-source SDD toolkit, 90K+ stars, May 2026. CLI (`specify`) + slash commands implementing a `specify → plan → tasks → implement` workflow. Agent-agnostic (20+ supported). Has a `constitution` concept for non-negotiable project principles.
|
||||
|
||||
**Key elements to adopt:**
|
||||
|
||||
| Element | What it is | Why adopt it |
|
||||
|---|---|---|
|
||||
| Constitution file | Non-negotiable principles (CLI-first, testing conventions) that govern all AI output | This is AGENTS.md under a different name. The Spec Kit implementation is worth inspecting for structure — particularly how it enforces constitution compliance during implementation |
|
||||
| `specify → plan → tasks → implement` command sequence | Repeatable 4-step workflow encoded as slash commands | Factory's workflow already mirrors this. The Spec Kit format for linking artifacts (requirements.md → design.md → tasks.md) is the reference implementation to follow |
|
||||
| Community extensions model | 70+ extensions across documentation, code, process | Adopt the extension/module model for the factory's skill library. Third-party skills should have a documented contribution pattern |
|
||||
| Brownfield support | Works on existing codebases, not just greenfield | Ensure factory's write-spec skill explicitly handles brownfield case — Spec Kit has the best documented approach here |
|
||||
| EARS notation | Easy Approach to Requirements Statements: "WHEN [trigger] AND [condition] THE system SHALL [response]" | Factory already mandates EARS in the write-spec skill. Spec Kit's validation approach (checking EARS syntax before implementation) is worth adopting as a spec linting step |
|
||||
|
||||
**What NOT to take:** Python/`uv` CLI dependency. The factory is CLI-optional; context lives in files. Don't introduce a Python runtime requirement.
|
||||
|
||||
**Caveat:** Thoughtworks Radar rates it "Assess, not Adopt" (November 2025) with the observation: *"We may be relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn't scale."* Treat Spec Kit as a reference pattern, not a silver bullet.
|
||||
|
||||
**Source:** `github.com/github/spec-kit` (MIT). Best for: EARS constraint examples, brownfield spec patterns, constitution file format.
|
||||
|
||||
---
|
||||
|
||||
### 4.4 OpenSpec
|
||||
|
||||
**What it is:** "Brownfield-first, fluid not rigid" SDD framework. MIT. Unique differentiator: per-change folder isolation (`openspec/changes/<name>/`) with delta specs that track what's changing relative to existing functionality.
|
||||
|
||||
**What to adopt — patterns only, no runtime:**
|
||||
|
||||
| Element | What it is | Where the pattern goes |
|
||||
|---|---|---|
|
||||
| Delta spec format | `ADDED/MODIFIED/REMOVED` markers track what's changing vs. existing | PR description template + `docs/spec/` update convention |
|
||||
| Proposal-first workflow | Explicit "why" before any implementation | Gitea issue template (proposal section before tasks) |
|
||||
| Verify step (`/opsx:verify`) | Post-implementation check against the spec | `test/review-test-coverage/SKILL.md` (verify-against-spec mode) |
|
||||
| Fast-forward concept | Single command scaffolds all planning artifacts | `factory/write-issue-spec/SKILL.md` (creates full issue body in one pass) |
|
||||
|
||||
**What NOT to adopt:**
|
||||
- The `openspec/changes/` directory — this is work tracking, which belongs in Gitea issues and feature branches. Committed working documents in the repo working tree create drift, not prevent it.
|
||||
- The archive lifecycle — git history is the archive. The working tree shows current truth only.
|
||||
- The Node.js CLI or any runtime dependency.
|
||||
|
||||
**Source:** `github.com/Fission-AI/OpenSpec` (MIT). Best for: delta notation, brownfield proposal structure, verify-against-spec pattern.
|
||||
|
||||
---
|
||||
|
||||
### 4.5 GSD (Get Stuff Done)
|
||||
|
||||
**What it is:** Lean, low-ceremony spec-driven framework built primarily for Claude Code. Positions itself as the "fast alternative to BMAD." Context engineering first; subagents in parallel; optimised for execution speed.
|
||||
|
||||
**Key elements to adopt:**
|
||||
|
||||
| Element | What it is | Why adopt it |
|
||||
|---|---|---|
|
||||
| "Lean default, escalate to ceremony" principle | Start with minimal process; add ceremony only when complexity warrants | This is the right default posture for a single-developer factory. Adopt as a meta-principle for skill invocation |
|
||||
| Context engineering patterns | 4 strategies: write (persist), select (RAG), compress, isolate | Factory already applies this. GSD's specific compression techniques are worth reviewing — particularly context isolation between subagents |
|
||||
| Subagent parallelism patterns | How to decompose tasks for parallel execution | Useful reference for orchestrator-worker implementation. Note: the factory currently runs roles sequentially by design. GSD's parallel approach is a future optimisation, not day-one |
|
||||
|
||||
**What NOT to take:** GSD's "speed first" philosophy as a governing principle. The factory prioritises quality and auditability over throughput.
|
||||
|
||||
**Source:** Search GitHub for `GSD framework Claude Code` — community tool, no single canonical repo. Best for: context engineering examples and lean skill implementations.
|
||||
|
||||
---
|
||||
|
||||
### 4.6 AWS Kiro (Inspiration Only)
|
||||
|
||||
**What it is:** AWS's agentic IDE (not open-source). Relevant because it popularised the three-document system (requirements.md + design.md + tasks.md) and EARS notation in production tooling.
|
||||
|
||||
**Elements to take as inspiration:**
|
||||
|
||||
- The three-document structure (requirements → design → tasks) is the reference for factory's spec artifact layout.
|
||||
- Autonomous agent hooks (auto-generating tests and docs after implementation) suggest a future enhancement to the CI/CD pipeline.
|
||||
- Spec-as-artifact model: structured requirements give future agents richer context than commit history. The factory should treat all spec artifacts as long-lived references, not throwaway planning docs.
|
||||
|
||||
**Source:** `kiro.dev` — closed source. Use only as a reference pattern; no code to adopt.
|
||||
|
||||
---
|
||||
|
||||
## 5. Framework File Mapping — What Goes Where
|
||||
|
||||
Nothing from external frameworks enters the repo as a runtime dependency. Every row below is: copy the pattern, adapt the content, commit it as your own file. The source column is for reference only.
|
||||
|
||||
> Rows marked **(pattern only)** mean adopt the structural approach or output format, not the file content verbatim. Rows marked **(adopt directly)** are close enough in format that the file can be copied and lightly adapted.
|
||||
|
||||
| Source | Element | Action | Factory file |
|
||||
|---|---|---|---|
|
||||
| Pocock/skills | `grill-me/SKILL.md` | Adopt directly | `.agents/skills/design/grill-me/SKILL.md` |
|
||||
| Pocock/skills | `to-prd/SKILL.md` | Pattern only — forcing-function approach | `.agents/skills/factory/write-issue-spec/SKILL.md` |
|
||||
| Pocock/skills | `to-issues/SKILL.md` | Pattern only — independently shippable slices | `.agents/skills/design/break-into-issues/SKILL.md` |
|
||||
| Pocock/skills | `tdd/` (multi-file) | Adopt directly — entire folder structure | `.agents/skills/implement/tdd/` (SKILL.md + tests.md + mocking.md + refactoring.md) |
|
||||
| Pocock/skills | `diagnose/SKILL.md` | Adopt directly | `.agents/skills/implement/diagnose/SKILL.md` |
|
||||
| Pocock/skills | `git-guardrails/SKILL.md` | Adopt directly | `.agents/skills/cross-cutting/git-guardrails/SKILL.md` |
|
||||
| Pocock/skills | CONTEXT.md generation (setup skill) | Pattern only — content is yours | `CONTEXT.md` at repo root (content file, not a skill) |
|
||||
| Pocock/skills | `zoom-out/SKILL.md` | Adopt directly — merge as a mode | `.agents/skills/roles/architect/SKILL.md` |
|
||||
| BMAD-METHOD | Scale-adaptive planning depth | Pattern only — complexity gate logic | `.agents/skills/factory/write-issue-spec/SKILL.md` (complexity-gating section) |
|
||||
| BMAD-METHOD | Scrum Master story format | Pattern only — embed full context | `.agents/skills/design/break-into-issues/SKILL.md` (output format) |
|
||||
| BMAD-METHOD | 5-module directory model | Pattern only — directory grouping | `.agents/skills/` directory structure (design/, implement/, factory/ etc.) |
|
||||
| GitHub Spec Kit | Constitution file concept | Already exists — inspect for improvements | `AGENTS.md` |
|
||||
| GitHub Spec Kit | `specify→plan→tasks→implement` sequence | Pattern only — orchestration order | `.agents/workflows/feature-workflow.md` |
|
||||
| GitHub Spec Kit | EARS notation examples | Pattern only — extract constraint examples | `.agents/skills/factory/write-issue-spec/SKILL.md` (acceptance criteria section) |
|
||||
| GitHub Spec Kit | Brownfield spec approach | Pattern only — delta notation | PR description template + `docs/spec/` update convention |
|
||||
| OpenSpec | Delta spec format (ADDED/MODIFIED/REMOVED) | Pattern only — apply to PR descriptions | PR description template in `deploy/pr-description/SKILL.md` |
|
||||
| OpenSpec | Proposal-first workflow | Pattern only — "why" section before tasks | Gitea issue template (section in `factory/write-issue-spec/SKILL.md`) |
|
||||
| OpenSpec | Verify step | Pattern only — post-impl spec check | `.agents/skills/test/review-test-coverage/SKILL.md` (verify-against-spec mode) |
|
||||
| Kiro (AWS) | EARS notation reference | Pattern only — examples | `.agents/skills/factory/write-issue-spec/SKILL.md` |
|
||||
| Kiro (AWS) | Three-document system (req + design + tasks) | Pattern only — maps to issue structure | Gitea issue template sections: requirements, design notes, task checklist |
|
||||
| GSD | "Lean default, escalate to ceremony" | Pattern only — meta-principle | `AGENTS.md` (skill invocation guidance) |
|
||||
| GSD | Context compression techniques | Pattern only | All skills (constraints section: prefer compressed output) |
|
||||
|
||||
**Gitea issue template structure** (replaces openspec/changes/): the spec lives in the issue, not in the working tree.
|
||||
|
||||
```markdown
|
||||
## Why (proposal)
|
||||
<!-- Problem being solved; out-of-scope explicitly stated -->
|
||||
|
||||
## Requirements
|
||||
<!-- EARS format: WHEN [trigger] AND [condition] THE system SHALL [response] -->
|
||||
<!-- Delta markers for brownfield: ADDED / MODIFIED / REMOVED -->
|
||||
|
||||
## Design notes
|
||||
<!-- Technical approach; data model; API surface if relevant -->
|
||||
|
||||
## Tasks
|
||||
- [ ] Task 1 (independently completable)
|
||||
- [ ] Task 2
|
||||
```
|
||||
|
||||
This is the three-document structure from Kiro and the proposal + delta format from OpenSpec — expressed as a single Gitea issue body instead of committed files.
|
||||
|
||||
---
|
||||
|
||||
## 6. Repository Structure
|
||||
|
||||
The working tree contains only permanent, final-state content. Work in progress lives in Gitea issues and feature branches — not in committed files.
|
||||
|
||||
```
|
||||
repo/
|
||||
├── AGENTS.md # Agent governance (all rules)
|
||||
├── CONTEXT.md # Shared vocabulary and conventions
|
||||
├── LESSONS.md # Feedback → graduates to CONTEXT.md at 3× threshold
|
||||
├── HUMANS.md # Human orientation guide
|
||||
├── CONTROLS.md # Operational safety controls
|
||||
│
|
||||
├── .agents/
|
||||
│ ├── skills/ # Canonical skill library
|
||||
│ │ ├── roles/
|
||||
│ │ │ ├── architect/SKILL.md
|
||||
│ │ │ ├── developer/SKILL.md
|
||||
│ │ │ ├── reviewer/SKILL.md
|
||||
│ │ │ ├── security/SKILL.md
|
||||
│ │ │ ├── qa/SKILL.md
|
||||
│ │ │ └── ops/SKILL.md
|
||||
│ │ ├── design/
|
||||
│ │ │ ├── grill-me/SKILL.md
|
||||
│ │ │ ├── architecture-review/SKILL.md
|
||||
│ │ │ └── break-into-issues/SKILL.md
|
||||
│ │ ├── factory/ ← Meta-skills: factory builds itself
|
||||
│ │ │ ├── write-issue-spec/SKILL.md # Drafts Gitea issue with EARS spec
|
||||
│ │ │ ├── write-skill/SKILL.md # Creates new SKILL.md files
|
||||
│ │ │ ├── write-workflow/SKILL.md # Creates workflow documents
|
||||
│ │ │ ├── write-adr/SKILL.md # Creates ADR documents
|
||||
│ │ │ ├── write-eval/SKILL.md # Creates eval.yaml for a skill
|
||||
│ │ │ ├── upgrade-skill/SKILL.md # Updates skill + eval on correction
|
||||
│ │ │ └── validate-skill/SKILL.md # Audits skill against standard
|
||||
│ │ ├── implement/
|
||||
│ │ │ ├── implement-feature/SKILL.md
|
||||
│ │ │ ├── refactor/SKILL.md
|
||||
│ │ │ ├── write-docs/SKILL.md
|
||||
│ │ │ ├── diagnose/SKILL.md
|
||||
│ │ │ └── tdd/
|
||||
│ │ │ ├── SKILL.md
|
||||
│ │ │ ├── tests.md
|
||||
│ │ │ ├── mocking.md
|
||||
│ │ │ └── refactoring.md
|
||||
│ │ ├── test/
|
||||
│ │ │ ├── write-tests/SKILL.md
|
||||
│ │ │ ├── generate-test-data/SKILL.md
|
||||
│ │ │ └── review-test-coverage/SKILL.md
|
||||
│ │ ├── review/
|
||||
│ │ │ ├── code-review/SKILL.md
|
||||
│ │ │ ├── security-review/SKILL.md
|
||||
│ │ │ ├── pr-description/SKILL.md # Includes ADDED/MODIFIED/REMOVED convention
|
||||
│ │ │ └── changelog-entry/SKILL.md
|
||||
│ │ ├── deploy/
|
||||
│ │ │ ├── write-ci-pipeline/SKILL.md
|
||||
│ │ │ ├── write-deployment-config/SKILL.md
|
||||
│ │ │ ├── write-ai-review-workflow/SKILL.md
|
||||
│ │ │ └── deployment-checklist/SKILL.md # Verifies linked issue exists
|
||||
│ │ ├── operate/
|
||||
│ │ │ ├── write-runbook/SKILL.md
|
||||
│ │ │ ├── incident-diagnosis/SKILL.md
|
||||
│ │ │ ├── post-mortem/SKILL.md
|
||||
│ │ │ └── inspect-deployment/SKILL.md
|
||||
│ │ ├── iac/
|
||||
│ │ │ ├── write-ansible-role/SKILL.md
|
||||
│ │ │ ├── write-terraform-module/SKILL.md
|
||||
│ │ │ ├── write-k8s-manifest/SKILL.md
|
||||
│ │ │ ├── write-docker-compose/SKILL.md
|
||||
│ │ │ ├── proxmox-vm-spec/SKILL.md
|
||||
│ │ │ ├── iac-security-review/SKILL.md
|
||||
│ │ │ └── write-molecule-test/SKILL.md
|
||||
│ │ ├── gitea/
|
||||
│ │ │ ├── setup-gitea-mcp/SKILL.md
|
||||
│ │ │ ├── post-pr-review/SKILL.md
|
||||
│ │ │ └── create-issue/SKILL.md
|
||||
│ │ └── cross-cutting/
|
||||
│ │ ├── session-handoff/SKILL.md
|
||||
│ │ ├── governance-check/SKILL.md
|
||||
│ │ ├── git-commit-message/SKILL.md
|
||||
│ │ └── git-guardrails/SKILL.md
|
||||
│ │
|
||||
│ ├── evals/ # Co-located eval suites (one dir per skill)
|
||||
│ │ └── <category>/<skill>/eval.yaml
|
||||
│ │
|
||||
│ └── workflows/ # Multi-skill orchestration documents
|
||||
│ ├── feature-workflow.md
|
||||
│ ├── bugfix-workflow.md
|
||||
│ └── iac-workflow.md
|
||||
│
|
||||
├── docs/
|
||||
│ ├── spec/ # Living system spec — what the system does NOW
|
||||
│ │ ├── overview.md # Purpose, scope, system boundaries
|
||||
│ │ ├── architecture.md # Current architecture (updated with code)
|
||||
│ │ └── <domain>.md # Per-domain functional specs (add as needed)
|
||||
│ ├── adr/ # Architecture Decision Records (append-only, never deleted)
|
||||
│ │ └── NNN-title.md
|
||||
│ ├── runbooks/ # Operational procedures
|
||||
│ └── api/ # Generated from code where possible
|
||||
│
|
||||
├── references/
|
||||
│ ├── stack-versions.md # Authoritative version registry
|
||||
│ └── proxmox-cluster.md # Cluster context (AI training data can't know this)
|
||||
│
|
||||
└── src/, tests/, infra/ # Implementation
|
||||
```
|
||||
|
||||
**What is NOT in the working tree:**
|
||||
|
||||
| Excluded | Where it lives instead |
|
||||
|---|---|
|
||||
| Active feature work | Gitea issue (spec) + feature branch (code) |
|
||||
| Completed change history | Git commit history + closed Gitea issues |
|
||||
| Working drafts / scratch | `.gitignore`d `scratch/` dir or local only |
|
||||
| Session handoff notes | Created locally; deleted once session resumes |
|
||||
|
||||
**Design rationale:**
|
||||
|
||||
`docs/spec/` replaces `openspec/specs/` and `openspec/changes/`. It is the single source of truth for what the system does now. It is updated in the same PR as any behaviour change — there is no separate archive step, because git history is the archive.
|
||||
|
||||
`docs/adr/` is append-only. ADRs are never deleted or edited after they are marked Accepted — only superseded by a new ADR that references them. This is what makes architectural decisions auditable across time.
|
||||
|
||||
`factory/` skill category is new. It contains meta-skills that build the factory itself: writing new skills, evals, ADRs, and issue specs. The factory is self-extending.
|
||||
|
||||
`.agents/skills/` as canonical skill path: consistent with Anthropic's SKILL.md open standard (December 2025). Tool adapters (CLAUDE.md, `.cursor/rules`) reference this; they do not duplicate it.
|
||||
|
||||
---
|
||||
|
||||
## 7. Phased Implementation Plan
|
||||
|
||||
### Phase 1 — Foundation (Week 1–2)
|
||||
|
||||
Goal: A working factory skeleton that a real task can be run through end-to-end.
|
||||
|
||||
**Acceptance criteria (EARS format):**
|
||||
- WHEN any task is started, THE system SHALL have a Gitea issue with an EARS-format spec before a branch is created
|
||||
- WHEN a Claude Code session begins, THE system SHALL have `AGENTS.md` and `CONTEXT.md` in context
|
||||
- WHEN any skill is invoked, THE system SHALL load only that skill's SKILL.md (not the full library)
|
||||
- WHEN a PR is raised, THE system SHALL include updates to `docs/spec/` if behaviour changed
|
||||
|
||||
**Tasks:**
|
||||
|
||||
1. Commit `AGENTS.md` — merge existing governance content; validate against principles doc
|
||||
2. Generate `CONTEXT.md` — manually populate shared vocabulary (10–15 domain terms to start)
|
||||
3. Create `docs/spec/overview.md` and `docs/spec/architecture.md` — initial living spec, even if sparse
|
||||
4. Create `docs/adr/` with an `adr-template.md`
|
||||
5. Implement Phase 1 skill set (7 files):
|
||||
- `roles/architect`, `roles/developer`, `roles/reviewer`
|
||||
- `factory/write-issue-spec` (primary pre-work skill — replaces old write-spec)
|
||||
- `factory/write-adr` (produces ADR documents)
|
||||
- `implement/tdd` (multi-file structure from Pocock)
|
||||
- `cross-cutting/session-handoff`
|
||||
6. Set up Gitea issue template using the spec structure from Section 5
|
||||
|
||||
**Quality gate:** Run one real feature end-to-end: write-issue-spec → post issue to Gitea → create branch → tdd → code-review → update docs/spec/ → pr-description → merge. Human reviews all outputs. No eval infrastructure yet — manual review is the gate.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2 — Full Skill Library (Week 3–4)
|
||||
|
||||
Goal: All 27 skills from the skills index exist as real SKILL.md files.
|
||||
|
||||
**Priority order within Phase 2:**
|
||||
1. Remaining factory meta-skills: `write-skill`, `write-eval`, `upgrade-skill`, `validate-skill`, `write-workflow`
|
||||
2. Remaining roles: `security`, `qa`, `ops`
|
||||
3. `review/code-review` and `review/security-review` — highest daily use
|
||||
4. `design/grill-me` and `design/break-into-issues` — completes the planning chain
|
||||
5. IaC skills (Ansible, Terraform, K8s, Docker Compose, Proxmox)
|
||||
6. Deploy and operate skills
|
||||
7. Gitea integration skills
|
||||
8. New skills from Pocock: `implement/diagnose`, `cross-cutting/git-guardrails`
|
||||
|
||||
**Skill authoring standard:**
|
||||
- Every SKILL.md must have: `name`, `description` (trigger), constraints section, explicit failure handling, version metadata in frontmatter
|
||||
- Complex skills (tdd, write-spec, architect) use multi-file pattern: `SKILL.md` + supplementary `.md` files
|
||||
- Description field is the trigger — test it by asking: "would this description activate the skill correctly for the intended case AND exclude adjacent cases?"
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Eval Infrastructure (Week 5–6)
|
||||
|
||||
Goal: Every skill has an `eval.yaml`; CI runs evals on change; weekly regression cron is live.
|
||||
|
||||
**Per-skill eval requirements:**
|
||||
- Trigger tests: explicit invocation, implicit invocation, ≥1 negative case (adjacent task that must NOT activate)
|
||||
- Output tests: deterministic assertions (JSON schema, contains, regex) as primary gate; LLM-as-judge for quality catches
|
||||
- Every human correction to a skill output generates a new test case
|
||||
|
||||
**Tooling:** promptfoo (MIT, YAML-based, self-hosted). Version-pin to a known-good release (not `latest`) given the OpenAI acquisition in March 2026. Document DeepEval and Arize Phoenix as fallback options.
|
||||
|
||||
**CI rule:**
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
paths:
|
||||
- '.agents/skills/**'
|
||||
jobs:
|
||||
skill-evals:
|
||||
runs-on: self-hosted
|
||||
steps:
|
||||
- run: promptfoo eval --config .agents/evals/${{ matrix.skill }}/eval.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Workflow Wiring and Token Optimisation (Week 7–8)
|
||||
|
||||
Goal: Multi-skill workflows are documented and tested; token cost is measurable and managed.
|
||||
|
||||
**Workflow documents** (in `.agents/workflows/`):
|
||||
- `feature-workflow.md`: grill-me → write-issue-spec → [human posts issue to Gitea] → architect-review → [human gate] → break-into-issues → tdd + implement-feature → code-review → security-review → [human gate] → update docs/spec/ → pr-description → deployment-checklist
|
||||
- `bugfix-workflow.md`: Lean path — diagnose → tdd → code-review → [human gate] → git-commit-message
|
||||
- `iac-workflow.md`: write-issue-spec → write-[iac-type] → iac-security-review → [human gate] → write-ci-pipeline
|
||||
|
||||
**Token optimisation measures to implement:**
|
||||
- Prompt caching enabled for all role and cross-cutting skills (stable system prompts)
|
||||
- Orientation block in each skill: role context loaded first, compresses subsequent interaction
|
||||
- `caveman` output mode (from Pocock) available as a flag for high-volume generation tasks
|
||||
- Model routing: route simple code review to Haiku; complex architectural reasoning to Sonnet/Opus
|
||||
|
||||
---
|
||||
|
||||
## 8. What NOT to Build
|
||||
|
||||
Explicit exclusions with rationale:
|
||||
|
||||
| Excluded pattern | Why |
|
||||
|---|---|
|
||||
| `openspec/changes/` directory | Work tracking belongs in Gitea issues and feature branches, not committed files. Working documents in the working tree create drift rather than prevent it. |
|
||||
| Import BMAD, Spec Kit, or OpenSpec as runtime dependencies | Factory convention lives in committed files; runtime dependencies create maintenance debt and tool-specific format lock-in |
|
||||
| Concurrent agent teams | Research confirms this compounds error rates and explodes token cost. Sequential roles remain the pattern. |
|
||||
| LLM auto-memory (`~/.claude/memory`) | Machine-local, not repo-shared. LESSONS.md is the repo mechanism. |
|
||||
| Single all-skills-loaded context | Skills load on demand. Dumping the full library into context is the failure mode skills solve. |
|
||||
| Spec Kit's Python CLI as a hard dependency | Python 3.11+ runtime requirement for a workflow tool is unnecessary friction. The slash command pattern matters — implement it in SKILL.md. |
|
||||
| "Living spec" synchronisation platforms (Intent/Augment) | Requires a platform product. `docs/spec/` updated in-PR is the factory's approach — spec drift is managed by process and CI, not external software. |
|
||||
| Concurrent role execution | Always sequential on demand. Parallel agent teams are not the factory pattern. |
|
||||
|
||||
---
|
||||
|
||||
## 9. Best Practice Sources
|
||||
|
||||
Organised by implementation area. Use these when writing each skill:
|
||||
|
||||
**Issue spec writing:**
|
||||
- EARS notation: Mavin & Maiden, *EARS: The Easy Approach to Requirements Syntax* (IEEE, 2009)
|
||||
- Addy Osmani's spec guide: `addyosmani.com/blog/good-spec/` — treat spec like code; commit it; agents read git diff to understand changes
|
||||
- OpenSpec delta notation (ADDED/MODIFIED/REMOVED): `github.com/Fission-AI/OpenSpec`
|
||||
- GitHub Spec Kit EARS examples: `github.com/github/spec-kit`
|
||||
|
||||
**TDD and testing:**
|
||||
- Matt Pocock's tdd skill (multi-file pattern): `github.com/mattpocock/skills/tdd/`
|
||||
- Test architecture: `test-driven.io` — Roy Osherove, *The Art of Unit Testing* (3rd ed.)
|
||||
- Spec-to-test derivation (not implementation-to-test): factory's own principles doc, Topic 3
|
||||
|
||||
**Security:**
|
||||
- OWASP LLM Top 10: `owasp.org/www-project-top-10-for-large-language-model-applications/`
|
||||
- OWASP Agentic AI Top 10 (AST10, Incubator draft, Q3 2026 target): `owasp.org/www-project-agentic-ai-top-10/`
|
||||
- Veracode 2025 GenAI Code Security Report (45% OWASP Top 10 in AI-generated samples)
|
||||
- CodeRabbit December 2025 report (1.7× more issues in AI-generated code)
|
||||
|
||||
**Prompt/context engineering:**
|
||||
- LangChain 4-strategy framework (write, select, compress, isolate): `blog.langchain.dev`
|
||||
- Context Engineering paper: `arxiv.org/pdf/2603.09619` (2026)
|
||||
- OOLONG benchmark (long context degradation): `arxiv.org/abs/2511.02817`
|
||||
|
||||
**IaC:**
|
||||
- Ansible idempotency: `docs.ansible.com/ansible/latest/playbook_guide/playbooks_best_practices.html`
|
||||
- Terraform style guide: `developer.hashicorp.com/terraform/language/style`
|
||||
- Molecule testing: `ansible.readthedocs.io/projects/molecule/`
|
||||
- Proxmox bpg provider: `github.com/bpg/terraform-provider-proxmox` (pin version; changes frequently)
|
||||
|
||||
**Eval infrastructure:**
|
||||
- promptfoo docs (version-pin): `promptfoo.dev/docs/` — acquired OpenAI March 2026; monitor neutrality
|
||||
- DeepEval (fallback): `docs.confident-ai.com`
|
||||
- Arize Phoenix (self-hosted fallback): `phoenix.arize.com`
|
||||
- SkillsBench (benchmark methodology): `arxiv.org/abs/2602.12670`
|
||||
|
||||
**Agent role design:**
|
||||
- gstack reference implementation: `github.com/garrytan/gstack` — inspect role structure; caveat productivity claims (810× self-reported, non-verifiable methodology)
|
||||
- AORCHESTRA orchestrator-worker paper: `arxiv.org/abs/2602.03786` (+16.28% on GAIA/SWE-Bench, Gemini-specific)
|
||||
- Cognitive separation principle: `cio.com/article/4166035` (Anthropic, May 2026)
|
||||
|
||||
**Token optimisation:**
|
||||
- Prompt caching: `docs.anthropic.com/en/docs/build-with-claude/prompt-caching`
|
||||
- Model routing guidance: Anthropic cost documentation (current pricing)
|
||||
|
||||
---
|
||||
|
||||
## 10. Skill Authoring Guide
|
||||
|
||||
When writing each skill file, follow this template:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <category>/<skill-name>
|
||||
description: <trigger description — this is what the agent reads to decide whether to load this skill>
|
||||
version: 1.0
|
||||
updated: YYYY-MM
|
||||
references:
|
||||
- <source 1>
|
||||
- <source 2>
|
||||
---
|
||||
|
||||
## Role
|
||||
|
||||
<One sentence: who is the agent when this skill is active>
|
||||
|
||||
## When to use / When NOT to use
|
||||
|
||||
<Explicit inclusion and exclusion criteria — the negative cases are as important as positive>
|
||||
|
||||
## Required inputs
|
||||
|
||||
<What must be provided before this skill proceeds>
|
||||
|
||||
## Constraints
|
||||
|
||||
<Non-negotiable rules — what the agent must always/never do>
|
||||
|
||||
## Process
|
||||
|
||||
<Step-by-step — numbered, not bullet points>
|
||||
|
||||
## Output format
|
||||
|
||||
<Exactly what the skill produces; structured if the next step in a workflow depends on it>
|
||||
|
||||
## Failure handling
|
||||
|
||||
<What to return if inputs are missing or a step fails — never silent failure>
|
||||
```
|
||||
|
||||
**Trigger description quality check:** Before writing the body, validate the description against three test queries: (1) an explicit invocation that should activate it, (2) an implicit invocation that should activate it, (3) an adjacent task that must NOT activate it. If the description fails any test, rewrite before proceeding.
|
||||
|
||||
---
|
||||
|
||||
## 11. Success Metrics
|
||||
|
||||
The factory is working when:
|
||||
|
||||
| Metric | Target |
|
||||
|---|---|
|
||||
| Issue spec before branch | 100% of non-trivial features have a Gitea issue with EARS spec before branch creation |
|
||||
| Spec–implementation sync | 0 PRs merged that change behaviour without updating `docs/spec/` |
|
||||
| HITL gate compliance | 100% of irreversible actions have a human approval before execution |
|
||||
| Eval coverage | Every skill has trigger tests and output tests before use in production |
|
||||
| LESSONS.md graduation rate | Recurring patterns promoted to CONTEXT.md within 2 weeks of third occurrence |
|
||||
| Session handoff rate | All sessions that pause mid-task have a handoff document |
|
||||
| Model regression detection | Weekly eval cron catches skill degradation before it reaches production use |
|
||||
|
||||
---
|
||||
|
||||
## 12. Open Questions Before Grill-Me Sessions
|
||||
|
||||
These must be resolved before implementation begins. Most are inputs *to* the grill-me sessions, not outputs from them — bring them as context, not as questions to answer from scratch.
|
||||
|
||||
1. **Primary language/stack for this repo:** Skill body content varies by language. What is the repo's primary stack? This determines test framework defaults (pytest vs Vitest), linting rules, and IaC tooling.
|
||||
|
||||
2. **Gitea instance available?** The gitea integration skills and the issue-spec workflow both require a live Gitea instance with MCP access. If not yet live, defer gitea skills and use manual issue creation in Phase 1.
|
||||
|
||||
3. **Gitea issue template:** The issue spec structure (why / requirements / design notes / tasks) needs to be set as the default issue template. Confirm which Gitea repo this applies to before Phase 1.
|
||||
|
||||
4. **promptfoo hosting:** Self-hosted promptfoo on the homelab, or Gitea Actions cloud runners? This affects eval architecture.
|
||||
|
||||
5. **`CONTEXT.md` bootstrap:** What are the 10–15 most important domain terms for this project? These must be manually written before any skill invokes CONTEXT.md.
|
||||
|
||||
6. **`docs/spec/` initial scope:** How much of the current system needs to be described in the initial living spec before Phase 1 begins? Even a sparse overview.md + architecture.md is better than nothing — agents need something to read before they can update it correctly.
|
||||
|
||||
---
|
||||
|
||||
## 13. Claude Code Grill-Me Sessions — Handoff
|
||||
|
||||
This document is the research and recommendation layer. It is not the authoritative implementation plan. The authoritative plan emerges from grill-me sessions in Claude Code, which produce the vision, roadmap, and ADRs that this guidance must then be adapted to.
|
||||
|
||||
### What needs a grill-me session
|
||||
|
||||
Each of the following warrants its own session. They are ordered by dependency — earlier sessions produce inputs that later sessions need.
|
||||
|
||||
| Session | Produces | Feeds into |
|
||||
|---|---|---|
|
||||
| 1. Repo vision and goals | Vision statement, success criteria, what the factory is and is not for | All subsequent sessions; CONTEXT.md bootstrap |
|
||||
| 2. Stack and tooling decisions | Primary language(s), test frameworks, IaC tooling, CI platform | Skill body content for implement/, test/, iac/ skills; stack-versions.md |
|
||||
| 3. Roadmap and phasing | Milestone sequence, phase gates, what "done" looks like per phase | Phase plan (Section 7 of this document becomes concrete) |
|
||||
| 4. ADR: documentation structure | Decision on docs/spec/ scope, Gitea issue template, ADR conventions, Diátaxis applicability | write-issue-spec skill, docs/spec/ initial structure, issue template |
|
||||
| 5. ADR: governance file structure | Decision on which governance files exist, what each owns, what adapters look like | AGENTS.md, CLAUDE.md, CONTEXT.md |
|
||||
| 6. ADR: skill library scope | Which skills to build first; factory/ vs other categories; which to defer | Phase 1 and 2 skill build order |
|
||||
|
||||
### What to give Claude Code before each session
|
||||
|
||||
For each grill-me session, load the following into Claude Code context:
|
||||
|
||||
```
|
||||
# Mandatory context for all sessions
|
||||
AGENTS.md
|
||||
ai-coding-factory-principles.md
|
||||
ai-coding-factory-skills-index.md
|
||||
|
||||
# This document (the implementation guidance)
|
||||
ai-coding-factory-implementation-guidance.md
|
||||
|
||||
# Add for sessions 2-6 (outputs from session 1)
|
||||
CONTEXT.md (once bootstrapped)
|
||||
docs/adr/ (any ADRs already produced)
|
||||
```
|
||||
|
||||
The grill-me skill will refuse to proceed until every decision is explicit. Let it. The friction is the point.
|
||||
|
||||
### How ADR outputs from Claude Code relate to this document
|
||||
|
||||
ADRs produced in Claude Code override specific recommendations in this document where they conflict. The override rule:
|
||||
|
||||
- This document recommends a pattern (e.g., `openspec/changes/<name>/` structure).
|
||||
- The ADR session may decide on a different structure for legitimate reasons.
|
||||
- The ADR is authoritative. Update this document's relevant section to reference the ADR and mark it as superseded.
|
||||
- This document's research rationale still stands — the ADR should record *why* it diverged, not just *what* it decided.
|
||||
|
||||
The intended final state: every concrete recommendation in this document either has a matching ADR confirming it, or a note saying "superseded by ADR-NNN." At that point, this document transitions from guidance to historical record and the ADRs become the implementation reference.
|
||||
|
||||
### Suggested grill-me session prompt for Claude Code
|
||||
|
||||
```
|
||||
/grill-me
|
||||
|
||||
Context: I'm building an AI coding factory — a governed, AI-assisted development environment
|
||||
implemented as committed files in a repo. I have completed research and implementation
|
||||
guidance (loaded in context). I need to establish [session topic] before building.
|
||||
|
||||
Grill me until every decision is explicit. Do not let me proceed with vague answers.
|
||||
Output: a structured decision record suitable for converting to an ADR.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*This is a living guidance document. Update it as grill-me sessions produce decisions. When an ADR supersedes a recommendation, mark the section with `> Superseded by ADR-NNN` and link the ADR. Version in git alongside the rest of the factory.*
|
||||
|
||||
261
docs/research/ai-coding-factory/ai-coding-factory-principles.md
Normal file
261
docs/research/ai-coding-factory/ai-coding-factory-principles.md
Normal file
@@ -0,0 +1,261 @@
|
||||
# AI Coding Factory — Principles
|
||||
|
||||
**Version:** 1.1 (Deep research corrections applied May 2026)
|
||||
**Scope:** Design, structure, and operation of the AI-assisted software development and infrastructure management factory
|
||||
**Audience:** Humans maintaining and extending the factory; AI agents making design decisions within it
|
||||
**Derivation:** Derived from sourced research across ten design topics. Principles are evidence-based, not aspirational.
|
||||
**Operative files:** `AGENTS.md` (agent governance) | `CONTEXT.md` (project conventions) | `.agents/skills/` (skill library)
|
||||
**Research basis:** `ai-coding-factory-research.md`
|
||||
|
||||
The factory is a governed AI-assisted development environment. This document defines the principles behind its structure — why it is built the way it is and what governs changes to it. It is not an implementation guide; the implementation lives in the repository files this document governs.
|
||||
|
||||
---
|
||||
|
||||
## 1. Core Architecture
|
||||
|
||||
**Tool-agnostic core, thin tool-specific adapters.**
|
||||
Write conventions, principles, and skills once in plain language with no tool-specific references. Translate into tool-specific format (CLAUDE.md, copilot-instructions.md, cursor rules) with thin adapter files that reference the source, not duplicate it. When tools change, only the adapter changes. This is the same separation-of-concerns pattern as the governance constitution: conventions are stable; tool formats are not.
|
||||
|
||||
**One source of truth per concern.**
|
||||
AGENTS.md is the single source of governance rules. CONTEXT.md is the single source of project conventions. Each skill is the single source for its task type. Duplicating content into tool-specific files creates maintenance debt and drift — it is prohibited.
|
||||
|
||||
**Spec before implementation.**
|
||||
Every non-trivial implementation task begins with a spec. The spec is the prompt for all downstream work — it defines what the Developer role builds, what QA tests, and what the Reviewer evaluates. Without a spec the factory has no contract between its stages. Spec-Driven Development is rapidly emerging but not yet mainstream in production: GitHub Spec Kit (~92K stars, May 2026), AWS Kiro (~250K signups over its first preview period), dedicated DeepLearning.AI course. No controlled study comparing SDD with ad-hoc prompting outcomes exists as of May 2026. Even so: SDD overhead is documented and real — skip the spec skill for single-file fixes, obvious one-line changes, or tasks the human can describe completely in one sentence. GitHub's own documentation notes Spec Kit is "best suited for greenfield builds and large feature work rather than small bug fixes." (Thoughtworks Radar: "Assess, not Adopt", November 2025, with the pointed observation: *"We may be relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn't scale."*)
|
||||
|
||||
**Context lives in files, not sessions.**
|
||||
AI sessions are ephemeral; committed files are durable. Every decision, lesson, and convention that needs to survive a session boundary must be committed to a file. Claude Code auto-memory is stored per-machine and is not repo-shared — it is not a substitute for committed context files and must not be treated as one.
|
||||
|
||||
**Agent instruction compliance is a supplementary signal, not a guarantee.**
|
||||
Even the best models perfectly follow fewer than 30% of complex multi-constraint agentic instructions (AGENTIF benchmark, Tsinghua University 2025). Skills improve compliance; they do not guarantee it. Deterministic enforcement — CI gates, linters, pre-commit hooks, CONTROLS.md — is the primary enforcement layer. Skill instructions are a supplementary signal. Hard prohibitions in AGENTS.md must never be replicated only in a skill.
|
||||
|
||||
---
|
||||
|
||||
## 2. Skills Library
|
||||
|
||||
**SKILL.md is the universal packaging unit.**
|
||||
The Agent Skills open standard (agentskills.io, December 2025) is implemented by 26+ tools from competing vendors: Anthropic, Microsoft, OpenAI, Google, JetBrains, AWS, Block, and others. ⚠️ Earlier research cited "32+"; consistent secondary sources as of May 2026 document 26 confirmed adopters. The canonical directory is `.agents/skills/`. Tool-specific locations (`.claude/skills/`, `~/.codex/skills/`) are thin adapters that reference this canonical path.
|
||||
|
||||
**The description field is the trigger — write it first.**
|
||||
If a skill does not activate, the problem is almost never the instructions. It is the description. Write and test the description field before writing the skill body. Test against explicit triggers, implicit triggers, and negative cases (adjacent tasks that must not activate the skill). The description-first discipline is the single most effective practice for reliable skill activation.
|
||||
|
||||
**Organise on a phase × domain matrix.**
|
||||
The phase axis (design, implement, test, review, deploy, operate) captures what is being done in the SDLC. The domain axis (python, typescript, ansible, terraform, docker, k8s, proxmox) captures the tool or language context. Phase-axis skills apply across domains; domain-axis skills are tool-specific. Cross-cutting skills (session-handoff, governance-check, git-commit-message) have no single phase home and live outside the matrix.
|
||||
|
||||
**Keep each skill focused; move depth to references/.**
|
||||
A skill with a large body and references can consume tens of thousands of tokens per invocation; cumulative cost dominates the context budget when multiple skills are active simultaneously (SkillReducer paper, arxiv.org/abs/2603.29919 — confirms a "less-is-more effect": 48% description / 39% body compression yields +2.8% quality improvement). Skill body: under 500 lines. The previous "3,000-token degradation ceiling" rationale (Levy, Jacoby, Goldberg 2024) was a misreading of that paper — 3,000 tokens was the maximum tested, not a degradation threshold. The 500-line limit is now justified by the SkillReducer "less-is-more" finding and OOLONG benchmark evidence that aggregation reasoning is <50% at 128K even for frontier models — keep skills lean for independent reasons.
|
||||
|
||||
**Version every skill; track changes in git.**
|
||||
Skill frontmatter carries a version field. Meaningful changes increment the version. IaC skills additionally carry explicit tool and provider version in frontmatter metadata — AI training data has a cutoff and provider APIs change; the skill must declare the version it was written for.
|
||||
|
||||
**Never install third-party skills without reading them in full.**
|
||||
The skills ecosystem had a documented security crisis in early 2026: 36% prompt injection rate across 3,984 public skills (Snyk ToxicSkills, February 2026); 341+ malicious skills on ClawHub delivering commodity macOS malware (ClawHavoc, February 2026); formalised as OWASP AST01 (Malicious Skills, Critical) in the OWASP Agentic Skills Top 10 Incubator project (draft, v1.0 targeted Q3 2026 — not a ratified standard). AST01 applies specifically to skills from public registries (ClawHub, skills.sh). For self-authored private factory skills, the relevant risk categories are AST03 (Over-Privileged), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance). Third-party skills use a "Markdown-as-installer" paradigm — read every file and bundled script before running anything.
|
||||
|
||||
**Apply `allowed-tools` restrictions, especially for production-touching skills.**
|
||||
Self-authored skills are not exposed to the supply chain attack vectors above, but over-privileged skills create blast radius risk. Scope each skill's tool access to the minimum it needs. Use `disable-model-invocation: true` for production-touching skills where appropriate.
|
||||
|
||||
---
|
||||
|
||||
## 3. Skill Design
|
||||
|
||||
**Constraints are the highest-ROI element.**
|
||||
AI models overengineer — extra files, unnecessary abstractions, unrequested scope expansion. An explicit constraints section prevents this more reliably than any other prompting technique. Every implementation skill must have one.
|
||||
|
||||
**End every implementation skill with a self-check instruction.**
|
||||
A self-check asking the agent to verify specific correctness criteria before responding catches a measurable proportion of errors before they reach human review. The criteria must be skill-specific, not generic.
|
||||
|
||||
**Reference files by path; never load more than the task requires.**
|
||||
Skills load context files on demand. Load specific files for specific tasks rather than broad codebase access. "Decisions not descriptions" framing reduces token cost and improves relevance.
|
||||
|
||||
**Use XML tags only for complex skills.**
|
||||
XML tags improve structure on complex, multi-section skills (500+ tokens, three or more logical sections). On simpler skills, XML adds tokens without improving accuracy. Default to prose; add XML when complexity genuinely warrants it.
|
||||
|
||||
**Use `<thinking>` and named output tags for review and security skills.**
|
||||
For skills where reasoning is auditable — code review, security audit, architecture review — emit reasoning to a thinking block and structured output to a named tag. The reasoning becomes the audit log; the structured output feeds the next workflow step.
|
||||
|
||||
**Use EARS notation for acceptance criteria in specs.**
|
||||
EARS (Easy Approach to Requirements Syntax, developed at Rolls-Royce for safety-critical systems) makes acceptance criteria AI-parseable and testable:
|
||||
|
||||
```
|
||||
WHEN [trigger condition] AND [optional precondition] THE system SHALL [required behaviour]
|
||||
```
|
||||
|
||||
Tests can be mechanically generated from EARS criteria; they cannot be generated from prose requirements. The write-spec skill must produce EARS-format acceptance criteria, not ambiguous prose.
|
||||
|
||||
---
|
||||
|
||||
## 4. Workflow and Agent Chains
|
||||
|
||||
**Orchestrator-worker is the production pattern for multi-phase workflows.**
|
||||
A single orchestrator decomposes the task; isolated subagents each run in a fresh context with a dedicated system prompt, execute one task, and return a compact summary. No peer-to-peer channels; no shared mutable state. This bounds coordination cost and prevents the quadratic communication explosion of peer architectures. (AORCHESTRA, arXiv 2602.03786: +16.28% relative improvement on GAIA, SWE-Bench, and Terminal-Bench — Gemini-3-Flash specific; transfer to Claude workflows not independently validated. The architectural pattern is independently confirmed across multiple sources.)
|
||||
|
||||
**Define the WorkflowContext schema before writing any skill in a chain.**
|
||||
Every skill in a multi-step workflow must know exactly what it receives and exactly what it must return. Compact structured outputs: only fields the next step needs. Every skill must handle failure explicitly — return error fields, never silent failures.
|
||||
|
||||
**Place HITL gates before irreversible actions, not at the end of a chain.**
|
||||
A human approval gate at the end of a long chain is not a safeguard — it is rubber-stamping. Gates go immediately before any action that is irreversible, high-blast-radius, or production-touching.
|
||||
|
||||
**The spec is the contract between all workflow stages.**
|
||||
Workflows without a spec drift; agents hallucinate direction and scope. The spec file is the shared reference that keeps Developer, Reviewer, QA, and Ops working toward the same target. Spec-first is both a quality practice and a token reduction strategy — a written spec eliminates repeated re-explanation of intent across sessions and agent calls.
|
||||
|
||||
---
|
||||
|
||||
## 5. Role Architecture
|
||||
|
||||
**Six roles cover the factory.**
|
||||
Each role activates a distinct cognitive mode. Roles run sequentially on demand — not concurrently.
|
||||
|
||||
| Role | Cognitive mode | Primary output | HITL gate |
|
||||
|---|---|---|---|
|
||||
| Architect | Systems thinking, adversarial design | Design doc, API contracts, ADRs | Human approves before build |
|
||||
| Developer | Implementation against plan | Code, tests, docs | Human reviews diff before commit |
|
||||
| Reviewer | Staff-engineer perspective | Severity-rated findings | Human decides which to act on |
|
||||
| Security | Adversarial audit (OWASP/STRIDE) | Security findings | Human approves before merge |
|
||||
| QA | Adversarial testing | Test results, regression tests | Human reviews before PR opened |
|
||||
| Ops | Infrastructure and deployment | Pipelines, manifests, runbooks | Human approves before any apply |
|
||||
|
||||
**Cognitive separation is the mechanism, not a convention.**
|
||||
Different phases require different cognitive modes; planning is not review; review is not shipping. Separating plan review from code review specifically prevents the failure mode where an agent simultaneously rubber-stamps at the macro level and drowns in detail at the micro level (gstack reference implementation, ~96K stars, May 2026). Concurrent agent teams compound error rates and explode token cost — they are not the factory pattern. Note: gstack's 810× productivity claim uses a self-defined "logical LOC" metric, a part-time 2013 baseline, and confounds parallel sessions with role design — the role-separation cognitive insight stands independently of the productivity claim.
|
||||
|
||||
**Each role's output is the next role's structured input.**
|
||||
If a role's output is unstructured, the next role's input is ambiguous and the human checkpoint between them cannot be meaningful. The structured handoff is what makes the chain auditable, not just convenient.
|
||||
|
||||
---
|
||||
|
||||
## 6. SDLC Phase Principles
|
||||
|
||||
**AI-generated code is untrusted by default and requires security-aware review.**
|
||||
Veracode's 2025 GenAI Code Security Report found that **45% of tested AI-generated samples introduced OWASP Top 10 vulnerabilities** across 100+ LLMs (4 languages, 80 coding tasks). ⚠️ The "2.74× more vulnerabilities" figure is misattributed to Veracode in many sources; it comes from CodeRabbit's December 2025 report and applies specifically to XSS vulnerabilities, not all vulnerability types. The overall CodeRabbit finding is 1.7× more issues, 1.57× more security findings. Independent academic data (Schreiber & Tippe, arXiv 2510.26103): real-world AI-generated files in GitHub repositories showed only 12.1% CWE-mapped vulnerability rate — substantially below controlled-prompt benchmark rates. All sources are vendor or vendor-adjacent; the directional finding (AI code has more vulnerabilities without review; security-aware review substantially reduces this) is independently confirmed. Source: businesswire.com/news/home/20250730694951; backslash.security (April 2026).
|
||||
|
||||
**Generate tests from the spec, not from the implementation.**
|
||||
Tests derived from implementation validate behaviour, which may itself be wrong — circular reasoning. Tests derived from the spec validate intent. This is a structural constraint on the write-tests skill: it requires spec input, not just code input.
|
||||
|
||||
**Design both feedback loops explicitly.**
|
||||
The short loop (finding → immediate fix) happens by default. The long loop (recurring pattern → standing convention) must be deliberately designed. The session-handoff skill explicitly prompts LESSONS.md extraction before closing a session. Convention graduation threshold: three or more LESSONS.md entries covering the same pattern promote to CONTEXT.md as a standing rule.
|
||||
|
||||
**LESSONS.md is the repo-level feedback mechanism.**
|
||||
Claude Code auto-memory is per-machine and not repo-shared. For lessons to survive machine boundaries and become team-inheritable, they must be committed to `LESSONS.md` in the repo root. This is a correction from common practice — auto-memory is not a substitute.
|
||||
|
||||
**Tier review findings by severity to prevent alert fatigue.**
|
||||
Critical: block. High: warn. Low: suggest. Alert fatigue from over-blocking is as destructive as under-blocking — it collapses the review loop into rubber-stamping, which is the failure mode it was designed to prevent.
|
||||
|
||||
---
|
||||
|
||||
## 7. IaC Principles
|
||||
|
||||
**Terraform provisions; Ansible configures. Keep them separate in CI.**
|
||||
Terraform handles Day 0 (initial provisioning and lifecycle management). Ansible handles Day 1+ (configuration of provisioned resources). Never call Ansible from Terraform provisioners — HashiCorp explicitly designates provisioners a "last resort"; they break idempotency and create state drift invisible to Terraform state. Keep them as distinct, sequential CI/CD stages.
|
||||
|
||||
**Idempotency is the required property for Ansible — not a best practice.**
|
||||
An Ansible task that is not idempotent is a bug. AI-generated Ansible correctly uses FQCNs and handlers but reliably misses idempotency guards. The write-ansible-role skill must explicitly review for: `shell`/`command` tasks without guards, missing `creates:` on file-creating tasks, unconditional handlers, and absent rollback strategies. Prefer `ansible.builtin.apt`, `copy`, and `template` over `shell`/`command`.
|
||||
|
||||
**Validation is deterministic and mandatory before any apply.**
|
||||
`terraform plan`, `ansible --check --diff`, `kubectl diff` are blocking validation steps, not optional review suggestions. No IaC apply runs without a prior validation pass.
|
||||
|
||||
**K8s manifests require resource limits, security contexts, and network policies.**
|
||||
41% of AI-generated backend code includes overly broad permissions. Resource limits, security contexts, and network policies are routinely absent from AI-generated K8s output. The iac-security-review skill checks these explicitly; the write-k8s-manifest skill must include them by default.
|
||||
|
||||
**Pin all versions — tools, providers, and images.**
|
||||
No `:latest` Docker tags. No unpinned Terraform providers. IaC skills carry tool/provider version in frontmatter metadata. `references/stack-versions.md` is the authoritative version registry for the repo. The bpg/proxmox provider changes frequently and without semver guarantees — pin explicitly, reference `references/proxmox-cluster.md` for cluster context that AI training data cannot know.
|
||||
|
||||
---
|
||||
|
||||
## 8. Validation and Evals
|
||||
|
||||
**Every skill has trigger tests and output tests.**
|
||||
Trigger tests verify the description field activates the skill correctly — explicit invocations, implicit invocations, and negative cases (adjacent tasks that must not activate). Output tests verify the skill body produces correct, structured output. Both are required; missing trigger tests is the most common oversight and the most consequential one.
|
||||
|
||||
**Co-locate `eval.yaml` with `SKILL.md`.**
|
||||
Evals live alongside the skill they test. A change to any file under `.agents/skills/**` triggers the co-located eval run in CI. Failing evals block merge.
|
||||
|
||||
**Deterministic assertions are the primary gate; LLM-rubric is secondary.**
|
||||
JSON schema, contains, and regex checks are the blocking assertions. LLM-as-judge catches quality issues that deterministic checks cannot. Both are needed; neither is sufficient alone.
|
||||
|
||||
**Every corrected skill behaviour generates a new eval case.**
|
||||
When a human corrects an agent's skill output, that correction becomes a test case. This is the mechanism by which the eval suite grows from real failures rather than hypothetical ones.
|
||||
|
||||
**Weekly regression evals catch model-version-induced degradation.**
|
||||
Model updates break established skill behaviour without warning and without changelogs that map to skill-level impact. A weekly cron run of the full eval matrix detects this before it reaches production use.
|
||||
|
||||
**Treat promptfoo with awareness of its acquisition.**
|
||||
Promptfoo is the current standard tool (MIT, YAML-based, self-hosted, CI-native). It was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. **Version-pin to a known-good release** rather than tracking `latest` — structural conflict of interest exists for non-OpenAI evaluation. Monitor for neutrality degradation. Maintain a fallback path: DeepEval (pytest-native, zero adoption friction for Python teams) and Arize Phoenix (self-hosted, vendor-neutral) are the documented alternatives. Validate eval suites per tool — no cross-tool empirical study of false-positive/false-negative rates has been published; portability is plausible but not confirmed.
|
||||
|
||||
---
|
||||
|
||||
## 9. Context and Token Architecture
|
||||
|
||||
**Orientation is the primary token sink — solve it with structure.**
|
||||
Agents spend up to 80% of token budget on orientation rather than work. The solution is well-structured, always-available context files, not prompt compression. CONTEXT.md and the reference file architecture solve this at the source.
|
||||
|
||||
**Progressive disclosure governs the load strategy.**
|
||||
Only skill name and description pairs load at startup (~30–50 tokens per skill). Full skill bodies load on trigger. Reference files load only when needed during execution. 30 skills cost approximately 900–1,500 tokens at startup. The full body cost is paid only when a skill activates.
|
||||
|
||||
| Context | Target | Load pattern |
|
||||
|---|---|---|
|
||||
| AGENTS.md | ≤2,000 tokens | Always loaded |
|
||||
| CONTEXT.md | ≤200 lines | Always loaded |
|
||||
| Per skill body | ≤500 lines | On demand — skill activation |
|
||||
| Per skill references | As needed | On demand — skill instruction |
|
||||
| Session handoff doc | ≤1,000 tokens | Compressed state per boundary |
|
||||
|
||||
**Route by task complexity — this is also a sustainability decision.**
|
||||
Using frontier models for tasks a smaller model handles adequately imposes unnecessary environmental and infrastructure cost on others, per the governance constitution §6.
|
||||
|
||||
| Task type | Model tier | Examples |
|
||||
|---|---|---|
|
||||
| Formatting, classification, commit messages | Lightweight (Haiku) | git-commit-message, changelog-entry, trigger eval runs |
|
||||
| Most coding and review | Standard (Sonnet) | implement-feature, write-tests, code-review, write-ci-pipeline, write-ansible-role |
|
||||
| Adversarial and complex | Flagship (Opus) | architecture-review, security-review (STRIDE), write-spec (ambiguous requirements), incident-diagnosis |
|
||||
|
||||
**Prompt caching is the highest single-ROI optimisation.**
|
||||
Place static content first (AGENTS.md, CONTEXT.md); dynamic input last. The per-token discount is 90% on cache reads ($0.30/MTok vs. $3.00/MTok for Sonnet 4.6). Actual aggregate savings depend on hit rate, which varies by workflow type; peer-quality agentic data (arXiv 2601.06007) shows 41–80% cost reduction. **TTL caveat:** Around March 2026, Anthropic silently regressed the default cache TTL from 1 hour to 5 minutes for Pro/API users (Max subscribers retained 1-hour TTL). Factor this into strategy — intra-session reuse is reliable under 5 minutes; inter-session reuse requires Max tier or explicit API configuration for 1-hour TTL at 2× write cost.
|
||||
|
||||
**Tokens per completed task is the sustainability metric.**
|
||||
It simultaneously tracks cost, carbon intensity, and whether AI is doing genuine work. Per-token costs fall; aggregate consumption rises because adoption scale outpaces efficiency gains (Jevons paradox). Measure it per task, not in aggregate.
|
||||
|
||||
---
|
||||
|
||||
## 10. MCP Integration
|
||||
|
||||
**Dedicated agent account per service — never use personal admin tokens.**
|
||||
Agent operations must be attributable to a non-human actor and revocable without affecting the human account. This improves audit attribution and selective revocation. Gitea PAT scopes are category-level — a `write:repository` token can still delete repositories regardless of account type. The dedicated account does not improve token granularity; it is an audit-hygiene and blast-radius control.
|
||||
|
||||
**Filter tools per skill with `--tools` (gitea-mcp).**
|
||||
Tool definition bloat reaches 55K–134K tokens before optimisation (Anthropic engineering blog, November 2025). Filter the MCP tool surface to reduce token cost and communicate intended scope. **Important: `--tools` is a client-facing capability filter, not a server-side security boundary** — the underlying PAT retains full Gitea permissions regardless of `--tools` configuration. gitea-mcp-compact fork claims 70–80% token reduction on list/search; ⚠️ very low maturity (0 stars, 0 forks).
|
||||
|
||||
**Ansible and Terraform execution stays human-controlled.**
|
||||
No MCP servers exist for these tools at production-ready quality. More importantly: Ansible and Terraform execution applies changes to infrastructure — these are irreversible, high-blast-radius actions that require HITL by the governance constitution §5. Human-controlled is not a gap in the design; it is the correct design.
|
||||
|
||||
**Ops role uses MCP for read operations; write operations require HITL.**
|
||||
Inspection (container status, deployment health, K8s node state) can run read-only via MCP. Remediation does not run via MCP without explicit human approval.
|
||||
|
||||
**Pin MCP server versions — gitea-mcp is at v1.2.0 as of May 2026.**
|
||||
gitea-mcp released v1.2.0 on May 10, 2026; pin to a specific tag. Retest on upgrade — tool schema changes break skills silently.
|
||||
|
||||
**MCP creates a distinct security boundary that needs explicit policy.**
|
||||
"MCP does not automatically make an AI system safe." Primary homelab threats: supply chain (malicious npm/uvx packages bundling MCP servers — multiple CVEs in 2025–2026) and prompt injection from untrusted content the agent reads. Transport-level auth (OAuth 2.1) is the correct model for multi-user HTTP deployments; for single-user stdio, local-trust is the designed security model. Log per the governance constitution §7.
|
||||
|
||||
**Claude Code does not currently support per-session or per-role MCP filtering natively.**
|
||||
Multiple open feature requests exist (issues #7328, #4380, #45293 in anthropics/claude-code). What currently exists: three install scopes (`local`, `project`, `user`) and an interactive `@` menu (v2.0.10+) for manual session-level tool selection. Sub-agents inherit all parent MCP tools. For programmatic per-role tool control, use the Claude Agent SDK (separate from Claude Code). Until native filtering ships, use the `@` menu manually per session role.
|
||||
|
||||
---
|
||||
|
||||
## Governance
|
||||
|
||||
**This document explains why the factory is structured as it is — it does not replace building it.**
|
||||
The principles here are the rationale for every file, skill, and workflow decision. The actual implementation lives in the repository. When an implementation decision conflicts with a principle here, update whichever is wrong — this document is not automatically correct.
|
||||
|
||||
**This document and the governance constitution govern different things and both apply.**
|
||||
`ai-constitution.md` governs AI behaviour, ethics, data handling, and security. This document governs the factory's design and structure. Neither replaces the other.
|
||||
|
||||
**Update this document when:**
|
||||
- A research finding it is based on is refuted or substantially updated (the challenges document lists the highest-priority open questions)
|
||||
- A new AI tool or MCP server is onboarded that changes how the factory operates
|
||||
- A design decision proves wrong in practice (three or more LESSONS.md entries on the same pattern is the signal)
|
||||
- At minimum: annually, aligned with the governance constitution review cycle
|
||||
|
||||
---
|
||||
|
||||
*Derived from: AI Coding Factory Research Session (May 2026). Version 1.1 — deep research corrections applied.*
|
||||
*Research basis: `ai-coding-factory-research.md` | Open challenges: `ai-coding-factory-challenges.md`*
|
||||
*Session audit trail: `ai-coding-factory-session.md`*
|
||||
*Governance foundation: `ai-constitution.md` | `AGENTS.md` | `HUMANS.md` | `CONTROLS.md`*
|
||||
638
docs/research/ai-coding-factory/ai-coding-factory-research.md
Normal file
638
docs/research/ai-coding-factory/ai-coding-factory-research.md
Normal file
@@ -0,0 +1,638 @@
|
||||
# AI Coding Factory — Research Document
|
||||
|
||||
**Version:** 1.4 (Deep research pass applied May 2026 — 12 challenges verified, 10 additional elements checked; corrections applied throughout)
|
||||
**Purpose:** Human reference document. The "why" behind every design decision in the factory. Contains sourced findings, counterarguments, bias flags, and provisional principles for all ten topics. Does not go into AI agent context — too long.
|
||||
|
||||
**How to use this document:** This is the research layer, not the operative layer. If you want to know *what to build*, read `ai-coding-factory-principles.md`. If you want to know *why a skill is structured the way it is*, challenge a finding, or update the evidence base, read the relevant topic here.
|
||||
|
||||
**Session:** May 2026
|
||||
**Methodology:** Topic-by-topic web research from reliable sources. Topics 1–4 were re-searched in a second pass after their results were cleared from session context. All source URLs are now verified from actual search results. A deep research pass (Session 3, May 2026) challenged all 12 documented open questions plus 10 additional elements; corrections are applied in version 1.4. Where specific numbers remain unconfirmed, ⚠️ is noted inline.
|
||||
|
||||
---
|
||||
|
||||
## Topic Index
|
||||
|
||||
| # | Topic | Core finding | Principles | Design impact |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Skills library design | SKILL.md (agentskills.io) is the cross-tool standard; description is the trigger; security risks from third-party skills are real and documented | 7 | Skill format, taxonomy, versioning, security |
|
||||
| 2 | Workflow and agent chain design | Orchestrator + isolated subagents is the dominant pattern; compact structured outputs; context pollution is the primary failure mode | 8 | Workflow patterns, state passing, HITL placement |
|
||||
| 3 | SDLC phase coverage and feedback loops | SDD rapidly emerging (not yet mainstream); AI-generated code has 45% OWASP Top 10 rate (Veracode) — 2.74× figure is CodeRabbit/XSS-specific; tests must derive from spec not implementation; LESSONS.md for repo-level feedback | 13 | Per-phase skills, feedback mechanisms, LESSONS.md |
|
||||
| 4 | IaC patterns | Terraform provisions, Ansible configures, keep separate in CI; idempotency is the critical Ansible property; Terraform provisioners are explicitly deprecated by HashiCorp | 8 | IaC skill structure, validation steps, Proxmox context files |
|
||||
| 5 | Gitea and self-hosted CI | Gitea Actions is mostly GitHub Actions compatible; no native AI integration; gitea-agent community project for autonomous workflows | 7 | CI pipeline skill, AI review workflow, security tooling |
|
||||
| 6 | Agent role design | gstack (80K+ stars) validates role separation; six roles cover the factory; roles are sequential cognitive modes not concurrent agents | 7 | Role skill files, HITL gates per role |
|
||||
| 7 | Prompt engineering for code | Context engineering supersedes prompt engineering; "3,000-token ceiling" framing refuted — degradation starts earlier; 500-line limit supported by SkillReducer less-is-more finding; constraints section is highest-ROI element | 8 | Skill body structure, context file design, self-check pattern |
|
||||
| 8 | Validation and testing of skills | Two test types required; trigger tests most commonly missed; promptfoo acquired by OpenAI; DeepEval/Arize Phoenix as alternatives | 8 | Eval file structure, promptfoo config, CI integration |
|
||||
| 9 | Token optimisation | Orientation is the primary sink; SkillReducer paper backs 500-line limit; prompt caching 90% savings; model routing 40–60% savings | 8 | Context architecture, caching strategy, model routing table |
|
||||
| 10 | MCP server integration | Official gitea-mcp (v1.2.0); no granular PAT token scopes; --tools is capability filter not security boundary; Claude Code per-role MCP filtering not currently supported; Ansible/Terraform execution stays human-controlled | 10 | MCP setup, tool scoping, homelab threat model |
|
||||
|
||||
---
|
||||
|
||||
## Topic 1: Skills Library Design
|
||||
|
||||
**The question:** What is the current standard for packaging reusable AI agent skills, how should a library be structured and taxonomised, and what security considerations apply?
|
||||
|
||||
### The SKILL.md open standard
|
||||
|
||||
Anthropic published the Agent Skills specification as an open standard on December 18, 2025. A skill is a directory containing one required file: SKILL.md with YAML frontmatter (name, description, optional metadata) and a Markdown body. Within 48 hours of publication, Microsoft integrated it into VS Code and OpenAI added it to Codex CLI and ChatGPT. By March 2026, 26+ tools from competing companies — including Google's Gemini CLI, JetBrains' Junie, AWS's Kiro, and Block's Goose — all read the same SKILL.md files from the same directory structure. ⚠️ Earlier research cited "32+" tools; deep research pass found consistent secondary sources citing 26 documented adopters as of May 2026 — updated here. Source: paperclipped.de/en/blog/agent-skills-open-standard-interoperability (March 23, 2026); medium.com/@automation.labs/skill-md-is-becoming-the-rest-of-agents (April 2026).
|
||||
|
||||
The official specification is published at agentskills.io/specification. The standard covers: file structure (a directory with SKILL.md), frontmatter format (YAML with required `name` and `description`, plus optional fields including `allowed-tools`, `license`, metadata block), instruction format (markdown), supporting directories (`scripts/`, `references/`, `assets/`), and the progressive disclosure model. Source: agentskills.io/specification; agensi.io/learn/agent-skills-open-standard.
|
||||
|
||||
**File structure:** SKILL.md at minimum. Optional subdirectories: `scripts/` (executable code), `references/` (documentation), `assets/` (templates), `examples/`. Source: medium.com/@loccarrre/the-agent-skills-standard (February 9, 2026).
|
||||
|
||||
**Progressive disclosure model:** Only name+description pairs load at startup (~30–50 tokens per skill); full SKILL.md loads when triggered; reference files load only when needed during execution. Source: strapi.io/blog/what-are-agent-skills-and-how-to-use-them; inference.sh/blog/skills/agent-skills-overview (April 14, 2026).
|
||||
|
||||
**The description field is the trigger.** "If your skill does not trigger, it is almost never the instructions. It is the description." Source: bibek-poudel.medium.com/the-skill-md-pattern-how-to-write-ai-agent-skills-that-actually-work (February 26, 2026).
|
||||
|
||||
**Canonical directory:** `.agents/skills/` per the open standard. Tool-specific locations (`.claude/skills/` for Claude Code, `~/.codex/skills/` for Codex) are thin adapters. Source: agensi.io/learn/agent-skills-open-standard; agentskills.io/specification.
|
||||
|
||||
**Size guidance:** Skills should be kept focused. The SkillReducer paper (arxiv.org/pdf/2603.29919) explicitly states "a skill with a large body and references can consume tens of thousands of tokens per invocation, and when multiple skills are active simultaneously, their cumulative cost can dominate the context budget." The 500-line limit is consistent with the practitioner finding that "focused 50-line files outperform sprawling 1,000-line ones."
|
||||
|
||||
### Security — documented and serious
|
||||
|
||||
The skills ecosystem had a significant security crisis in early 2026, providing the empirical basis for the "never install third-party skills without reading them" principle.
|
||||
|
||||
**ToxicSkills (Snyk, February 5, 2026):** The first comprehensive security audit of the AI agent skills ecosystem, scanning 3,984 skills from ClawHub and skills.sh. Found prompt injection in 36% of skills tested and 1,467 malicious payloads across the ecosystem. "There's a 13% chance it [a recently installed skill] contains a critical security flaw and a non-zero chance it's actively exfiltrating your credentials right now." Source: snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub (February 5, 2026); snyk.io/articles/skill-md-shell-access (February 3, 2026).
|
||||
|
||||
**ClawHavoc (February 2026):** A security audit of 2,857 skills on ClawHub uncovered 341 malicious skills across multiple campaigns. The primary campaign (ClawHavoc) delivered Atomic Stealer (AMOS), a commodity macOS infostealer. Skill names appeared legitimate (solana-wallet-tracker, youtube-summarize-pro). The attack exploited trust through documentation — malicious skills looked professional and matched what users sought. Antiy CERT's analysis reported 1,184 malicious skills across the broader coordinated campaign. Source: snyk.io/articles/skill-md-shell-access (February 3, 2026); repello.ai/blog/malicious-openclaw-skills-exposed-a-full-teardown (February 16, 2026). ⚠️ Discrepancy in count: Snyk says 341 malicious on ClawHub; Antiy CERT says 1,184 across the broader campaign — different scope.
|
||||
|
||||
**Mobb.ai audit:** 22,511 public skills across four registries; 140,963 total issues found. Source: agensi.io/learn/toxicskills-clawhavoc-agent-skills-security-crisis-2026.
|
||||
|
||||
**OWASP Agentic Skills Top 10 (AST10):** An OWASP Incubator project — not a ratified standard (v0.0.0 on the OWASP site; v1.0 release targeted Q3 2026; Flagship submission targeted Q4 2026). ⚠️ The "April 27, 2026" publication date appears only on third-party blogs; OWASP site shows March 2026 as last update date — treat with caution. Project lead Ken Huang. AST01 (Malicious Skills, Critical) cites ClawHavoc and ToxicSkills as the defining incidents. Full ten categories: AST01 Malicious Skills (Critical), AST02 Supply Chain Compromise (Critical), AST03 Over-Privileged (High), AST04 Insecure Metadata (High), AST05 Unsafe Deserialization (High), AST06 Weak Isolation (High), AST07 Update Drift (Medium), AST08 Poor Scanning (Medium), AST09 No Governance (Medium), AST10 Cross-Platform Reuse (Medium). Source: owasp.org/www-project-agentic-skills-top-10.
|
||||
|
||||
**Attack vector:** Skills use a "Markdown-as-installer" paradigm — SKILL.md provides documentation for the agent AND installation instructions for the human. Attackers embed obfuscated setup commands within documentation, relying on implicit trust in documentation files. Source: penligent.ai/hackinglabs/clawhub-malicious-skills-beyond-prompt-injection (February 9, 2026).
|
||||
|
||||
**For self-authored skills:** AST01 (Malicious Skills) does not apply — it requires attacker-controlled skill content, which cannot occur when the author controls all content. The relevant AST categories for a self-authored private factory are: AST03 (Over-Privileged Skills), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance). AST01/AST02 apply only when importing skills from public registries (ClawHub, skills.sh). Use `allowed-tools` to restrict invocable tools; use `disable-model-invocation: true` for production-touching skills.
|
||||
|
||||
### Taxonomy
|
||||
|
||||
Two-axis matrix: **phase** (design, implement, test, review, deploy, operate) × **domain** (python, typescript, ansible, terraform, docker, k8s, proxmox). No empirically validated taxonomy exists in the research — this structure is principled but not confirmed at scale.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
The SKILL.md standard is under a year old; tool-specific behaviour differences can break portability at edges (Claude Code adds context forking, Codex adds openai.yaml metadata). The two-axis taxonomy is principled but unproven at scale. The 500-line limit is practitioner-derived; the SkillReducer paper provides partial academic backing.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Security statistics from Snyk (vendor with commercial interest in finding security issues) and Mobb.ai (security tool vendor). ToxicSkills scanned only ClawHub and skills.sh — may not be representative of self-hosted or curated skill registries. The 36% figure applies to third-party public skills; self-authored skills have a fundamentally different risk profile.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Use SKILL.md format as the universal packaging unit — the official spec is at agentskills.io.
|
||||
2. Write the description field first — it is the trigger.
|
||||
3. Keep each SKILL.md focused; move reference material to `references/` (backed by SkillReducer paper, arxiv.org/pdf/2603.29919).
|
||||
4. Organise the library on a phase × domain matrix.
|
||||
5. Version every skill in frontmatter; track changes in git.
|
||||
6. Apply `allowed-tools` restrictions, especially for skills touching production.
|
||||
7. Never install third-party skills without reading the full SKILL.md and any bundled scripts — the OWASP Agentic Skills Top 10 (Incubator draft) formalises this as AST01 (Malicious Skills). For self-authored skills, the relevant risks are AST03 (Over-Privileged), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance).
|
||||
|
||||
---
|
||||
|
||||
## Topic 2: Workflow and Agent Chain Design
|
||||
|
||||
**The question:** How should skills be composed into multi-step workflows, what patterns govern state passing, and what are the documented failure modes of chaining?
|
||||
|
||||
### Orchestration patterns
|
||||
|
||||
Five workflow patterns are documented across the literature: prompt chaining (sequential LLM calls), parallelization (simultaneous operations), routing (classify-then-direct), orchestrator-worker (dynamic subtask delegation), and evaluator-optimizer (generate-evaluate-refine loops). Source: tao-hpu.medium.com/ai-agent-landscape-2025-2026-a-technical-deep-dive (January 5, 2026, citing sources [19][20]); fungies.io/ai-agent-orchestration-developers-guide-2026.
|
||||
|
||||
**Orchestrator-worker is now the dominant production pattern.** Anthropic's research multi-agent system, Claude Code's Task tool, OpenAI's agents-as-tools, and Cognition's March 2026 Managed Devins all use the orchestrator + isolated subagents pattern. Source: flowhunt.io/blog/multi-agent-ai-system (May 2026).
|
||||
|
||||
"Multi-agent system inquiries surged 1,445% in 2025." Source: fungies.io/ai-agent-orchestration-developers-guide-2026.
|
||||
|
||||
**Subagent isolation:** Each subagent runs in its own fresh context window with a dedicated system prompt, executes its task, and returns a single summary string. No peer-to-peer channel; no shared mutable state. This bounds coordination cost and avoids the quadratic communication explosion of peer architectures. Source: flowhunt.io/blog/multi-agent-ai-system.
|
||||
|
||||
**Academic backing:** AORCHESTRA (arXiv 2602.03786, February 2026) models every subagent as a 4-tuple (INSTRUCTION, CONTEXT, TOOLS, MODEL) and reports +16.28% **relative** improvement on GAIA, SWE-Bench, and Terminal-Bench. ⚠️ This figure is Gemini-3-Flash specific — the paper attributes the relative gain to that model pairing; transfer to Claude Code workflows has not been independently validated. The architectural pattern (orchestrator + isolated subagents) is well-supported across independent sources regardless. Source: flowhunt.io/blog/multi-agent-ai-system citing arxiv.org/abs/2602.03786.
|
||||
|
||||
### State passing
|
||||
|
||||
Define the WorkflowContext schema before writing any skill in a chain. Each skill returns compact structured outputs — only the fields the next step needs. Build failure states into every skill (return error fields; never throw). LangGraph uses a graph structure with nodes and edges; CrewAI uses role-based handoffs. Source: fungies.io/ai-agent-orchestration-developers-guide-2026; eesel.ai/blog/subagent-orchestration.
|
||||
|
||||
**SkillsBench — confirmed, but precision correction required:** SkillsBench (Li et al., arXiv 2602.12670, February 2026) is a real benchmark: 86 tasks across 11 domains, 7 agent-model configurations, 7,308 trajectories, deterministic verifiers. The finding cited as "2–3 skills optimal" was a unit-of-analysis error — **"2–3 modules per skill" is the finding, not "2–3 skills per task."** Exact finding: focused skills with 2–3 modules outperform comprehensive documentation. Two additional findings the research document should carry: (a) **self-generated skills yielded no benefit** (−1.3pp average); (b) **19% of tasks regressed** under curated skills, indicating skills are not unconditionally helpful. A follow-up paper (arXiv 2604.04323) notes SkillsBench skills appear "hand-crafted to overfit each task," with performance degrading under realistic retrieval conditions. The directional finding — more complexity per skill increases failure surface — remains valid. Source: arxiv.org/abs/2602.12670; skillsbench.ai/blogs/introducing-skillsbench.
|
||||
|
||||
### Session handoff pattern
|
||||
|
||||
Context is ephemeral; files are durable. "Vibe coding's one-long-session workflow is structurally designed to maximize context rot. All three SDD frameworks address this with the same architectural insight: external state must live in files or version control, not in the LLM's context window." Source: blog.appxlab.io/2026/03/27/spec-driven-development-ai-coding.
|
||||
|
||||
### Spec-driven development as workflow anchor
|
||||
|
||||
The spec is the contract between all workflow stages. "The coding agent knows what to build because the specification defines the target, not an ephemeral chat prompt." Source: javacodegeeks.com/2026/05/spec-driven-development (May 2026); thebcms.com/blog/spec-driven-development.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
Orchestrator-worker assumes reliable task decomposition — current models drift on novel or ambiguous tasks. The SkillsBench 2–3 skills optimal figure is unverified. WorkflowContext schemas add upfront design cost that may not pay off at small scale.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Use orchestrator-worker for complex multi-phase workflows; sequential pipeline for deterministic steps.
|
||||
2. Define the WorkflowContext schema before writing any skill in a chain.
|
||||
3. Every skill returns compact structured output — minimum needed by the next step.
|
||||
4. Every skill handles failure explicitly — return error fields, never silent failures.
|
||||
5. Apply subagent isolation for heavy subtasks; return only essential results to parent.
|
||||
6. Place HITL gates before irreversible actions, not at end of chain.
|
||||
7. Use session-handoff documents for workflows spanning sessions.
|
||||
8. Anchor every workflow to a spec — it is the contract between stages.
|
||||
|
||||
---
|
||||
|
||||
## Topic 3: SDLC Phase Coverage and Feedback Loops
|
||||
|
||||
**The question:** What skills and workflows are needed at each SDLC phase, what does AI do well and poorly, and how do review findings feed back reliably?
|
||||
|
||||
### Spec-first: now mainstream
|
||||
|
||||
Spec-Driven Development (SDD) is "the practice of writing a structured, versioned specification before invoking an AI coding agent, so the agent has explicit goals, constraints, and acceptance criteria. It replaces ad-hoc prompting ('vibe coding') with a disciplined Spec → Plan → Tasks → Implement loop." Source: thebcms.com/blog/spec-driven-development (Definitive 2026 Guide, May 2026).
|
||||
|
||||
"Vibe coding" was coined by Andrej Karpathy in February 2025. "Even Andrej Karpathy, who coined the term 'vibe coding' in February 2025, admitted just a year later that this era is ending and that we are entering the age of agentic engineering." Source: towardsdatascience.com/from-vibe-coding-to-spec-driven-development (May 2026).
|
||||
|
||||
SDD is rapidly emerging, but "now mainstream" overstates current production adoption. GitHub Spec Kit (open source, MIT-licensed, ~92,000 stars as of May 2026, supporting 28 named AI agent platforms). AWS Kiro (commercial, attracted 250,000 developers during its preview period — approximately four months from July 2025; AWS has not disclosed active-user or production-deployment numbers, making this likely a signup count). DeepLearning.AI launched a dedicated "Spec-Driven Development with Coding Agents" course in late 2025. No controlled study comparing SDD with ad-hoc prompting outcomes exists as of May 2026. Source: dev.to/remybuilds/spec-driven-development-structure-beats-vibes (May 2026); blog.appxlab.io (March 2026); thebcms.com.
|
||||
|
||||
Counterargument documented: Thoughtworks Radar rated SDD "Assess, not Adopt" in November 2025 (Vol 33). Full rationale: *"We may be relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn't scale."* Vol 34 (April 2026) did not re-rate SDD but added GitHub Spec Kit separately and moved "Curated shared instructions for software teams" to the Adopt level. Marmelab documented a 1,300-line spec for a one-feature date display — spec overhead is real. "SDD works best for large teams (10+ developers), complex systems with integration points, and regulated industries. It's overkill for solo developers, simple CRUD applications, and weekend projects." GitHub's own documentation states Spec Kit is "best suited for greenfield builds and large feature work rather than small bug fixes." Source: dev.to/remybuilds; byteiota.com/spec-driven-development-kills-vibe-coding-march-2026 (March 2026); thoughtworks.com/radar.
|
||||
|
||||
### AI-generated code security — corrected attribution
|
||||
|
||||
**Veracode 2025 GenAI Code Security Report:** Tested 100+ LLMs across 4 languages (Java, JavaScript, Python, C#), 80 coding tasks. **45% of code samples introduced OWASP Top 10 vulnerabilities; Java had a 72% security failure rate.** Source: businesswire.com/news/home/20250730694951 (July 30, 2025); growexx.com/blog/ai-code-security-crisis-2026 (February 2026).
|
||||
|
||||
**2.74× more vulnerabilities — corrected attribution:** This figure does NOT come from Veracode. It comes from **CodeRabbit's "State of AI vs Human Code Generation" report (December 2025)** analysing 470 GitHub PRs — and the 2.74× figure applies **specifically to XSS vulnerabilities, not all vulnerability types**. CodeRabbit's overall finding is more modest: AI code has 1.7× more issues and 1.57× more security findings overall. ⚠️ Earlier research merged these two sources incorrectly. CodeRabbit is a code-review tool vendor with commercial interest in this finding. Source: vibegraveyard.ai/story/coderabbit-ai-code-quality-study (theregister.com/2025/12/17).
|
||||
|
||||
**Independent academic corroboration:**
|
||||
- Yan et al. (arXiv 2506.23034, GMU): 9.8–42.1% of AI-generated code is vulnerable, depending on benchmark and model.
|
||||
- Schreiber & Tippe (arXiv 2510.26103, October 2025): large-scale GitHub analysis of 7,703 AI-generated files found **only 12.1% contained CWE-mapped vulnerabilities** — substantially below controlled-prompt benchmark rates. Real-world figures may be much lower than benchmark-driven figures when developers apply typical hygiene.
|
||||
- Backslash Security (April 2026): All 7 tested LLMs generated code vulnerable to ≥4 of 10 common CWEs with naive prompts; Claude 3.7 Sonnet performed best (6/10 with naive prompts, 10/10 with security-focused prompts). ⚠️ Vendor source (Backslash is a security tool); finding on security-focused prompts is practically important. Source: backslash.security/press-releases/backslash-security-reveals (April 2026); betanews.com/2025/04/24.
|
||||
|
||||
⚠️ The "60%+ vulnerability reduction after human review" claim in earlier research has no traceable primary source — removed. The directional finding (review reduces vulnerability rates substantially) is reasonable but unquantified.
|
||||
|
||||
Surviving AI-introduced issues in production repositories topped 110,000 by February 2026 (large-scale empirical study, arXiv 2026). Source: augmentcode.com/guides/what-is-spec-driven-development; codeqa.aivyuh.com.
|
||||
|
||||
AI agents inside real business workflows fail roughly one attempt in three. Source: mydataschool.com/blog/ai-spec-driven-development (citing Stanford HAI 2026 AI Index).
|
||||
|
||||
### Phase-by-phase findings
|
||||
|
||||
**Design:** Without explicit scope, agents make assumptions and head in the wrong direction fast. "The spec is the prompt." Source: thebcms.com; javacodegeeks.com (May 2026).
|
||||
|
||||
**Testing — critical gap:** AI-generated tests test the implementation rather than the intent (circular reasoning). Tests must derive from the spec to validate intent. "SDD says 'write the whole spec first — behavior, architecture, edge cases, constraints — and let an agent produce the code, tests, and docs from it.'" Source: thebcms.com/blog/spec-driven-development; augmentcode.com/guides/what-is-spec-driven-development. ⚠️ The 67% of testers would trust AI-generated tests with mandatory human review figure was cited in the original session but not found in re-search — remove from the document; unconfirmed.
|
||||
|
||||
**Code review:** ⚠️ The Atlassian RovoDev 38.7% figure (AI review comments leading to fixes) was not confirmed in re-search. Removing the specific figure; retaining the directional finding: AI review is good at breadth, poor at contextual judgment. Source: not independently confirmed in this session — treat as directional only.
|
||||
|
||||
### Feedback loops
|
||||
|
||||
**Two loops required:** short (finding → immediate fix, happens by default) and long (recurring pattern → convention, must be explicitly designed). Source: principle derived from practitioner consensus; no single primary citation.
|
||||
|
||||
**Auto memory scope — verified correction:** Claude Code's auto-memory is stored at `~/.claude/projects/<hash>/memory/` — per-machine, not repo-shared. For repo-level lessons, `LESSONS.md` committed to the repo root is the correct mechanism. Source: Claude Code memory documentation — corrected during session when user identified the error.
|
||||
|
||||
**Tiered review severity:** Block critical, warn high, suggest low. Alert fatigue from over-blocking breaks the loop. Source: general practitioner guidance across CI/review tooling.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
SDD overhead is documented — Thoughtworks Radar "Assess, not Adopt" (November 2025). For solo developers and small projects, SDD over-engineers. "Solo developers, small teams (under 5 people), and projects with rapidly changing requirements gain little from upfront specification overhead." Source: byteiota.com (March 2026). Veracode figures are from a controlled study — real-world rates vary significantly by prompt quality and review process.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Veracode is a security tool vendor with commercial interest in finding vulnerabilities. GitHub Spec Kit statistics come partly from GitHub itself. SDD adoption figures (90K stars, 250K developers for Kiro) are from the tool makers.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Every implementation workflow begins with a spec skill — SDD is now the mainstream approach, not experimental.
|
||||
2. Generate tests from the spec, not from the implementation — prevents circular reasoning.
|
||||
3. AI review supplements human review; never replaces the HITL gate before merge.
|
||||
4. Design both feedback loops explicitly — short (finding → fix) and long (pattern → convention).
|
||||
5. Repo-level lessons learned live in committed `LESSONS.md`, not in auto memory.
|
||||
6. Tier review findings by severity — alert fatigue breaks the loop.
|
||||
7. Convention graduation: 3+ LESSONS.md entries on same pattern → promote to CONTEXT.md.
|
||||
8. Session handoff skill explicitly prompts LESSONS.md extraction before closing.
|
||||
|
||||
---
|
||||
|
||||
## Topic 4: Infrastructure-as-Code Patterns
|
||||
|
||||
**The question:** How should AI-assisted IaC work across Ansible, Terraform, Docker, Kubernetes, and Proxmox?
|
||||
|
||||
### Division of concerns — verified
|
||||
|
||||
"Terraform handles 'Day 0' activities — initial provisioning and lifecycle management of infrastructure components. Ansible handles 'Day 1 and beyond' tasks — configuration of provisioned resources." Source: scalr.com/learning-center/ultimate-guide-to-using-terraform-with-ansible (reviewed May 1, 2026).
|
||||
|
||||
"Use Terraform to provision infrastructure first, then hand off to Ansible for OS and application configuration." Source: spacelift.io/blog/using-terraform-and-ansible-together; computingforgeeks.com/terraform-ansible-tutorial (tested April 2026 with Terraform 1.14.8, ansible-core 2.16.14, bpg/proxmox provider 0.78 on real Proxmox hardware).
|
||||
|
||||
Both tools are idempotent by design. "Running them again should produce no changes if the desired state already matches reality." Source: computingforgeeks.com/terraform-ansible-tutorial.
|
||||
|
||||
### Tool-specific failure modes — verified
|
||||
|
||||
**Terraform provisioners — explicitly deprecated by HashiCorp:** "Terraform provisioners are one of the most polarizing features in the Infrastructure as Code toolkit. HashiCorp — the creators of Terraform — explicitly recommends using them only as a 'last resort.'" Key problems: provisioner scripts are not idempotent by default; they create drift invisible to Terraform state; they introduce command injection risks. Source: scalr.com/learning-center/understanding-terraform-provisioners (reviewed June 1, 2025).
|
||||
|
||||
"Avoid calling Ansible via Terraform provisioners except for rare bootstrap cases." Source: spacelift.io/blog/using-terraform-and-ansible-together.
|
||||
|
||||
**Ansible idempotency:** "While Ansible modules aim for idempotency, the tool doesn't rely on stored state like Terraform does." Non-idempotent patterns are documented: Terraform `local-exec` provisioners appending to files on every run; Ansible `shell`/`command` tasks running unconditionally. Best practice: prefer `ansible.builtin.apt`, `copy`, `template` over `shell`/`command`. Source: dev.to/bowale/infrastructure-as-code-iac-challenges (January 2025); scalr.com/learning-center/understanding-terraform-provisioners.
|
||||
|
||||
AI-generated Ansible: AI correctly uses FQCNs and handlers, but misses idempotency guards (e.g., missing `creates:` on certbot tasks), validation tasks before reloads, and rollback strategies. Source: computingforgeeks.com/ai-coding-agents-devops-terraform-ansible-kubernetes (tested April 2026 with OpenCode on Rocky Linux 9.5).
|
||||
|
||||
**AI-generated K8s:** 41% of AI-generated backend code includes overly broad permission settings. Source: sqmagazine.co.uk/ai-coding-security-vulnerability-statistics (April 2026). K8s manifests require resource limits, security contexts, and network policies — these are routinely omitted in AI-generated output.
|
||||
|
||||
**Proxmox — confirmed field experience:** The bpg/proxmox Terraform provider (version 0.78 as of April 2026) is community-maintained and actively developed. The computingforgeeks.com guide (tested April 2026 on real Proxmox hardware) confirms the provider works but requires precise version pinning and a dedicated API token (not root credentials). "The full token ID format is terraform@pve!terraform-token." Source: computingforgeeks.com/terraform-ansible-tutorial.
|
||||
|
||||
### Validation — standard practice
|
||||
|
||||
`terraform plan`, `ansible --check --diff`, `kubectl diff` are the deterministic validation mechanisms. These are standard IaC practice, not novel governance requirements. Source: computingforgeeks.com; spacelift.io; scalr.com.
|
||||
|
||||
### Skills needed for IaC
|
||||
|
||||
`write-ansible-role`, `write-terraform-module`, `write-k8s-manifest`, `write-docker-compose`, `proxmox-vm-spec` (loads `references/proxmox-cluster.md`), `iac-security-review`, `write-molecule-test`. Each skill carries the relevant tool/provider version in frontmatter metadata.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
The Ansible/Terraform separation is the recommended pattern but not universal — Ansible's `community.proxmox` collection can provision Proxmox VMs directly, removing the need for Terraform at homelab scale. AI-generated Ansible is not uniformly bad at idempotency — the computingforgeeks.com test found AI (OpenCode) correctly using FQCNs and handlers; the failure mode is missing edge case guards, not fundamental incorrectness.
|
||||
|
||||
### Bias flags
|
||||
|
||||
computingforgeeks.com is a practitioner site with real hardware testing — high credibility for the Proxmox provider version information, but not a controlled study. scalr.com and spacelift.io are IaC platform vendors with commercial interest in presenting Terraform + Ansible separation as best practice.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Skills are tool-specific — `write-ansible-role` and `write-terraform-module` have different validation steps.
|
||||
2. Every IaC skill carries explicit tool and provider version in frontmatter metadata.
|
||||
3. Idempotency is the required property for Ansible — the skill review step must explicitly check for `shell`/`command` without guards and missing `creates:` parameters.
|
||||
4. Never call Ansible from Terraform provisioners — HashiCorp itself calls this a last resort; keep them as separate CI/CD stages.
|
||||
5. Validation is deterministic: plan/check/diff are mandatory before any apply.
|
||||
6. Proxmox skills carry cluster context in `references/proxmox-cluster.md`; pin bpg/proxmox to a specific version.
|
||||
7. K8s manifests require resource limits, security contexts, and network policies — review skill checks these explicitly.
|
||||
8. Docker images must use pinned versions, not `:latest`.
|
||||
|
||||
---
|
||||
|
||||
## Topic 5: Gitea and Self-Hosted CI
|
||||
|
||||
**The question:** What does Gitea Actions support and not support, and how should the factory handle CI/CD?
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
### Gitea Actions capabilities
|
||||
|
||||
Gitea Actions production-stable since Gitea 1.19. GitHub Actions-compatible YAML syntax. `act_runner` forked from nektos/act. `DEFAULT_ACTIONS_URL = github` allows GitHub marketplace actions. Runs in Docker containers. Source: docs.gitea.com/usage/actions/overview; docs.gitea.com/usage/actions/design; about.gitea.com/products/runner.
|
||||
|
||||
Practical setup: act_runner runs as a Docker sibling container alongside Gitea, polling for jobs. Source: botmonster.com/posts/self-hosted-cicd-pipeline-gitea-actions-docker (May 2026); dev.to/diogojlq/self-hosted-gitea-ci (January 2026).
|
||||
|
||||
### Compatibility gaps
|
||||
|
||||
Concurrency groups ignored. `GITHUB_TOKEN` maps to owner-level Gitea API token, not fine-grained scopes. Cache requires explicit setup (Gitea 1.21+). `runs-on` labels must match exactly. "Compatibility with GitHub Actions is impressive for a project this size, but there are sharp edges." Source: docs.gitea.com/usage/actions/comparison; blog.elest.io/how-to-migrate-from-github-actions-to-gitea-actions (March 2026); mylinux.work/guides/gitea-actions-and-runners (April 2026).
|
||||
|
||||
### What Gitea lacks — and community responses
|
||||
|
||||
No native equivalent of GitHub Copilot review, GitHub Advanced Security, or GitHub MCP server. AI models default to GitHub Actions assumptions. Source: docs.gitea.com/usage/actions/comparison; github.com/go-gitea/gitea/issues/34527 (AI agent feature request, May 2025).
|
||||
|
||||
**Gitea vs Forgejo:** In late 2024, Forgejo forked from Gitea over governance concerns. Both share the same workflow syntax; Forgejo uses `forgejo-runner`. Source: serverspan.com/en/blog/the-2026-guide-to-self-hosted-git-gitea-forgejo (February 2026).
|
||||
|
||||
**gitea-agent (April 2026):** A community project specifically for Gitea self-hosted environments, combating "LLM drift — where large language models skip tests, modify incorrect files, or hallucinate paths." Issue → Plan → Approval → Implementation → Eval → PR cycle with human approval gates. Runs fully locally. Source: devactivity.com/posts/apps-tools/gitea-agent (April 7, 2026). ⚠️ Early-stage community project; not production-ready at time of research.
|
||||
|
||||
### Replacing GitHub-native security tooling
|
||||
|
||||
- Secret scanning: `trufflehog` (ghcr.io/trufflesecurity/trufflehog) or `gitleaks`
|
||||
- Dependency scanning: `trivy` (aquasec/trivy)
|
||||
- Licence scanning: `licensee` or `fossa-cli`
|
||||
|
||||
Source: botmonster.com/posts/self-hosted-cicd-pipeline-gitea-actions-docker; mylinux.work/guides/gitea-actions-and-runners.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
"Mostly compatible" overstates for complex pipelines. Gitea token model may vary by version. gitea-agent is community and early-stage.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Gitea compatibility claims from Gitea's own documentation. gitea-agent assessment from community discussion threads.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Port workflows to `.gitea/workflows/`; fix Gitea-specific edge cases.
|
||||
2. Configure the cache server before writing pipelines.
|
||||
3. The write-ci-pipeline skill explicitly targets Gitea; never assumes GitHub defaults.
|
||||
4. Secret, dependency, and licence scanning are default pipeline steps.
|
||||
5. AI PR review is a custom workflow step — gitea-agent is worth watching as it matures.
|
||||
6. Agent tokens scoped carefully in workflows posting comments or modifying PRs.
|
||||
7. All CI runners self-hosted; no code routed through external services.
|
||||
|
||||
---
|
||||
|
||||
## Topic 6: Agent Role Design
|
||||
|
||||
**The question:** How should AI agent roles be structured, what roles are needed for the factory?
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
### The core insight
|
||||
|
||||
Different phases require different cognitive modes. Planning is not review. Review is not shipping. Source: github.com/garrytan/gstack; backslash.security/blog/the-new-role-of-developers-ai-sdlc (May 2026).
|
||||
|
||||
**Industry-wide confirmation:** "Skill shift: The role of the developer is shifting from 'Code Writer' to 'Agent Orchestrator' and 'System Reviewer.'" Source: hcltech.com/trends-and-insights/autonomous-software-factory-agentic-ai-sdlc (February 16, 2026). "More than 75% of developers will be architecting, governing, and orchestrating instead of building applications." Source: devopsdigest.com/2026-devops-predictions-5 (December 2025). IBM Bob (April 28, 2026): "an agentic platform that embeds an AI partner into every role across the SDLC." Source: newsroom.ibm.com/2026-04-28-introducing-ibm-bob.
|
||||
|
||||
### Reference implementation: gstack
|
||||
|
||||
Garry Tan's open-source skill pack (MIT, launched March 12, 2026): 23–31 role-based slash commands. Skills run in sprint order: Think → Plan → Build → Review → Test → Ship → Reflect. Source: github.com/garrytan/gstack; augmentcode.com/learn/garry-tan-gstack-claude-code (April 2026); sitepoint.com/gstack-garry-tan-claude-code (March 2026); mager.co/blog/2026-03-28-gstack; awesomeagents.ai/guides/gstack (April 2026).
|
||||
|
||||
**Star count (rapidly changing):** 39K within 11 days of launch; 56K by late March; **~96K by May 2026**. High adoption rate for an opinionated tool indicates genuine pain point being addressed. Source: producthunt.com/products/gstack; awesomeagents.ai/guides/gstack; dev.to/max_quimby/gstack.
|
||||
|
||||
Key finding: separating plan review from code review prevents the failure mode where the agent either rubber-stamps or gets lost in details. Source: producthunt.com/products/gstack discussion thread (March 2026).
|
||||
|
||||
**Productivity claim — actual detail and caveats:** Garry Tan claims **810× his 2013 part-time output**, measured in self-defined "logical LOC" (non-blank, non-comment lines across all garrytan/* repos via custom script). The 2013 baseline was **part-time output while running YC and Posterous** — well below historical developer norms (Capers Jones: 16–38; McConnell: 20–125). Tan acknowledges: *"if you think the right baseline is 50, the multiple drops from 810× to 228×."* The comparison confounds three variables: better 2026 LLMs, **10–15 parallel Claude Code sessions via Conductor.build**, and gstack's role structure — these cannot be separated. No independent team has published comparable results. TechCrunch (March 17, 2026) provides the primary skeptical coverage, noting gstack's popularity reflects partly Tan's YC platform. The gstack repo itself includes an `ON_THE_LOC_CONTROVERSY.md` file acknowledging the measurement disputes. Source: github.com/garrytan/gstack/blob/main/docs/ON_THE_LOC_CONTROVERSY.md; techcrunch.com/2026/03/17.
|
||||
|
||||
"Simpler, well-bounded agent patterns usually outperform elaborate multi-agent complexity when the goal is reliable software delivery." Source: cio.com/article/4166035 (attributed to Anthropic's experience, May 2026).
|
||||
|
||||
"The real ceiling isn't the harness — it's the 1–5% error rate from frontier models. Cascading failures still need human taste to catch before they compound." Source: producthunt.com/products/gstack discussion thread.
|
||||
|
||||
### Six factory roles
|
||||
|
||||
| Role | Cognitive mode | Primary output | HITL gate |
|
||||
|---|---|---|---|
|
||||
| Architect | Systems thinking, adversarial design review | Design doc, API contracts, data models | Human approves before build |
|
||||
| Developer | Implementation against plan | Code, tests, docs | Human reviews diff before commit |
|
||||
| Reviewer | Staff-engineer perspective | Severity-rated findings | Human decides which to act on |
|
||||
| Security | Adversarial audit (OWASP/STRIDE) | Security findings | Human approves before merge |
|
||||
| QA | Adversarial testing | Test results, regression tests | Human reviews before PR opened |
|
||||
| Ops | Infrastructure and deployment | Pipelines, manifests, runbooks | Human approves before any apply |
|
||||
|
||||
### Counterarguments
|
||||
|
||||
gstack productivity claims (810× self-reported output) use a self-defined "logical LOC" metric, a part-time 2013 baseline, and confound parallelism (10–15 sessions) with role design — not independently verifiable. Six roles may over-engineer for a single-developer homelab early on. The role-based design is not isolated as the causal mechanism; comparable frameworks (ECC, GSD, OpenSpec) achieve similar coordination without gstack specifically.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Primary evidence from gstack — created by YC CEO with platform interest in demonstrating agentic productivity. Star count reflects novelty and author's platform, not adoption at scale. No independent team has published comparable productivity results.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Roles enforce cognitive separation.
|
||||
2. Six roles cover the factory: Architect, Developer, Reviewer, Security, QA, Ops.
|
||||
3. Roles run sequentially on demand — not concurrently.
|
||||
4. Each role's output is the next role's structured input.
|
||||
5. Ops is a distinct role from Developer.
|
||||
6. HITL gates apply per role.
|
||||
7. Roles reduce error surface but don't eliminate it.
|
||||
|
||||
---
|
||||
|
||||
## Topic 7: Prompt Engineering for Code
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
**The question:** What prompting patterns are effective for code generation and review, and what anti-patterns must skills avoid?
|
||||
|
||||
### The shift to context engineering
|
||||
|
||||
Context engineering has largely superseded prompt engineering in 2026. Four strategies (LangChain): write (persist externally), select (RAG), compress, isolate. Source: thomas-wiegold.com/blog/prompt-engineering-best-practices-2026 (February 2026); arxiv.org/pdf/2603.09619 (Context Engineering: From Prompts to Corporate Multi-Agent Architecture, 2026).
|
||||
|
||||
### Context length and reasoning quality — corrected 2026 evidence
|
||||
|
||||
⚠️ **The "3,000-token degradation ceiling" framing is inaccurate.** Levy, Jacoby, and Goldberg (2024) found degradation beginning as early as **~500 tokens** in the models they tested; 3,000 tokens was simply the maximum length tested, not a ceiling. Citing this as "reasoning degrades around 3,000 tokens" inverts the finding.
|
||||
|
||||
**2026 evidence updates this substantially:**
|
||||
- **Anthropic MRCR v2 (multi-needle retrieval at 1M tokens):** Opus 4.6 = 76% vs. Sonnet 4.5 = 18.5% — a 4× improvement attributed to context compaction. Targeted retrieval at very long context is now feasible for flagship models.
|
||||
- **OOLONG benchmark (arXiv 2511.02817, November 2025):** All frontier models (including GPT-5, Claude Sonnet 4, Gemini 2.5 Pro) score **<50% at 128K context on aggregation reasoning** — directly contradicting the assumption that large context windows solve comprehension problems.
|
||||
- **"Intelligence Degradation in Long-Context LLMs" (arXiv 2601.15300):** F1 collapses ~45% at 50% of maximum context for tested models — degradation is catastrophic past a threshold, not gradual.
|
||||
|
||||
**Design implication:** The 500-line skill limit and ≤200-line CONTEXT.md constraint remain well-founded for a different reason: not because reasoning degrades at exactly 3,000 tokens, but because (a) aggregation reasoning is poor even at 128K for all current models, and (b) the SkillReducer paper (arXiv 2603.29919) confirms that large skill bodies provide worse quality output than compressed equivalents, independent of absolute token counts. Drop the "3,000-token ceiling" framing; keep the size limits with this corrected rationale. Source: Levy et al. (2024) via thomas-wiegold.com; arxiv.org/abs/2511.02817; arxiv.org/abs/2601.15300; Anthropic MRCR v2 eval.
|
||||
|
||||
### XML tags: when they help
|
||||
|
||||
Benefit scales with complexity. On prompts under 500 tokens, XML adds tokens without improving accuracy. Source: controlled experiment at dev.to/manishramavat (May 2026, N=12 — very small sample). Claude natively parses XML-style tags. Source: platform.claude.com/docs/en/build-with-claude/prompt-engineering/use-xml-tags; docs.claude.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags.
|
||||
|
||||
### The constraints section as highest-ROI element
|
||||
|
||||
Claude Opus 4.5 and 4.6 tend to overengineer — extra files, unnecessary abstractions. Explicit constraints prevent this. Source: code.claude.com/docs/en/best-practices.
|
||||
|
||||
### Context file design
|
||||
|
||||
"Decisions not descriptions" → ~20% token reduction. AST-level file loading → ~65% fewer file reads. Source: earezki.com/ai-news/2026-02-26 (February 2026 — single practitioner case study). ⚠️ **Commercial bias: Nicola Alessi is the developer of vexp** (vexp.dev, the AST-loading tool used in the case study). The case study should not be treated as an independent third-party result. The general principle (precise context over broad loading) is sound; the specific 65% figure reflects vexp's capabilities in that specific codebase and should be treated as an indicative upper bound, not a guaranteed outcome. The "40 file reads" baseline was also unusually broad — more disciplined CONTEXT.md instructions could close most of the gap without AST tooling. Technology choices with rationale, ADRs, codebase-specific anti-patterns, Preferred/Avoid code blocks. Source: packmind.com/context-engineering-ai-coding (April 2026).
|
||||
|
||||
### Chain-of-thought for review skills
|
||||
|
||||
`<thinking>` + named output tags: reasoning to audit log, structured output feeds next step. Source: kindatechnical.com/claude-ai/claudes-prompt-style-xml-tags; platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
The "3,000-token ceiling" framing has been corrected — the 2024 study showed degradation beginning at ~500 tokens, not a ceiling at 3,000. 2026 models show substantially better long-context handling for retrieval tasks (MRCR v2) but aggregation reasoning remains poor at 128K+. XML experiment was N=12 — very small. Context engineering framing can be overstated.
|
||||
|
||||
### Bias flags
|
||||
|
||||
XML experiment small sample (N=12). "20% reduction" and "65% reduction" are single practitioner case studies. Claude prompting documentation is from Anthropic.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Context engineering is the primary discipline — CONTEXT.md and reference files are the primary artefact.
|
||||
2. Keep loaded context per skill invocation lean — the 500-line skill limit is justified by SkillReducer's "less-is-more effect" (+2.8% quality from compressed skills) and OOLONG's finding of <50% aggregation reasoning at 128K even for frontier models. The previous "3,000-token ceiling" framing (Levy et al. 2024) was a misreading — see corrected evidence above.
|
||||
3. XML for complex skills (500+ tokens, 3+ sections); plain prose for simple skills.
|
||||
4. Every implementation skill includes an explicit constraints section.
|
||||
5. Use `<thinking>` + named output tags for review and security skills.
|
||||
6. Reference files by path rather than loading full content.
|
||||
7. End every implementation skill with a self-check instruction.
|
||||
8. Document Preferred/Avoid code blocks in CONTEXT.md for every convention that matters.
|
||||
|
||||
---
|
||||
|
||||
## Topic 8: Validation and Testing of Skills
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
**The question:** How do you verify a skill works correctly, what testing frameworks exist, and how does skill testing integrate into CI?
|
||||
|
||||
### Why skill testing differs from unit testing
|
||||
|
||||
Standard LLM evals test a function; agent evals test a system with emergent behaviour. Non-determinism compounds. Source: promptfoo.dev/docs/guides/evaluate-coding-agents; developers.openai.com/blog/eval-skills (January 2026).
|
||||
|
||||
### The trigger eval pattern
|
||||
|
||||
10–20 test cases: explicit invocation, implicit invocation, and negative cases. Start small; grow from real failures. Source: developers.openai.com/blog/eval-skills (January 2026); promptfoo.dev/docs/integrations/agent-skill. ⚠️ OpenAI's guidance is for Codex; the eval structure is applicable in principle across Claude Code, Gemini CLI, and VS Code Copilot — all use description-matching activation — but **no cross-tool empirical comparison of false-positive/false-negative rates has been published as of May 2026**. Validate eval suites per tool; do not assume portability.
|
||||
|
||||
### Output eval assertions
|
||||
|
||||
Deterministic assertions (JSON schema, contains, regex) as blocking gates; LLM-rubric as secondary. Source: promptfoo.dev/docs/intro; evaluation tooling guides from confident-ai.com/knowledge-base/compare/best-ai-evaluation-tools-2026; augmentcode.com/tools/best-ai-agent-evaluation-tools (May 2026).
|
||||
|
||||
### Tooling
|
||||
|
||||
**Promptfoo:** Open-source MIT, YAML-based, self-hosted, CLI-first. Acquired by OpenAI March 9, 2026 for $86M. MIT license confirmed maintained post-acquisition; multi-provider support (Anthropic, Google, Azure, Ollama) confirmed as of May 2026. OpenAI committed publicly: *"Promptfoo will remain open source under the current license, and we will continue to service and support current customers."* Structural conflict of interest exists for non-OpenAI evaluation — monitor for neutrality degradation; **version-pin to a known-good release** rather than tracking `latest`. Source: github.com/promptfoo/promptfoo; openai.com/index/openai-to-acquire-promptfoo; techcrunch.com/2026/03/09.
|
||||
|
||||
**CI integration:** `--fail-on-error` flag as quality gate; trigger on `.agents/skills/**` path changes. Source: promptfoo.dev/docs/integrations/ci-cd.
|
||||
|
||||
**Alternatives if vendor neutrality degrades:** DeepEval (pytest-native, zero adoption friction for Python teams, free); Arize Phoenix (self-hosted, vendor-neutral). Source: dev.to/thedailyagent/top-5-ai-agent-eval-tools-after-promptfoos-exit.
|
||||
|
||||
### Three eval types
|
||||
|
||||
| Type | When | What | Tool |
|
||||
|---|---|---|---|
|
||||
| Trigger | Every skill change in CI | Activation correct? | promptfoo |
|
||||
| Output | Every skill change in CI | Structured, quality output? | promptfoo + LLM-rubric |
|
||||
| Regression | Weekly cron | Did model update break behaviour? | promptfoo matrix |
|
||||
|
||||
### Counterarguments
|
||||
|
||||
LLM-as-judge introduces model bias. Full output evals on every PR is expensive at API token rates.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Promptfoo documentation is now from an OpenAI-acquired company. confident-ai.com and augmentcode.com are commercial eval platforms.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Every skill has trigger tests and output tests.
|
||||
2. Trigger evals: 10–20 cases including explicit, implicit, and negative.
|
||||
3. Deterministic assertions as blocking gates; LLM-rubric as secondary.
|
||||
4. Co-locate `eval.yaml` with `SKILL.md`.
|
||||
5. Skill file changes trigger CI eval runs; failing eval blocks merge.
|
||||
6. Weekly regression evals catch model-version-induced degradation.
|
||||
7. Every corrected skill behaviour generates a new eval case.
|
||||
8. Promptfoo is the starting point; track OpenAI acquisition; DeepEval or Arize Phoenix as fallbacks.
|
||||
|
||||
---
|
||||
|
||||
## Topic 9: Token Optimisation
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
**The question:** Where do tokens actually go, what are the highest-ROI reduction techniques, and how does this apply across the factory?
|
||||
|
||||
### The orientation problem
|
||||
|
||||
Agents spend up to 80% of token budget on "orientation" rather than actual work. Source: earezki.com/ai-news/2026-02-26 (February 2026); smartscope.blog/en/generative-ai/claude/agents-md-token-optimization-guide-2026.
|
||||
|
||||
Average prompt length grew ~4× from 2024 to 2025 (1,500 → 6,000 tokens). Source: Tokalator paper (arxiv.org/pdf/2604.08290), 100 trillion tokens from OpenRouter platform.
|
||||
|
||||
Multi-agent systems consume 4–15× more tokens than single calls if not optimised. Source: obviousworks.ch/en/token-optimization-saves-up-to-80-percent-llm-costs (2026).
|
||||
|
||||
**Spec-driven development as token optimisation:** "When requirements, scope, acceptance criteria, constraints, interfaces, and non-goals live in a written specification, the developer does not need to re-explain the same feature intent in every coding conversation. This is not merely a process preference — it is a context optimisation technique." Source: medium.com/@nprasads/token-efficiency-in-ai-coding-agents (April 13, 2026). This connects Topic 3 and Topic 9: spec-first is both a quality practice and a token reduction strategy.
|
||||
|
||||
### Academic backing for skill size limits
|
||||
|
||||
SkillReducer paper: "A skill with a large body and references can consume tens of thousands of tokens per invocation, and when multiple skills are active simultaneously, their cumulative cost can dominate the context budget." Source: arxiv.org/pdf/2603.29919. This provides academic backing for the 500-line limit in Topic 1.
|
||||
|
||||
### Prompt caching — corrected framing and TTL regression
|
||||
|
||||
**Two distinct metrics — do not conflate:** (a) **Per-token discount:** Sonnet 4.6 cache reads cost $0.30/MTok vs. $3.00/MTok standard — a 90% per-token reduction on cached content. This is documented and confirmed. (b) **Cache hit rate:** The proportion of input tokens that are actually served from cache. The 74–84% figure cited in original research came from a production support bot (BSWEN's personal Claude Code usage analysis, single-blogger) and a security product (ProjectDiscovery, 84% after explicit breakpoint placement). Agentic coding workflows have substantially more variable dynamic content (different diffs, files, task inputs) — **no published study measures hit rates specifically for agentic coding workflows**. Treat 74–84% as an upper bound from a different workflow type, not an expected coding workflow result.
|
||||
|
||||
**Critical March 2026 TTL regression:** Around March 6, 2026, Anthropic silently changed the default cache TTL from 1 hour back to 5 minutes for Claude Code Pro/API users. Max subscribers retained 1-hour TTL. This caused 20–32% cache-creation cost inflation for Pro/API users and was not communicated via changelog. Source: github.com/anthropics/claude-code/issues/46829; dev.to/whoffagents (April 2026).
|
||||
|
||||
**Current TTL structure:**
|
||||
- Claude Code Max / Anthropic Max subscribers: 1-hour TTL ($6/MTok write, $0.30/MTok read)
|
||||
- Claude Code Pro / API users: 5-minute TTL ($3.75/MTok write, $0.30/MTok read)
|
||||
- Break-even: 1 hit for 5-minute TTL; ≥2 hits for 1-hour TTL
|
||||
|
||||
**Peer-quality data (arXiv 2601.06007, "Don't Break the Cache", February 2026):** Evaluated caching on DeepResearch Bench (10K-token system prompts across agentic tasks). Found **41–80% cost reduction** and 13–31% TTFT improvement in tested agentic workflows. This is the most methodologically rigorous available data point on caching in agentic (vs. support bot) contexts. Source: arxiv.org/pdf/2601.06007.
|
||||
|
||||
### Context architecture (40–60% savings)
|
||||
|
||||
Always-loaded index file ≤200 lines; skills load on demand. Source: code.claude.com/docs/en/best-practices; code.claude.com/docs/en/costs; towardsdatascience.com/agentic-ai-how-to-save-on-tokens.
|
||||
|
||||
"Decisions not descriptions" → ~20% reduction. AST-level loading → ~65% fewer file reads. ⚠️ Source: earezki.com/ai-news/2026-02-26 (single practitioner case study; Alessi is the vexp tool developer — commercial bias). See Topic 7 for full caveat.
|
||||
|
||||
### Model routing (40–60% cost reduction)
|
||||
|
||||
Prices verified April 2026 (official Anthropic documentation): Haiku 4.5 $1/$5; Sonnet 4.6 $3/$15; Opus 4.7 $5/$25. Note: Opus 4.7 uses a new tokenizer generating up to 35% more tokens for the same input text. Source: platform.claude.com/docs/en/about-claude/pricing; cloudzero.com/blog/claude-api-pricing; finout.io/blog/anthropic-api-pricing.
|
||||
|
||||
### Batch API
|
||||
|
||||
50% discount on both input and output for non-time-sensitive work. Source: platform.claude.com/docs/en/about-claude/pricing (official documentation).
|
||||
|
||||
### Agent team costs
|
||||
|
||||
~7× more tokens than standard sessions. Source: code.claude.com/docs/en/costs (official documentation).
|
||||
|
||||
### Tool definition bloat
|
||||
|
||||
55,000–134,000 tokens of tool definitions before optimisation. Source: **Anthropic engineering blog "Introducing advanced tool use" (November 24, 2025)** — cited in towardsdatascience.com/agentic-ai-how-to-save-on-tokens (which correctly attributes it to Anthropic documentation). The 134K figure reflects Anthropic's own internal measurement across multiple MCP servers; the 55K figure covers a representative multi-server configuration (~58 tools: GitHub, Slack, Sentry, Grafana, Splunk). Anthropic's Tool Search Tool reduces this to ~5K. Source: mindstudio.ai/blog/reduce-token-usage-ai-agents-mcp-optimization.
|
||||
|
||||
### Counterarguments
|
||||
|
||||
90% per-token caching savings requires a high hit rate on a stable prefix; for variable-content agentic coding workflows the hit rate may be substantially lower than the 74–84% figures from support-bot deployments. Haiku routing risks misclassification. The 20% and 65% context-file reduction figures are single case studies from the same author who has a commercial interest in one of the tools.
|
||||
|
||||
### Bias flags
|
||||
|
||||
Most token cost figures from Anthropic documentation or third-party analysis. Case studies should not be treated as guaranteed results. Vexp case study: commercial bias as noted above. Caching hit-rate figures from non-coding workflow deployments.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Orientation is the primary token sink — solve with structured context before micro-optimising.
|
||||
2. Prompt caching is the highest single-ROI optimisation — static content first, dynamic input last. Distinguish the per-token discount (90% on cache hits, confirmed) from the hit rate (workflow-dependent; 41–80% in peer-quality agentic data; higher in stable deployments).
|
||||
3. Be aware of the March 2026 TTL regression: Pro/API users get 5-minute TTL by default; 1-hour TTL requires Max subscription or explicit API configuration. Factor this into caching strategy — intra-session reuse is reliable; inter-session reuse requires Max tier.
|
||||
4. Always-loaded context file ≤200 lines; specialised instructions in skills.
|
||||
5. Route by complexity: lightweight model for formatting/classification, standard for most coding, flagship for adversarial review.
|
||||
6. Batch API for non-time-sensitive factory work.
|
||||
7. Sequential roles, not concurrent agent teams.
|
||||
8. Scope MCP tools per skill; concise tool descriptions.
|
||||
9. Measure tokens per completed task — the sustainability metric.
|
||||
|
||||
---
|
||||
|
||||
## Topic 10: MCP Server Integration
|
||||
|
||||
Sources for this topic are verified from session search results (preserved in original search pass).
|
||||
|
||||
**The question:** What MCP servers exist for the factory's stack, how do you connect them securely?
|
||||
|
||||
### What MCP enables
|
||||
|
||||
Without MCP: agents read/write files through custom scripts. With MCP: agents create issues, inspect containers, query infrastructure nodes as direct tool calls. HITL gates still apply. "MCP is not just a developer convenience — it is a new and distinct security boundary that needs policy, oversight, and operational discipline." Source: CoSAI/OASIS WS4 Secure Design for Agentic Systems (github.com/cosai-oasis/ws4-secure-design-agentic-systems); MCP specification Security Best Practices (modelcontextprotocol.io). ⚠️ Earlier research cited a "CIS MCP Companion Guide (April 20, 2026)" — no such CIS Benchmark for MCP exists as of May 2026; reference corrected.
|
||||
|
||||
### Gitea MCP Server
|
||||
|
||||
Official `gitea.com/gitea/gitea-mcp` — MIT-licensed, actively maintained. **At v1.2.0 as of May 10, 2026** (not pre-v1.0 as originally stated). Tool schema reduced 26.6% and `--tools` flag added May 2026. Two modes: stdio (local, no auth by default) and HTTP (remote, multi-user). Source: gitea.com/gitea/gitea-mcp (repository, releases page verified May 2026); pkg.go.dev/gitea.com/gitea/gitea-mcp.
|
||||
|
||||
**`--tools` flag clarification:** The `--tools` flag is a **client-facing capability filter** — it restricts which tools appear in the `tools/list` JSON-RPC response (reducing context token cost). It does NOT restrict what the underlying PAT can do on the Gitea server side. A PAT with `write:repository` still enables `DELETE /repos/{owner}/{repo}` regardless of `--tools` configuration. Treat `--tools` as a token-optimisation and scope-communication mechanism, not a security boundary.
|
||||
|
||||
**Token efficiency:** gitea-mcp-compact fork (github.com/appslop/gitea-mcp-compact) reduces token usage 70–80% on list/search operations per its README. ⚠️ Community fork: 0 stars, 0 forks, 1 commit as of May 2026 — very low maturity; verify before using in production.
|
||||
|
||||
**Critical security gap — Gitea token scopes:** Gitea PAT scopes are category-level (read/write × repository/issue/user/etc.) but have no sub-scopes to prohibit destructive operations within a category. `write:repository` includes `DELETE /repos/{owner}/{repo}`. Gitea 1.20 redesigned the PAT scope system; Gitea 1.23 (the relevant version) extended granular scopes to OAuth2 applications. **The underlying PAT category model has not become finer-grained since 1.20 — the warning about repository deletion remains correct for PAT-based access.** Source: github.com/appslop/gitea-mcp-compact README; docs.gitea.com/development/oauth2-provider.
|
||||
|
||||
**Forgejo alternative:** `github.com/raohwork/forgejo-mcp` — supports both Gitea and Forgejo, available in stdio and HTTP (multi-user) modes. Source: github.com/raohwork/forgejo-mcp.
|
||||
|
||||
### Infrastructure MCP servers
|
||||
|
||||
Docker: `ckreiling/mcp-server-docker` and Docker's official MCP Toolkit. Source: mcp.aibase.com/class/Virtualization (2026); docker.com/blog.
|
||||
|
||||
Kubernetes: K8m — AI-driven K8s console with MCP services, multi-cluster support. Source: mcp.aibase.com/class/Virtualization.
|
||||
|
||||
Proxmox: Python-based community server (~10K GitHub stars). Source: mcp.aibase.com/class/Virtualization. ⚠️ Community-maintained; requires source review before connecting to VM management tokens.
|
||||
|
||||
Ansible MCP: None found. Execution stays human-controlled. Source: absence confirmed by search.
|
||||
|
||||
Terraform MCP: None stable found. Same reasoning. Source: absence confirmed by search.
|
||||
|
||||
### MCP security
|
||||
|
||||
⚠️ **"CIS MCP Companion Guide" reference unverifiable:** No CIS Benchmark for MCP exists as of May 2026. The most likely intended primary sources are the **CoSAI/OASIS WS4 Secure Design for Agentic Systems** (github.com/cosai-oasis/ws4-secure-design-agentic-systems) and the **MCP specification's own Security Best Practices**. The reference has been updated below.
|
||||
|
||||
"Using MCP does not automatically make an AI system safe." Source: CoSAI Secure Agentic AI Design Guide.
|
||||
|
||||
"The authorization spec says authorization is optional overall, applies to HTTP-based transports, and **should not be used for stdio the same way.**" Source: modelcontextprotocol.io/specification/draft/basic/authorization. **Clarification:** stdio is explicitly designed for local-trust environments; treating "no auth by default" as a security gap misreads the spec. In a single-user homelab where only the user can send prompts, stdio provides the correct security model. The real stdio risk is **supply-chain** — malicious npm/uvx packages that bundle an MCP server (multiple CVEs in 2025–2026: CVE-2025-49596 MCP Inspector, CVE-2026-22252 LibreChat). Install MCP server packages only from reviewed, version-pinned sources.
|
||||
|
||||
"If an unmanaged AI agent is compromised, it can use legitimately authorised tokens to exfiltrate data. The activity appears as normal API traffic." Source: nudgesecurity.com/post/mcp-security-risks (May 2026).
|
||||
|
||||
**Homelab vs. enterprise threat model:** Enterprise security guidance (Red Hat OAuth 2.1, multi-tenant auth) addresses adversarial multi-user environments. In a single-user self-hosted homelab, the relevant threat model is: (1) **supply chain** — malicious MCP server packages; (2) **prompt injection** from untrusted content the agent reads (PR bodies, issue descriptions, code comments). Transport-level auth requirements do not apply to single-user stdio. The dedicated agent account mitigation addresses audit trail and selective revocation, not granular token permissions — scope accordingly.
|
||||
|
||||
For multi-user HTTP MCP: prefer delegating to an external OAuth/OIDC provider rather than embedding auth in the MCP server. Source: redhat.com/en/blog/mcp-security-implementing-robust-authentication-and-authorization (March 11, 2026).
|
||||
|
||||
**Claude Code per-session/per-role MCP filtering: not currently supported.** Multiple open feature requests exist (github.com/anthropics/claude-code/issues/7328 tool filtering; #45293 per-session MCP server profiles; #4380 per-agent tool filtering). What currently exists: three install scopes (`local`, `project`, `user`); an interactive `@` menu (v2.0.10+) for session-level tool selection. Sub-agents inherit all parent MCP tools. The Claude Agent SDK (separate from Claude Code) does support `allowed_tools` / `skills` filtering programmatically. Until native Claude Code per-role filtering ships, use the interactive `@` menu manually or the Agent SDK for programmatic workflows. Source: github.com/anthropics/claude-code/issues/4380.
|
||||
|
||||
**forgejo-mcp HTTP mode:** Does not add OAuth scoping over the underlying PAT — relays a bearer PAT in the Authorization header. Users manage scopes themselves. Source: github.com/raohwork/forgejo-mcp README.
|
||||
|
||||
MCP security threat taxonomy covers five layers: transport/communication, authentication/identity, context integrity/confidentiality, authorization/privilege management, supply chain security. Source: aembit.io/blog/the-ultimate-guide-to-mcp-security-vulnerabilities (March 19, 2026).
|
||||
|
||||
### Counterarguments
|
||||
|
||||
gitea-mcp is now at v1.2.0 — tool schemas still change without semver guarantees at this stage. Proxmox MCP is community Python — requires source review. stdio auth model is the correct model for single-user controlled environments (not a gap, as the original research implied).
|
||||
|
||||
### Bias flags
|
||||
|
||||
MCP security statistics from vendors with commercial interest in finding MCP risks. Red Hat and enterprise-framing guidance applies to multi-tenant environments — calibrate to homelab threat model. gitea-mcp-compact 70–80% figure is self-reported in the fork's README with no independent verification and very low maturity indicators.
|
||||
|
||||
### Provisional principles
|
||||
|
||||
1. Dedicated agent user account per service — never use personal admin tokens. This improves audit attribution and selective revocation; it does NOT improve token granularity (same PAT category scopes apply).
|
||||
2. Use `--tools` filtering on every MCP server to reduce token cost and communicate intended scope. This is a capability/token filter, not a security boundary — the underlying PAT retains full Gitea-side permissions.
|
||||
3. Gitea PAT scopes are category-level with no sub-category prohibitions — a `write:repository` token can delete repos. Mitigate with: dedicated `agent` user account, repo-scoped access only, quarterly rotation. Gitea 1.23 extended granular scopes to OAuth2 applications; PAT model remains category-level.
|
||||
4. Ansible and Terraform execution stays human-controlled — no MCP servers for these.
|
||||
5. Ops role uses MCP for read operations; write operations still require HITL approval.
|
||||
6. Claude Code does not currently support per-session/per-role MCP filtering natively — use the interactive `@` menu for manual scoping, or the Claude Agent SDK for programmatic control.
|
||||
7. Pin MCP server versions — gitea-mcp is at v1.2.0 as of May 2026; pin to a specific tag. Retest on upgrade.
|
||||
8. Every MCP tool call logged per governance constitution §7.
|
||||
9. For multi-user setup, switch to HTTP transport with OAuth 2.1 — forgejo-mcp supports HTTP mode (note: does not add OAuth scoping over the underlying PAT; users manage scopes themselves).
|
||||
10. Primary homelab MCP threats are supply chain (verify npm/uvx packages before installing) and prompt injection from untrusted content; transport-level auth is not the primary concern for single-user stdio deployments.
|
||||
|
||||
---
|
||||
|
||||
*Research session completed May 2026. Version 1.4 — deep research pass (Session 3) applied across all topics. Twelve challenges verified; ten additional elements checked.*
|
||||
*Design specification: `ai-coding-factory-principles.md`*
|
||||
*Session audit trail: `ai-coding-factory-session.md`*
|
||||
214
docs/research/ai-coding-factory/ai-coding-factory-session.md
Normal file
214
docs/research/ai-coding-factory/ai-coding-factory-session.md
Normal file
@@ -0,0 +1,214 @@
|
||||
# AI Coding Factory — Session Audit Trail
|
||||
|
||||
**Purpose:** Auditability of the artifact creation process. Documents what was done, how, why, and what decisions were made or deferred. Not a task list — a process record.
|
||||
|
||||
**Project:** AI coding skills, workflows, and templates research for building a personal AI coding factory spanning software development and infrastructure management.
|
||||
**Sessions:** Three sessions, May 2026.
|
||||
**Artifacts produced:** See artifact registry below.
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
The goal was to produce a research foundation and governance principles before building an AI-assisted software development and infrastructure management setup. The author wanted practices grounded in current research — not opinion — that would govern skill design, workflow structure, and agent behaviour across a homelab and development context.
|
||||
|
||||
The work was deliberately sequenced: research first, then distil into principles. The research must be independently challengeable; the principles must be applicable at both homelab and team scale.
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
### Research approach
|
||||
- Web search from reliable sources per topic. Training data avoided as a source; sourced findings preferred throughout. Whether this was fully achieved cannot be independently verified — some analytical and synthesis sections may reflect training data rather than sourced research, particularly where citations are sparse.
|
||||
- Research conducted topic by topic so findings can be challenged before they harden into design decisions.
|
||||
- Provisional design decisions established during scoping were treated as hypotheses to be tested, not assumptions to be confirmed.
|
||||
- Bias flags reference limitations within the sourced research itself (vendor-funded studies, single-study findings, practitioner observation rather than controlled research) — not the AI assistant's training data. However, the line between "synthesising sourced findings" and "drawing on training data" is not always clear, and some analytical conclusions may have training data influence that went undetected.
|
||||
|
||||
### Output format per topic
|
||||
Each topic in the research document follows: question being researched → findings (with sources) → counterarguments → bias flags → provisional principles.
|
||||
|
||||
### Distillation logic
|
||||
Research document = full sourced reasoning (human reference, never in agent context).
|
||||
Principles document = operative design principles derived from research (agent-readable, repo artifact).
|
||||
Skills index = temporary pre-implementation build reference for skill trigger descriptions and constraints.
|
||||
Session document = this file — audit trail of the creation process.
|
||||
|
||||
### Standing integrity caveat
|
||||
The instruction to use web sources was given and followed to the best of the session's ability. However, AI-generated research cannot self-audit for training data influence with certainty. Analytical framing, the choice of which findings to emphasise, and the synthesis of sources into conclusions all carry some risk of undetected training data influence. Where citations are absent or thin, treat findings with additional scepticism and verify independently before relying on them for high-stakes decisions.
|
||||
|
||||
---
|
||||
|
||||
## Scope Decisions
|
||||
|
||||
These decisions were made during the initial scoping interview before research began. Each shaped all subsequent work.
|
||||
|
||||
| Decision | Conclusion | Rationale |
|
||||
|---|---|---|
|
||||
| Output form | Research doc + principles doc + skills index (temporary) | Separates reasoning from operative principles; skills index preserves implementation-level detail not appropriate for principles doc |
|
||||
| Audience | Solo now, team-inheritable later | Every design choice must be legible without the author present |
|
||||
| Tool approach | Tool-agnostic with thin adapters | Will be used across Claude Code, Copilot, Cursor, and others; cannot be tool-locked |
|
||||
| Stack scope | Python + TypeScript (app), Ansible, Terraform (future), Docker, Proxmox, K8s (infra) | Reflects actual current and near-future tooling |
|
||||
| SDLC scope | Full cycle — design through operations | Factory that only covers implementation is incomplete |
|
||||
| Research depth | Sourced, with tradeoffs | Design decisions are only as strong as the evidence behind them |
|
||||
| Gitea assumption | Self-hosted, Gitea Actions, no GitHub | Changes CI skill design significantly; established before research began |
|
||||
| Research sequencing | Topic by topic, user challenges before proceeding | Prevents hypotheses hardening before they are tested |
|
||||
|
||||
---
|
||||
|
||||
## Topics Researched
|
||||
|
||||
| # | Topic | Key finding | Status |
|
||||
|---|---|---|---|
|
||||
| 1 | Skills library design | SKILL.md open standard published Dec 2025; description field is the trigger; security risks real even for self-authored skills | Complete |
|
||||
| 2 | Workflow and agent chain design | WorkflowContext schema before any skill; compact structured outputs; context pollution is the primary failure mode | Complete |
|
||||
| 3 | SDLC phase coverage | Spec-first prevents drift; AI-generated tests must derive from spec not implementation; circular reasoning is a documented failure mode | Complete |
|
||||
| 3b | Feedback loops | Two loops required (short + long); auto memory is machine-local not repo; LESSONS.md must be repo-committed | Complete — merged with Topic 3 in research doc |
|
||||
| 4 | IaC patterns | Provider version drift is the primary AI IaC failure mode; Ansible idempotency must be explicit; Proxmox has thin training data coverage | Complete |
|
||||
| 5 | Gitea and self-hosted CI | Mostly GitHub Actions compatible; AI defaults to GitHub assumptions; no native AI integration; must build AI review as custom workflow | Complete |
|
||||
| 6 | Agent role design | Cognitive separation by role outperforms generalist; gstack is the reference implementation; six roles cover the factory | Complete |
|
||||
| 7 | Prompt engineering for code | Context engineering superseded prompt engineering; 3,000-token ceiling framing corrected (see Session 3); constraints section is highest-ROI element | Complete — corrected in v1.4 |
|
||||
| 8 | Validation and testing of skills | Two test types required; trigger tests most commonly missing; promptfoo acquired by OpenAI March 2026 — track vendor neutrality; version-pin | Complete |
|
||||
| 9 | Token optimisation | Added by user request; orientation problem is the primary sink; prompt caching per-token savings confirmed; TTL regression documented (March 2026); model routing 40–60% savings | Complete — corrected in v1.4 |
|
||||
| 10 | MCP server integration | Added per recommendation; official gitea-mcp (v1.2.0 as of May 2026); no granular Gitea PAT token scopes; Claude Code per-role MCP filtering not supported natively; Ansible/Terraform execution stays human-controlled | Complete — corrected in v1.4 |
|
||||
|
||||
---
|
||||
|
||||
## Key Decisions Log
|
||||
|
||||
| Decision | Why |
|
||||
|---|---|
|
||||
| SKILL.md as the packaging format | Closest thing to a universal standard across Claude Code, Codex CLI, Gemini CLI, Copilot, Cursor. Published by Anthropic Dec 2025, MIT-licensed. |
|
||||
| `.agents/skills/` as canonical directory | Open standard path. Tool-specific directories (`.claude/skills/`, etc.) are thin adapters. Single source of truth per governance pattern. |
|
||||
| AGENTS.md as primary operative file | Consistent with existing governance foundation. Tool adapters reference it; content not duplicated per tool. |
|
||||
| Spec-first as mandatory workflow step | Research confirmed vibe coding as the documented failure mode. Spec is the contract between all workflow stages. |
|
||||
| LESSONS.md committed to repo | Auto memory (`~/.claude/projects/.../memory/`) is machine-local, not shared. Repo-level lessons require an explicit committed file. |
|
||||
| Two feedback loops explicitly designed | Only the short loop (finding → fix) happens by default. The long loop (pattern → convention) must be deliberately built. |
|
||||
| Gitea MCP dedicated agent user | Gitea PAT scopes are category-level; a write:repository token can still delete repos regardless of account type. Dedicated account improves audit attribution and selective revocation — it does not improve token granularity. Corrected framing in Session 3. |
|
||||
| Ansible/Terraform: no MCP execution | Both make infrastructure changes; human-controlled execution aligns with the governance constitution. Agents generate; humans apply. |
|
||||
| Sequential roles not concurrent agents | Agent teams use ~7× more tokens. Solo homelab context; parallelism overhead not justified. |
|
||||
| Six roles | Covers the full SDLC with distinct cognitive modes. Based on gstack (~96K stars, May 2026) as the primary reference implementation. The 810× productivity claim uses a self-defined metric and part-time baseline — the cognitive-separation insight stands independently. |
|
||||
| Token routing: Haiku/Sonnet/Opus | 40–60% cost reduction from proper routing documented. Haiku for formatting; Sonnet for most coding; Opus for architecture and adversarial review. |
|
||||
| Topic 9 (token optimisation) added | User request during research. Justified by existing governance constitution's sustainability metric: tokens per unit of value. |
|
||||
| Topic 10 (MCP integration) added | Recommended based on gap analysis. Without MCP research, the factory would produce agents that generate code about infrastructure but cannot act on it. |
|
||||
| Topics 3 and 3b merged in research doc | Feedback loops are part of the SDLC cycle, not a separate topic. Editorial decision to improve document coherence. |
|
||||
| Design doc reframed as principles doc | Original design doc was positioned as a blueprint/implementation guide — templates, CI YAML, build order, repo structure diagrams. The appropriate role is operative design principles, analogous to the governance constitution. Implementation artifacts belong in the repo, not in a documentation artifact. |
|
||||
| Agent notes merged into principles | The notes were created by analogy with the governance project's ai-agent-instructions-notes.md. But the factory has no single short operative file equivalent to AGENTS.md — the notes' rationale purpose collapses into the principles doc. Two documents answering the same question with no clear boundary between them. |
|
||||
| Skills index as temporary artifact | Skill trigger descriptions and key constraints (27 skills) are not appropriate for the principles doc (too implementation-specific) and not in the research doc (too specific). Preserved as a pre-implementation build reference. Delete once actual SKILL.md files exist in the repo. |
|
||||
| Name: ai-coding-factory-principles.md | Preferred over FACTORY.md. Consistent with research doc naming convention; explicit about content (principles, not configuration). |
|
||||
|
||||
---
|
||||
|
||||
## Corrections and Integrity Notes
|
||||
|
||||
| What was wrong | When caught | How corrected |
|
||||
|---|---|---|
|
||||
| MEMORY.md described as a repo-level shared file | During Topic 3b | User caught it. Corrected: MEMORY.md / auto memory is machine-local, per `~/.claude/projects/<hash>/memory/`. LESSONS.md (repo-committed) is the correct mechanism for shared lessons. Design document updated. |
|
||||
| Design document used Claude Code-specific paths and syntax | Post-artifact review | User caught it: `.claude/skills/`, `CLAUDE.md`, `@filename` syntax, `ccusage` are Claude Code-specific. Corrected: design document updated to use `.agents/skills/`, tool-agnostic context file references, generic adapter pattern. |
|
||||
| Research document lacked source citations | Post-artifact review | User caught it. Research doc updated to add source URLs per topic (v1.1), then source audit applied (v1.2): Topics 1–4 flagged ⚠️ UNVERIFIED URL because search results were cleared from context; Topics 5–10 checked against preserved search results and missing citations added. |
|
||||
| Topics 1–4 re-searched (v1.3) | User-requested re-search pass | All four topics re-searched with new web searches. Key findings added: Agent Skills spec confirmed at agentskills.io (December 18, 2025); ToxicSkills security figures now sourced to snyk.io/blog (February 5, 2026), ClawHavoc to snyk.io/articles (February 3, 2026), Mobb.ai 22,511 skills audit to agensi.io, OWASP Agentic Skills Top 10 to owasp.org (April 27, 2026); SDD now confirmed mainstream with verified sources (thebcms.com, augmentcode.com, growexx.com); Veracode 2025 GenAI Code Security Report confirmed as primary AI code vulnerability benchmark (2.74× more vulnerabilities, 45% OWASP Top 10 rate); Terraform provisioner deprecation confirmed from scalr.com and spacelift.io; bpg/proxmox provider version 0.78 confirmed from computingforgeeks.com tested on real Proxmox hardware April 2026. The 67% testers figure and Atlassian RovoDev 38.7% figure were NOT found in re-search — removed from research doc. |
|
||||
| Multiple found-but-not-cited sources | Source audit (v1.2) | Sources found in preserved search results (Topics 5–10) but not included in v1.1: gitea-agent (devactivity.com); serverspan.com 2026 self-hosted git guide; HCLTech autonomous software factory; IBM Bob; DevOps Digest 2026 predictions; gstack additional sources (mager.co, awesomeagents.ai); SkillReducer paper (arxiv.org/pdf/2603.29919) — most significant miss, provides academic backing for 500-line skill limit; medium.com/@nprasads spec-driven-as-token-optimisation; forgejo-mcp alternative; redhat.com MCP OAuth 2.1 guidance; aembit.io MCP security taxonomy; promptfoo agent-skill integration docs; eval tooling comparison sources. All added in v1.2. |
|
||||
| Design document not updated after research re-search (v1.2) | Post-v1.3-research review | Five skill registry entries updated based on finding changes: write-spec now assesses whether spec overhead is warranted before proceeding; security-review now distinguishes OWASP LLM Top 10 / Agentic AI Top 10 / Agentic Skills Top 10; write-ansible-role now names specific idempotency failure modes; write-terraform-module now cites HashiCorp "last resort" authority for provisioner prohibition; iac-security-review now includes non-idempotent Ansible patterns. |
|
||||
| Session audit trail not created initially | Post-artifact review | User caught it. This document created as correction. |
|
||||
| SkillsBench "2–3 skills optimal" figure unverified | Session 2 review | Removed from session topics table. Flagged ⚠️ unverified in research doc. Not incorporated into principles doc. |
|
||||
| SkillsBench unit-of-analysis error | Session 3 deep research | Confirmed as a real benchmark (arXiv 2602.12670) but the finding was misread: "2–3 modules per skill" not "2–3 skills per task." Also: 19% of tasks regressed under curated skills; self-generated skills yielded no benefit. Research doc updated in Topic 2. |
|
||||
| "3,000-token degradation ceiling" framing | Session 3 deep research | Misread of Levy et al. (2024): 3,000 tokens was the maximum tested, not a ceiling; degradation begins at ~500 tokens in that study. 2026 evidence (OOLONG, arXiv 2601.15300, Anthropic MRCR v2) supersedes the 2024 study. Research doc Topic 7 rewritten; principles doc skill-limit rationale updated to SkillReducer "less-is-more" finding. |
|
||||
| "Veracode 2.74× more vulnerabilities" misattribution | Session 3 deep research | 2.74× figure comes from CodeRabbit (December 2025), not Veracode, and applies specifically to XSS vulnerabilities. Veracode's actual finding is 45% OWASP Top 10 rate. Added Schreiber & Tippe (arXiv 2510.26103): 12.1% real-world CWE rate. Research doc Topic 3 corrected; principles doc Section 6 corrected. |
|
||||
| SDD "now mainstream" | Session 3 deep research | Overstated. No controlled study of SDD vs ad-hoc prompting outcomes exists. Kiro 250K figure is signups over ~4 months, not active users. Thoughtworks full rationale added. Updated to "rapidly emerging." Research doc Topic 3 and principles doc Section 1 corrected. |
|
||||
| "CIS MCP Companion Guide" reference | Session 3 deep research | No CIS Benchmark for MCP exists as of May 2026. Reference replaced with CoSAI/OASIS WS4 and MCP specification security guidance. |
|
||||
| stdio MCP framed as a security gap | Session 3 deep research | Misframed. MCP spec explicitly designates stdio as the local-trust transport; "no auth by default" is the correct design for single-user environments. Real homelab threats are supply chain and prompt injection. Research doc Topic 10 and principles doc Section 10 corrected. |
|
||||
| Dedicated agent account framed as token-granularity control | Session 3 deep research | Corrected: dedicated account improves audit attribution and selective revocation; it does not improve token granularity (same PAT category scopes apply). Research doc Topic 10 and principles doc Section 10 corrected. |
|
||||
| AST10 described as a published OWASP standard | Session 3 deep research | AST10 is an OWASP Incubator project, not a ratified standard (v0.0.0 on OWASP site; v1.0 targeted Q3 2026). "April 27, 2026" publication date appears only on third-party blogs. AST01 corrected to apply only to third-party registry skills; self-authored factory risks are AST03/04/06/07/09. Research doc Topic 1 and principles doc Section 2 corrected. |
|
||||
| gstack "80K+ stars" and "10× productivity" | Session 3 deep research | Stars updated to ~96K (May 2026). Actual claim is 810× vs a part-time 2013 baseline — uncontrolled, confounds parallelism with role design, no independent replication. Research doc Topic 6 corrected; principles doc Section 5 corrected. |
|
||||
| "32+ tools" adopted Agent Skills standard | Session 3 deep research | Corrected to "26+" — consistent secondary sources as of May 2026 document 26 confirmed adopters. Research doc Topic 1 and principles doc Section 2 corrected. |
|
||||
| gitea-mcp described as pre-v1.0 | Session 3 deep research | gitea-mcp is at v1.2.0 as of May 10, 2026 — not pre-v1.0. Research doc Topic 10 and principles doc Section 10 corrected. |
|
||||
| `--tools` flag framed as a security boundary | Session 3 deep research | --tools is a client-facing capability filter (token-optimisation, scope communication). The underlying PAT retains full Gitea-side permissions regardless. Research doc Topic 10 and principles doc Section 10 corrected. |
|
||||
| Claude Code per-session/per-role MCP filtering | Session 3 deep research | This capability does not exist in Claude Code as of May 2026. Multiple open feature requests. Available workarounds: interactive `@` menu, Claude Agent SDK for programmatic control. Research doc Topic 10 and principles doc Section 10 corrected. |
|
||||
| Prompt caching TTL | Session 3 deep research | March 2026 silent regression: Anthropic changed default TTL from 1 hour back to 5 minutes for Pro/API users (Max retains 1-hour). This was not communicated via changelog. Research doc Topic 9 and principles doc Section 9 corrected. |
|
||||
| Prompt caching hit rate vs. per-token discount conflated | Session 3 deep research | Two distinct metrics treated as one. Hit rate is workflow-dependent (peer data: 41–80% in agentic tasks, arXiv 2601.06007). Per-token discount is 90% on cache reads (confirmed). Research doc Topic 9 and principles doc Section 9 clarified. |
|
||||
| Alessi case study figures cited without commercial bias flag | Session 3 deep research | Nicola Alessi is the developer of vexp, the AST-loading tool used in the "40 → 5 file reads" case study. Both figures (20% and 65%) are self-reported by someone with commercial interest in one of the techniques. Research doc Topic 7 and Topic 9 updated with bias flag. |
|
||||
| Tool definition bloat source cited as Towards Data Science | Session 3 deep research | Primary source is the Anthropic engineering blog "Introducing advanced tool use" (November 24, 2025). Towards Data Science correctly attributes it to Anthropic but is a secondary source. Research doc Topic 9 corrected. |
|
||||
| AORCHESTRA +16.28% improvement presented as unconditional | Session 3 deep research | +16.28% is relative improvement and Gemini-3-Flash specific — not a general claim across models. Research doc Topic 2 and principles doc Section 4 updated with qualifier. |
|
||||
| forgejo-mcp HTTP mode adds OAuth scoping | Session 3 deep research | Refuted: forgejo-mcp HTTP mode relays a bearer PAT in the Authorization header; users manage scopes themselves. Research doc Topic 10 corrected. |
|
||||
|
||||
---
|
||||
|
||||
## Session 1 — Research and Initial Artifact Creation
|
||||
|
||||
### What was done
|
||||
1. **Scoping session:** Established purpose, audience, methodology, and scope decisions documented above. Output: shared understanding of what to build and why.
|
||||
2. **Topic-by-topic research:** Ten governance topics researched in sequence (with Topic 3b on feedback loops merged into Topic 3). Each topic was web-searched, synthesised, and reviewed before proceeding.
|
||||
3. **Source corrections:** Multiple rounds of source verification and citation fixes (v1.1 → v1.2 → v1.3). Topics 1–4 fully re-searched after search results were cleared from context.
|
||||
4. **Artifact distillation:** Research document → design specification → agent notes → session doc.
|
||||
5. **Post-artifact corrections:** Tool-agnostic path fixes, session audit trail created, skill registry entries updated to reflect re-search findings.
|
||||
|
||||
---
|
||||
|
||||
## Session 2 — Artifact Restructuring
|
||||
|
||||
### What was done
|
||||
|
||||
1. **Design doc repositioning identified:** The design doc (v1.2) was structured as a blueprint — templates, CI YAML, build order, repo structure diagrams. The appropriate role for a permanent artifact is operative principles (what to do and why), not implementation scaffolding (how to implement it). Implementation artifacts belong in the repo.
|
||||
|
||||
2. **Agent notes overlap identified:** The agent notes were created by analogy with the governance project but serve a different need. In the governance project, the notes explain the rationale for AGENTS.md (a single short operative file). The factory has no equivalent single short operative file — the skills library is the operative layer, distributed across many SKILL.md files. Both docs were answering the same question with no clear boundary.
|
||||
|
||||
3. **Decision: reframe design doc as principles doc, merge agent notes.** Strip implementation content; keep and strengthen principles and rationale; absorb agent notes content inline. Name: `ai-coding-factory-principles.md` (over `FACTORY.md` — consistency with research doc naming convention).
|
||||
|
||||
4. **Content at risk identified:** The skill registry tables (trigger descriptions and key constraints for all 27 skills) are not appropriate for the principles doc and not in the research doc. Without preservation, anyone building the skills would lose carefully specified trigger language and constraint detail.
|
||||
|
||||
5. **Skills index created:** `ai-coding-factory-skills-index.md` — temporary build reference containing the full skill registry. Flagged for deletion once actual SKILL.md files exist in the repo.
|
||||
|
||||
6. **Research doc footer updated:** Reference to `ai-coding-factory-design.md` changed to `ai-coding-factory-principles.md`.
|
||||
|
||||
7. **Design doc and agent notes deleted:** Both superseded. Content is fully preserved in the principles doc and skills index.
|
||||
|
||||
---
|
||||
|
||||
## Session 3 — Deep Research and Corrections
|
||||
|
||||
### What was done
|
||||
|
||||
1. **Deep research session launched:** All 12 challenges in `ai-coding-factory-challenges.md` researched against independent sources, plus 10 additional elements identified by the research team as requiring verification. Every cited paper, URL, and quantitative claim checked.
|
||||
|
||||
2. **Verdicts assigned:** Of 12 challenges: 4 confirmed, 5 partially confirmed / precision-corrected, 3 contain material errors requiring correction. Of 10 additional elements: AORCHESTRA confirmed with qualifier; adoption count corrected (32+ → 26+); Claude Code per-session MCP filtering refuted (does not exist); tool definition bloat source corrected; SkillReducer confirmed with detail.
|
||||
|
||||
3. **Three architecturally significant corrections identified:**
|
||||
- The "3,000-token degradation ceiling" is a misreading of Levy et al. 2024 (3,000 was max tested, not a ceiling; degradation begins at ~500 tokens in that study). 2026 evidence now supersedes it.
|
||||
- The "2.74× more vulnerabilities" figure was misattributed to Veracode — it comes from CodeRabbit and applies to XSS specifically.
|
||||
- Claude Code per-session/per-role MCP filtering does not exist as of May 2026. Multiple open feature requests. Principles doc corrected.
|
||||
|
||||
4. **All corrections applied:** Research doc updated to v1.4; principles doc updated to v1.1. Session doc updated to v3.0. Challenges doc left as-is (reference artifact).
|
||||
|
||||
5. **Three governance gaps identified and documented:** Silent March 2026 prompt caching TTL regression (1h → 5min for Pro/API); MCP supply-chain threat as the primary homelab risk (not transport auth); promptfoo post-acquisition version-pin and fallback guidance.
|
||||
|
||||
### Deep research verdict summary
|
||||
|
||||
| Challenge | Verdict | Key correction |
|
||||
|---|---|---|
|
||||
| 1. SkillsBench "2–3 skills optimal" | Partially confirmed — precision error | "2–3 modules per skill," not "per task." 19% regression rate added. Self-generated skills yield no benefit. |
|
||||
| 2. 3,000-token degradation ceiling | Refuted as stated | Misread of Levy et al.; degradation starts at ~500 tokens in that study. 2026 models covered by OOLONG, arXiv 2601.15300, MRCR v2. |
|
||||
| 3. Skill trigger reliability across tools | Confirmed — gap acknowledged | All four tools use description matching; no empirical cross-tool false-positive/negative study published. |
|
||||
| 4. Veracode 2.74× vulnerability finding | Refuted attribution | 2.74× is CodeRabbit/XSS-specific. Veracode finding is 45% OWASP Top 10. Added Schreiber & Tippe 12.1% real-world rate. |
|
||||
| 5. SDD adoption — stars vs. production usage | Partially confirmed | "Mainstream" overstated; no controlled study; Kiro 250K is signups not active users; Thoughtworks full rationale added. |
|
||||
| 6. Prompt caching hit rate in coding contexts | Partially confirmed — metrics conflated | Hit rate (workflow-dependent) vs. per-token discount (90%, confirmed) are distinct. March 2026 TTL regression documented. |
|
||||
| 7. gstack productivity claims | Partially confirmed | Stars: ~96K. Claim: 810× vs part-time baseline, no independent replication. Role separation insight stands independently. |
|
||||
| 8. MCP security posture at homelab scale | Partially confirmed — enterprise framing corrected | stdio is designed for local-trust. Real threats: supply chain + prompt injection. CIS reference unverifiable. |
|
||||
| 9. OWASP AST10 applies to self-authored skills | Refuted as stated | AST10 is an Incubator draft. AST01 requires attacker-controlled content. Self-authored risks: AST03/04/06/07/09. |
|
||||
| 10. Token optimisation case study figures | Confirmed — commercial bias flagged | Alessi is the vexp developer. Figures are plausible but self-reported; not independently replicated. |
|
||||
| 11. gitea-mcp permission model | Partially confirmed — version corrected | v1.2.0 (not pre-v1.0). --tools is a capability filter, not a security boundary. 1.23 extended OAuth2 scopes. |
|
||||
| 12. Skill eval pattern transfers across tools | Confirmed in principle — gap noted | All tools use description matching; no published cross-tool false-positive/negative data. Version-pin promptfoo. |
|
||||
|
||||
---
|
||||
|
||||
## Artifact Registry
|
||||
|
||||
Current state of all artifacts as of end of Session 3.
|
||||
|
||||
| File | Version | Purpose | Status |
|
||||
|---|---|---|---|
|
||||
| `ai-coding-factory-research.md` | 1.4 | Full research document. Sourced findings, counterarguments, bias flags, provisional principles for all 10 topics. | Complete. Deep research corrections applied. |
|
||||
| `ai-coding-factory-principles.md` | 1.1 | Operative design principles derived from research. Agent-readable. | Complete. Deep research corrections applied. |
|
||||
| `ai-coding-factory-skills-index.md` | 1.0 | Temporary build reference. Trigger descriptions and key constraints for all 27 skills. Input for writing individual SKILL.md files. | **Delete once skills are written in repo.** |
|
||||
| `ai-coding-factory-challenges.md` | 1.0 | 12 structured research challenges. Verdicts documented in Session 3 of this document. | Reference only — no changes needed. |
|
||||
| `ai-coding-factory-session.md` | 3.0 | This document. Audit trail of the research, restructuring, and correction process. | Current. |
|
||||
| ~~`ai-coding-factory-design.md`~~ | ~~1.2~~ | ~~Factory design specification.~~ | **Deleted — superseded by principles doc and skills index.** |
|
||||
| ~~`ai-coding-factory-agent-notes.md`~~ | ~~1.0~~ | ~~Design rationale for the skills library.~~ | **Deleted — merged into principles doc.** |
|
||||
@@ -0,0 +1,135 @@
|
||||
# AI Coding Factory — Skills Index
|
||||
|
||||
**Purpose:** Temporary build reference. Specifies the trigger description and key constraints for every skill in the library. Use this as the direct input when writing each skill's SKILL.md description field and constraints section. Delete once the actual skill files exist in the repo.
|
||||
|
||||
**Status:** Pre-implementation reference — not a permanent artifact.
|
||||
**Source:** Derived from `ai-coding-factory-design.md` (v1.2, May 2026), superseded by `ai-coding-factory-principles.md`. Security-review skill updated in Session 3 to reflect AST10 Incubator status and correct AST01 scope.
|
||||
**Total skills:** 34 across 9 categories (7 factory meta-skills added; design/write-spec replaced by factory/write-issue-spec).
|
||||
|
||||
---
|
||||
|
||||
## Role Skills
|
||||
|
||||
| Skill | Trigger description | Primary output |
|
||||
|---|---|---|
|
||||
| `roles/architect` | Architecture review, system design decisions, data model design, ADR creation | Design doc, API contracts, tradeoff analysis |
|
||||
| `roles/developer` | Feature implementation, coding tasks, writing production code against a spec | Working code with tests following project conventions |
|
||||
| `roles/reviewer` | Code review, diff review, reviewing changes before commit or PR | Severity-rated findings, auto-fixed style issues |
|
||||
| `roles/security` | Security audit, OWASP review, STRIDE threat modelling, pre-merge security check | Security findings with remediation guidance |
|
||||
| `roles/qa` | QA, testing a feature end-to-end, finding edge cases in an implementation | Test results, regression tests |
|
||||
| `roles/ops` | Deployment tasks, infrastructure review, CI/CD configuration, runbook creation | Pipelines, manifests, runbooks, deployment plans |
|
||||
|
||||
---
|
||||
|
||||
## Design Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `design/grill-me` | Grill me on this idea, help me think through X before building, interrogate my plan | Must refuse to proceed until all decisions are explicit and all branches of the decision tree are resolved. Never code until the interview is complete. Output: structured decision summary. |
|
||||
| `design/architecture-review` | Review architecture, assess system design, evaluate technical approach | Must present ≥2 options with tradeoffs; never recommend a single option without alternatives |
|
||||
| `design/break-into-issues` | Break this into issues, decompose this spec into tasks, what issues do I need for this | Outputs independently shippable Gitea issue bodies (each can be implemented without the others); embeds full context in each issue (BMAD Scrum Master pattern); does not post issues — outputs ready-to-post bodies for human review |
|
||||
|
||||
---
|
||||
|
||||
## Factory Skills (Meta-skills — the factory builds itself)
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `factory/write-issue-spec` | Write a spec for this issue, draft the issue description for X, create a Gitea issue spec | Produces a complete Gitea issue body: Why (proposal + out-of-scope), Requirements (EARS format; delta-tagged for brownfield: ADDED/MODIFIED/REMOVED), Design notes, Tasks (independently completable checklist). Scale-adaptive: skips design notes for bug fixes; adds architecture section for complex features. Does not post the issue — outputs body for human review. |
|
||||
| `factory/write-skill` | Write a new skill for X, create a SKILL.md that does Y | Produces a complete SKILL.md following the authoring standard (frontmatter, role, when/when-not, required inputs, constraints, process, output format, failure handling). Validates trigger description against three test queries before completing. Flags if the skill overlaps with an existing skill in the library. |
|
||||
| `factory/write-workflow` | Write a workflow for X, chain these skills into a workflow, create a workflow document | Produces a `.agents/workflows/<name>.md` with: skill sequence, WorkflowContext schema (what each skill receives and returns), HITL gates marked before every irreversible action, failure paths documented. |
|
||||
| `factory/write-adr` | Write an ADR, document this architectural decision, record this decision | Produces `docs/adr/NNN-title.md` in standard format: title, date, status (Proposed/Accepted/Superseded), context, decision, consequences, alternatives considered. Increments ADR number from existing files. Never edits an existing Accepted ADR — creates a new one that supersedes it. |
|
||||
| `factory/write-eval` | Write evals for this skill, create eval.yaml for X, add tests for this skill | Produces `.agents/evals/<category>/<skill>/eval.yaml` with: ≥1 explicit trigger test, ≥1 implicit trigger test, ≥1 negative trigger test (adjacent task that must NOT activate), ≥2 output tests (deterministic: schema/contains/regex), ≥1 LLM-rubric quality test. |
|
||||
| `factory/upgrade-skill` | This skill is wrong, fix this skill, update skill X, skill X is behaving incorrectly | Updates SKILL.md body, bumps version in frontmatter, adds a new eval test case that captures the correction (so the same failure cannot recur silently). Never reduces the existing eval suite — only adds to it. |
|
||||
| `factory/validate-skill` | Check this skill, does this skill meet the standard, review this SKILL.md, audit skill X | Validates against: frontmatter completeness, trigger description quality (three-test check), constraints section presence, failure handling section, co-located eval exists. Outputs severity-rated findings: missing eval = critical; missing failure handling = high; weak trigger description = high. |
|
||||
|
||||
---
|
||||
|
||||
## Implementation Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `implement/implement-feature` | Implement a feature, build this, write the code for X | Must start from a linked Gitea issue with an EARS spec; flags if none exists; no unrequested abstractions; updates docs/spec/ as part of implementation if behaviour changes |
|
||||
| `implement/refactor` | Refactor this code, improve structure, clean up | Preserves all existing behaviour; documents what changed and why |
|
||||
| `implement/write-docs` | Write documentation, add docstrings, generate README | Derives from code and spec; does not invent behaviour |
|
||||
|
||||
---
|
||||
|
||||
## Test Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `test/write-tests` | Write tests, generate test cases, add unit tests | Derives from spec NOT implementation; pytest for Python, Vitest/Jest for TypeScript |
|
||||
| `test/generate-test-data` | Generate test data, create fixtures, sample data | Structurally valid, semantically unusual; flags PII risk |
|
||||
| `test/review-test-coverage` | Review test coverage, find untested paths, coverage gaps | Reports against spec acceptance criteria |
|
||||
|
||||
---
|
||||
|
||||
## Review Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `review/code-review` | Review this code, check this diff, pre-commit review | Severity-rated: critical/high/low; auto-fixes obvious style issues; flags architectural concerns |
|
||||
| `review/security-review` | Security review, OWASP check, pre-merge security scan | OWASP LLM Top 10 + Agentic AI Top 10 for application code; OWASP Agentic Skills Top 10 (AST10, Incubator draft) if reviewing third-party skills — for self-authored factory skills use AST03/04/06/07/09 categories, not AST01 (which requires attacker-controlled content); includes credential and licence checks |
|
||||
| `review/pr-description` | Write PR description, describe this change | Derives from diff; includes what changed, why, and what to review carefully |
|
||||
| `review/changelog-entry` | Write changelog entry, add to CHANGELOG, release notes | Conventional changelog format; derives from PR description and diff |
|
||||
|
||||
---
|
||||
|
||||
## Deploy Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `deploy/write-ci-pipeline` | Write CI pipeline, create Gitea Actions workflow | Gitea Actions YAML; includes secret scan, dependency scan, licence scan, test, build by default |
|
||||
| `deploy/write-deployment-config` | Write deployment config, Docker Compose, K8s manifest | Pinned versions; resource limits on all K8s resources; no hardcoded secrets |
|
||||
| `deploy/write-ai-review-workflow` | Create AI review workflow, automated PR review | Calls AI API via script; posts via Gitea API; never auto-merges |
|
||||
| `deploy/deployment-checklist` | Pre-deployment checklist, ready to deploy, deployment validation | Checks: linked Gitea issue exists and is closed or in-progress, secrets scan, dependency scan, licence scan, tests passing, rollback plan, docs/spec/ updated if behaviour changed; also checks review readiness — which roles (Architect, Reviewer, Security) have been invoked on this change |
|
||||
|
||||
---
|
||||
|
||||
## Operate Phase Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `operate/write-runbook` | Write runbook, operational guide, on-call playbook | Covers: common failure modes, detection, remediation steps, escalation |
|
||||
| `operate/incident-diagnosis` | Diagnose this incident, analyse these logs, root cause analysis | Produces structured finding with confidence levels; never recommends production remediation directly |
|
||||
| `operate/post-mortem` | Write post-mortem, incident review, after-action report | Blameless format; covers timeline, root cause, governance change |
|
||||
| `operate/inspect-deployment` | Check deployment health, container status, what's running | Read-only; uses Docker MCP and/or K8s MCP; summarises health |
|
||||
|
||||
---
|
||||
|
||||
## IaC Domain Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `iac/write-ansible-role` | Write an Ansible role, create Ansible automation, Ansible role for X | Idempotency review must specifically check: missing `creates:` on file-creating tasks, unconditional handlers (should use `notify`), absent rollback strategies, `shell`/`command` tasks without guards. Molecule test scaffold included; version pinned in frontmatter |
|
||||
| `iac/write-terraform-module` | Write Terraform module, Terraform config for X | Pin provider versions; never use Terraform provisioners to call Ansible — HashiCorp designates provisioners a "last resort" and they break idempotency; use a separate CI/CD Ansible stage instead; includes variables, outputs, validation blocks |
|
||||
| `iac/write-k8s-manifest` | Write Kubernetes manifest, K8s deployment for X | Resource limits required; security context required; network policy required |
|
||||
| `iac/write-docker-compose` | Write Docker Compose, compose stack for X | Pinned image versions; secrets via env vars; healthchecks included |
|
||||
| `iac/proxmox-vm-spec` | Create Proxmox VM, Proxmox config for X | Loads references/proxmox-cluster.md; bpg/proxmox provider; version pinned |
|
||||
| `iac/iac-security-review` | Security review this IaC, check Terraform/Ansible for issues | Checks: hardcoded secrets, overly permissive access, missing limits, unpinned versions, Terraform provisioners, non-idempotent Ansible patterns (shell/command without guards, missing creates:, unconditional handlers) |
|
||||
| `iac/write-molecule-test` | Write Molecule tests, Ansible testing, role test | Generates converge.yml, verify.yml, molecule.yml |
|
||||
|
||||
---
|
||||
|
||||
## Gitea Integration Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `gitea/setup-gitea-mcp` | Set up Gitea MCP, connect agent to Gitea, configure MCP | Generates config + dedicated agent user setup; includes --tools filter recommendations |
|
||||
| `gitea/post-pr-review` | Post review findings to PR, add review comments, submit review | Uses Gitea MCP; tiered severity; never auto-merges |
|
||||
| `gitea/create-issue` | Create a Gitea issue, log this as a bug, file an issue | Uses Gitea MCP; requires structured finding as input |
|
||||
|
||||
---
|
||||
|
||||
## Cross-Cutting Skills
|
||||
|
||||
| Skill | Trigger description | Key constraints |
|
||||
|---|---|---|
|
||||
| `cross-cutting/session-handoff` | Session handoff, save context, pausing work | Captures: current state, next steps, decisions with rationale, linked Gitea issue number; prompts LESSONS.md extraction; does NOT manage spec files — docs/spec/ is updated in-PR, not at handoff |
|
||||
| `cross-cutting/governance-check` | Check this against governance rules, is this allowed | Validates proposed action against AGENTS.md |
|
||||
| `cross-cutting/git-commit-message` | Write commit message, conventional commit, git message | Conventional commits format; derives from diff |
|
||||
|
||||
---
|
||||
|
||||
*Extracted and updated from `ai-coding-factory-design.md` v1.2 and `ai-coding-factory-implementation-guidance.md` v1.2, May 2026. Delete this file once all skills exist as actual SKILL.md files in the repo. The factory/ skills category is new in v1.2 — these meta-skills are Phase 2 priority 1.*
|
||||
Reference in New Issue
Block a user