holocron/docs/research/ai-coding-factory/ai-coding-factory-principles.md

AI Coding Factory — Principles

Version: 1.1 (Deep research corrections applied May 2026) Scope: Design, structure, and operation of the AI-assisted software development and infrastructure management factory Audience: Humans maintaining and extending the factory; AI agents making design decisions within it Derivation: Derived from sourced research across ten design topics. Principles are evidence-based, not aspirational. Operative files: AGENTS.md (agent governance) | CONTEXT.md (project conventions) | .agents/skills/ (skill library) Research basis: ai-coding-factory-research.md

The factory is a governed AI-assisted development environment. This document defines the principles behind its structure — why it is built the way it is and what governs changes to it. It is not an implementation guide; the implementation lives in the repository files this document governs.

---

1. Core Architecture

Tool-agnostic core, thin tool-specific adapters. Write conventions, principles, and skills once in plain language with no tool-specific references. Translate into tool-specific format (CLAUDE.md, copilot-instructions.md, cursor rules) with thin adapter files that reference the source, not duplicate it. When tools change, only the adapter changes. This is the same separation-of-concerns pattern as the governance constitution: conventions are stable; tool formats are not.

One source of truth per concern. AGENTS.md is the single source of governance rules. CONTEXT.md is the single source of project conventions. Each skill is the single source for its task type. Duplicating content into tool-specific files creates maintenance debt and drift — it is prohibited.

Spec before implementation. Every non-trivial implementation task begins with a spec. The spec is the prompt for all downstream work — it defines what the Developer role builds, what QA tests, and what the Reviewer evaluates. Without a spec the factory has no contract between its stages. Spec-Driven Development is rapidly emerging but not yet mainstream in production: GitHub Spec Kit (~92K stars, May 2026), AWS Kiro (~250K signups over its first preview period), dedicated DeepLearning.AI course. No controlled study comparing SDD with ad-hoc prompting outcomes exists as of May 2026. Even so: SDD overhead is documented and real — skip the spec skill for single-file fixes, obvious one-line changes, or tasks the human can describe completely in one sentence. GitHub's own documentation notes Spec Kit is "best suited for greenfield builds and large feature work rather than small bug fixes." (Thoughtworks Radar: "Assess, not Adopt", November 2025, with the pointed observation: "We may be relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn't scale.")

Context lives in files, not sessions. AI sessions are ephemeral; committed files are durable. Every decision, lesson, and convention that needs to survive a session boundary must be committed to a file. Claude Code auto-memory is stored per-machine and is not repo-shared — it is not a substitute for committed context files and must not be treated as one.

Agent instruction compliance is a supplementary signal, not a guarantee. Even the best models perfectly follow fewer than 30% of complex multi-constraint agentic instructions (AGENTIF benchmark, Tsinghua University 2025). Skills improve compliance; they do not guarantee it. Deterministic enforcement — CI gates, linters, pre-commit hooks, CONTROLS.md — is the primary enforcement layer. Skill instructions are a supplementary signal. Hard prohibitions in AGENTS.md must never be replicated only in a skill.

---

2. Skills Library

SKILL.md is the universal packaging unit. The Agent Skills open standard (agentskills.io, December 2025) is implemented by 26+ tools from competing vendors: Anthropic, Microsoft, OpenAI, Google, JetBrains, AWS, Block, and others. ⚠️ Earlier research cited "32+"; consistent secondary sources as of May 2026 document 26 confirmed adopters. The canonical directory is .agents/skills/. Tool-specific locations (.claude/skills/, ~/.codex/skills/) are thin adapters that reference this canonical path.

The description field is the trigger — write it first. If a skill does not activate, the problem is almost never the instructions. It is the description. Write and test the description field before writing the skill body. Test against explicit triggers, implicit triggers, and negative cases (adjacent tasks that must not activate the skill). The description-first discipline is the single most effective practice for reliable skill activation.

Organise on a phase × domain matrix. The phase axis (design, implement, test, review, deploy, operate) captures what is being done in the SDLC. The domain axis (python, typescript, ansible, terraform, docker, k8s, proxmox) captures the tool or language context. Phase-axis skills apply across domains; domain-axis skills are tool-specific. Cross-cutting skills (session-handoff, governance-check, git-commit-message) have no single phase home and live outside the matrix.

Keep each skill focused; move depth to references/. A skill with a large body and references can consume tens of thousands of tokens per invocation; cumulative cost dominates the context budget when multiple skills are active simultaneously (SkillReducer paper, arxiv.org/abs/2603.29919 — confirms a "less-is-more effect": 48% description / 39% body compression yields +2.8% quality improvement). Skill body: under 500 lines. The previous "3,000-token degradation ceiling" rationale (Levy, Jacoby, Goldberg 2024) was a misreading of that paper — 3,000 tokens was the maximum tested, not a degradation threshold. The 500-line limit is now justified by the SkillReducer "less-is-more" finding and OOLONG benchmark evidence that aggregation reasoning is <50% at 128K even for frontier models — keep skills lean for independent reasons.

Version every skill; track changes in git. Skill frontmatter carries a version field. Meaningful changes increment the version. IaC skills additionally carry explicit tool and provider version in frontmatter metadata — AI training data has a cutoff and provider APIs change; the skill must declare the version it was written for.

Never install third-party skills without reading them in full. The skills ecosystem had a documented security crisis in early 2026: 36% prompt injection rate across 3,984 public skills (Snyk ToxicSkills, February 2026); 341+ malicious skills on ClawHub delivering commodity macOS malware (ClawHavoc, February 2026); formalised as OWASP AST01 (Malicious Skills, Critical) in the OWASP Agentic Skills Top 10 Incubator project (draft, v1.0 targeted Q3 2026 — not a ratified standard). AST01 applies specifically to skills from public registries (ClawHub, skills.sh). For self-authored private factory skills, the relevant risk categories are AST03 (Over-Privileged), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance). Third-party skills use a "Markdown-as-installer" paradigm — read every file and bundled script before running anything.

Apply allowed-tools restrictions, especially for production-touching skills. Self-authored skills are not exposed to the supply chain attack vectors above, but over-privileged skills create blast radius risk. Scope each skill's tool access to the minimum it needs. Use disable-model-invocation: true for production-touching skills where appropriate.

---

3. Skill Design

Constraints are the highest-ROI element. AI models overengineer — extra files, unnecessary abstractions, unrequested scope expansion. An explicit constraints section prevents this more reliably than any other prompting technique. Every implementation skill must have one.

End every implementation skill with a self-check instruction. A self-check asking the agent to verify specific correctness criteria before responding catches a measurable proportion of errors before they reach human review. The criteria must be skill-specific, not generic.

Reference files by path; never load more than the task requires. Skills load context files on demand. Load specific files for specific tasks rather than broad codebase access. "Decisions not descriptions" framing reduces token cost and improves relevance.

Use XML tags only for complex skills. XML tags improve structure on complex, multi-section skills (500+ tokens, three or more logical sections). On simpler skills, XML adds tokens without improving accuracy. Default to prose; add XML when complexity genuinely warrants it.

Use <thinking> and named output tags for review and security skills. For skills where reasoning is auditable — code review, security audit, architecture review — emit reasoning to a thinking block and structured output to a named tag. The reasoning becomes the audit log; the structured output feeds the next workflow step.

Use EARS notation for acceptance criteria in specs. EARS (Easy Approach to Requirements Syntax, developed at Rolls-Royce for safety-critical systems) makes acceptance criteria AI-parseable and testable:

WHEN [trigger condition] AND [optional precondition] THE system SHALL [required behaviour]

Tests can be mechanically generated from EARS criteria; they cannot be generated from prose requirements. The write-spec skill must produce EARS-format acceptance criteria, not ambiguous prose.

---

4. Workflow and Agent Chains

Orchestrator-worker is the production pattern for multi-phase workflows. A single orchestrator decomposes the task; isolated subagents each run in a fresh context with a dedicated system prompt, execute one task, and return a compact summary. No peer-to-peer channels; no shared mutable state. This bounds coordination cost and prevents the quadratic communication explosion of peer architectures. (AORCHESTRA, arXiv 2602.03786: +16.28% relative improvement on GAIA, SWE-Bench, and Terminal-Bench — Gemini-3-Flash specific; transfer to Claude workflows not independently validated. The architectural pattern is independently confirmed across multiple sources.)

Define the WorkflowContext schema before writing any skill in a chain. Every skill in a multi-step workflow must know exactly what it receives and exactly what it must return. Compact structured outputs: only fields the next step needs. Every skill must handle failure explicitly — return error fields, never silent failures.

Place HITL gates before irreversible actions, not at the end of a chain. A human approval gate at the end of a long chain is not a safeguard — it is rubber-stamping. Gates go immediately before any action that is irreversible, high-blast-radius, or production-touching.

The spec is the contract between all workflow stages. Workflows without a spec drift; agents hallucinate direction and scope. The spec file is the shared reference that keeps Developer, Reviewer, QA, and Ops working toward the same target. Spec-first is both a quality practice and a token reduction strategy — a written spec eliminates repeated re-explanation of intent across sessions and agent calls.

---

5. Role Architecture

Six roles cover the factory. Each role activates a distinct cognitive mode. Roles run sequentially on demand — not concurrently.

Role	Cognitive mode	Primary output	HITL gate
Architect	Systems thinking, adversarial design	Design doc, API contracts, ADRs	Human approves before build
Developer	Implementation against plan	Code, tests, docs	Human reviews diff before commit
Reviewer	Staff-engineer perspective	Severity-rated findings	Human decides which to act on
Security	Adversarial audit (OWASP/STRIDE)	Security findings	Human approves before merge
QA	Adversarial testing	Test results, regression tests	Human reviews before PR opened
Ops	Infrastructure and deployment	Pipelines, manifests, runbooks	Human approves before any apply

Cognitive separation is the mechanism, not a convention. Different phases require different cognitive modes; planning is not review; review is not shipping. Separating plan review from code review specifically prevents the failure mode where an agent simultaneously rubber-stamps at the macro level and drowns in detail at the micro level (gstack reference implementation, ~96K stars, May 2026). Concurrent agent teams compound error rates and explode token cost — they are not the factory pattern. Note: gstack's 810× productivity claim uses a self-defined "logical LOC" metric, a part-time 2013 baseline, and confounds parallel sessions with role design — the role-separation cognitive insight stands independently of the productivity claim.

Each role's output is the next role's structured input. If a role's output is unstructured, the next role's input is ambiguous and the human checkpoint between them cannot be meaningful. The structured handoff is what makes the chain auditable, not just convenient.

---

6. SDLC Phase Principles

AI-generated code is untrusted by default and requires security-aware review. Veracode's 2025 GenAI Code Security Report found that 45% of tested AI-generated samples introduced OWASP Top 10 vulnerabilities across 100+ LLMs (4 languages, 80 coding tasks). ⚠️ The "2.74× more vulnerabilities" figure is misattributed to Veracode in many sources; it comes from CodeRabbit's December 2025 report and applies specifically to XSS vulnerabilities, not all vulnerability types. The overall CodeRabbit finding is 1.7× more issues, 1.57× more security findings. Independent academic data (Schreiber & Tippe, arXiv 2510.26103): real-world AI-generated files in GitHub repositories showed only 12.1% CWE-mapped vulnerability rate — substantially below controlled-prompt benchmark rates. All sources are vendor or vendor-adjacent; the directional finding (AI code has more vulnerabilities without review; security-aware review substantially reduces this) is independently confirmed. Source: businesswire.com/news/home/20250730694951; backslash.security (April 2026).

Generate tests from the spec, not from the implementation. Tests derived from implementation validate behaviour, which may itself be wrong — circular reasoning. Tests derived from the spec validate intent. This is a structural constraint on the write-tests skill: it requires spec input, not just code input.

Design both feedback loops explicitly. The short loop (finding → immediate fix) happens by default. The long loop (recurring pattern → standing convention) must be deliberately designed. The session-handoff skill explicitly prompts LESSONS.md extraction before closing a session. Convention graduation threshold: three or more LESSONS.md entries covering the same pattern promote to CONTEXT.md as a standing rule.

LESSONS.md is the repo-level feedback mechanism. Claude Code auto-memory is per-machine and not repo-shared. For lessons to survive machine boundaries and become team-inheritable, they must be committed to LESSONS.md in the repo root. This is a correction from common practice — auto-memory is not a substitute.

Tier review findings by severity to prevent alert fatigue. Critical: block. High: warn. Low: suggest. Alert fatigue from over-blocking is as destructive as under-blocking — it collapses the review loop into rubber-stamping, which is the failure mode it was designed to prevent.

---

7. IaC Principles

Terraform provisions; Ansible configures. Keep them separate in CI. Terraform handles Day 0 (initial provisioning and lifecycle management). Ansible handles Day 1+ (configuration of provisioned resources). Never call Ansible from Terraform provisioners — HashiCorp explicitly designates provisioners a "last resort"; they break idempotency and create state drift invisible to Terraform state. Keep them as distinct, sequential CI/CD stages.

Idempotency is the required property for Ansible — not a best practice. An Ansible task that is not idempotent is a bug. AI-generated Ansible correctly uses FQCNs and handlers but reliably misses idempotency guards. The write-ansible-role skill must explicitly review for: shell/command tasks without guards, missing creates: on file-creating tasks, unconditional handlers, and absent rollback strategies. Prefer ansible.builtin.apt, copy, and template over shell/command.

Validation is deterministic and mandatory before any apply. terraform plan, ansible --check --diff, kubectl diff are blocking validation steps, not optional review suggestions. No IaC apply runs without a prior validation pass.

K8s manifests require resource limits, security contexts, and network policies. 41% of AI-generated backend code includes overly broad permissions. Resource limits, security contexts, and network policies are routinely absent from AI-generated K8s output. The iac-security-review skill checks these explicitly; the write-k8s-manifest skill must include them by default.

Pin all versions — tools, providers, and images. No :latest Docker tags. No unpinned Terraform providers. IaC skills carry tool/provider version in frontmatter metadata. references/stack-versions.md is the authoritative version registry for the repo. The bpg/proxmox provider changes frequently and without semver guarantees — pin explicitly, reference references/proxmox-cluster.md for cluster context that AI training data cannot know.

---

8. Validation and Evals

Every skill has trigger tests and output tests. Trigger tests verify the description field activates the skill correctly — explicit invocations, implicit invocations, and negative cases (adjacent tasks that must not activate). Output tests verify the skill body produces correct, structured output. Both are required; missing trigger tests is the most common oversight and the most consequential one.

Co-locate eval.yaml with SKILL.md. Evals live alongside the skill they test. A change to any file under .agents/skills/** triggers the co-located eval run in CI. Failing evals block merge.

Deterministic assertions are the primary gate; LLM-rubric is secondary. JSON schema, contains, and regex checks are the blocking assertions. LLM-as-judge catches quality issues that deterministic checks cannot. Both are needed; neither is sufficient alone.

Every corrected skill behaviour generates a new eval case. When a human corrects an agent's skill output, that correction becomes a test case. This is the mechanism by which the eval suite grows from real failures rather than hypothetical ones.

Weekly regression evals catch model-version-induced degradation. Model updates break established skill behaviour without warning and without changelogs that map to skill-level impact. A weekly cron run of the full eval matrix detects this before it reaches production use.

Treat promptfoo with awareness of its acquisition. Promptfoo is the current standard tool (MIT, YAML-based, self-hosted, CI-native). It was acquired by OpenAI in March 2026. OpenAI has committed to keeping it open-source and multi-provider. Version-pin to a known-good release rather than tracking latest — structural conflict of interest exists for non-OpenAI evaluation. Monitor for neutrality degradation. Maintain a fallback path: DeepEval (pytest-native, zero adoption friction for Python teams) and Arize Phoenix (self-hosted, vendor-neutral) are the documented alternatives. Validate eval suites per tool — no cross-tool empirical study of false-positive/false-negative rates has been published; portability is plausible but not confirmed.

---

9. Context and Token Architecture

Orientation is the primary token sink — solve it with structure. Agents spend up to 80% of token budget on orientation rather than work. The solution is well-structured, always-available context files, not prompt compression. CONTEXT.md and the reference file architecture solve this at the source.

Progressive disclosure governs the load strategy. Only skill name and description pairs load at startup (~30–50 tokens per skill). Full skill bodies load on trigger. Reference files load only when needed during execution. 30 skills cost approximately 900–1,500 tokens at startup. The full body cost is paid only when a skill activates.

Context	Target	Load pattern
AGENTS.md	≤2,000 tokens	Always loaded
CONTEXT.md	≤200 lines	Always loaded
Per skill body	≤500 lines	On demand — skill activation
Per skill references	As needed	On demand — skill instruction
Session handoff doc	≤1,000 tokens	Compressed state per boundary

Route by task complexity — this is also a sustainability decision. Using frontier models for tasks a smaller model handles adequately imposes unnecessary environmental and infrastructure cost on others, per the governance constitution §6.

Task type	Model tier	Examples
Formatting, classification, commit messages	Lightweight (Haiku)	git-commit-message, changelog-entry, trigger eval runs
Most coding and review	Standard (Sonnet)	implement-feature, write-tests, code-review, write-ci-pipeline, write-ansible-role
Adversarial and complex	Flagship (Opus)	architecture-review, security-review (STRIDE), write-spec (ambiguous requirements), incident-diagnosis

Prompt caching is the highest single-ROI optimisation. Place static content first (AGENTS.md, CONTEXT.md); dynamic input last. The per-token discount is 90% on cache reads ($0.30/MTok vs. $3.00/MTok for Sonnet 4.6). Actual aggregate savings depend on hit rate, which varies by workflow type; peer-quality agentic data (arXiv 2601.06007) shows 41–80% cost reduction. TTL caveat: Around March 2026, Anthropic silently regressed the default cache TTL from 1 hour to 5 minutes for Pro/API users (Max subscribers retained 1-hour TTL). Factor this into strategy — intra-session reuse is reliable under 5 minutes; inter-session reuse requires Max tier or explicit API configuration for 1-hour TTL at 2× write cost.

Tokens per completed task is the sustainability metric. It simultaneously tracks cost, carbon intensity, and whether AI is doing genuine work. Per-token costs fall; aggregate consumption rises because adoption scale outpaces efficiency gains (Jevons paradox). Measure it per task, not in aggregate.

---

10. MCP Integration

Dedicated agent account per service — never use personal admin tokens. Agent operations must be attributable to a non-human actor and revocable without affecting the human account. This improves audit attribution and selective revocation. Gitea PAT scopes are category-level — a write:repository token can still delete repositories regardless of account type. The dedicated account does not improve token granularity; it is an audit-hygiene and blast-radius control.

Filter tools per skill with --tools (gitea-mcp). Tool definition bloat reaches 55K–134K tokens before optimisation (Anthropic engineering blog, November 2025). Filter the MCP tool surface to reduce token cost and communicate intended scope. Important: --tools is a client-facing capability filter, not a server-side security boundary — the underlying PAT retains full Gitea permissions regardless of --tools configuration. gitea-mcp-compact fork claims 70–80% token reduction on list/search; ⚠️ very low maturity (0 stars, 0 forks).

Ansible and Terraform execution stays human-controlled. No MCP servers exist for these tools at production-ready quality. More importantly: Ansible and Terraform execution applies changes to infrastructure — these are irreversible, high-blast-radius actions that require HITL by the governance constitution §5. Human-controlled is not a gap in the design; it is the correct design.

Ops role uses MCP for read operations; write operations require HITL. Inspection (container status, deployment health, K8s node state) can run read-only via MCP. Remediation does not run via MCP without explicit human approval.

Pin MCP server versions — gitea-mcp is at v1.2.0 as of May 2026. gitea-mcp released v1.2.0 on May 10, 2026; pin to a specific tag. Retest on upgrade — tool schema changes break skills silently.

MCP creates a distinct security boundary that needs explicit policy. "MCP does not automatically make an AI system safe." Primary homelab threats: supply chain (malicious npm/uvx packages bundling MCP servers — multiple CVEs in 2025–2026) and prompt injection from untrusted content the agent reads. Transport-level auth (OAuth 2.1) is the correct model for multi-user HTTP deployments; for single-user stdio, local-trust is the designed security model. Log per the governance constitution §7.

Claude Code does not currently support per-session or per-role MCP filtering natively. Multiple open feature requests exist (issues #7328, #4380, #45293 in anthropics/claude-code). What currently exists: three install scopes (local, project, user) and an interactive @ menu (v2.0.10+) for manual session-level tool selection. Sub-agents inherit all parent MCP tools. For programmatic per-role tool control, use the Claude Agent SDK (separate from Claude Code). Until native filtering ships, use the @ menu manually per session role.

---

Governance

This document explains why the factory is structured as it is — it does not replace building it. The principles here are the rationale for every file, skill, and workflow decision. The actual implementation lives in the repository. When an implementation decision conflicts with a principle here, update whichever is wrong — this document is not automatically correct.

This document and the governance constitution govern different things and both apply. ai-constitution.md governs AI behaviour, ethics, data handling, and security. This document governs the factory's design and structure. Neither replaces the other.

Update this document when:

• A research finding it is based on is refuted or substantially updated (the challenges document lists the highest-priority open questions)
• A new AI tool or MCP server is onboarded that changes how the factory operates
• A design decision proves wrong in practice (three or more LESSONS.md entries on the same pattern is the signal)
• At minimum: annually, aligned with the governance constitution review cycle

---

Derived from: AI Coding Factory Research Session (May 2026). Version 1.1 — deep research corrections applied. Research basis: ai-coding-factory-research.md | Open challenges: ai-coding-factory-challenges.md Session audit trail: ai-coding-factory-session.md Governance foundation: ai-constitution.md | AGENTS.md | HUMANS.md | CONTROLS.md