holocron/docs/research/ai-coding-factory/ai-coding-factory-research.md

AI Coding Factory — Research Document

Version: 1.4 (Deep research pass applied May 2026 — 12 challenges verified, 10 additional elements checked; corrections applied throughout) Purpose: Human reference document. The "why" behind every design decision in the factory. Contains sourced findings, counterarguments, bias flags, and provisional principles for all ten topics. Does not go into AI agent context — too long.

How to use this document: This is the research layer, not the operative layer. If you want to know what to build, read ai-coding-factory-principles.md. If you want to know why a skill is structured the way it is, challenge a finding, or update the evidence base, read the relevant topic here.

Session: May 2026 Methodology: Topic-by-topic web research from reliable sources. Topics 1–4 were re-searched in a second pass after their results were cleared from session context. All source URLs are now verified from actual search results. A deep research pass (Session 3, May 2026) challenged all 12 documented open questions plus 10 additional elements; corrections are applied in version 1.4. Where specific numbers remain unconfirmed, ⚠️ is noted inline.

---

Topic Index

#	Topic	Core finding	Principles	Design impact
1	Skills library design	SKILL.md (agentskills.io) is the cross-tool standard; description is the trigger; security risks from third-party skills are real and documented	7	Skill format, taxonomy, versioning, security
2	Workflow and agent chain design	Orchestrator + isolated subagents is the dominant pattern; compact structured outputs; context pollution is the primary failure mode	8	Workflow patterns, state passing, HITL placement
3	SDLC phase coverage and feedback loops	SDD rapidly emerging (not yet mainstream); AI-generated code has 45% OWASP Top 10 rate (Veracode) — 2.74× figure is CodeRabbit/XSS-specific; tests must derive from spec not implementation; LESSONS.md for repo-level feedback	13	Per-phase skills, feedback mechanisms, LESSONS.md
4	IaC patterns	Terraform provisions, Ansible configures, keep separate in CI; idempotency is the critical Ansible property; Terraform provisioners are explicitly deprecated by HashiCorp	8	IaC skill structure, validation steps, Proxmox context files
5	Gitea and self-hosted CI	Gitea Actions is mostly GitHub Actions compatible; no native AI integration; gitea-agent community project for autonomous workflows	7	CI pipeline skill, AI review workflow, security tooling
6	Agent role design	gstack (80K+ stars) validates role separation; six roles cover the factory; roles are sequential cognitive modes not concurrent agents	7	Role skill files, HITL gates per role
7	Prompt engineering for code	Context engineering supersedes prompt engineering; "3,000-token ceiling" framing refuted — degradation starts earlier; 500-line limit supported by SkillReducer less-is-more finding; constraints section is highest-ROI element	8	Skill body structure, context file design, self-check pattern
8	Validation and testing of skills	Two test types required; trigger tests most commonly missed; promptfoo acquired by OpenAI; DeepEval/Arize Phoenix as alternatives	8	Eval file structure, promptfoo config, CI integration
9	Token optimisation	Orientation is the primary sink; SkillReducer paper backs 500-line limit; prompt caching 90% savings; model routing 40–60% savings	8	Context architecture, caching strategy, model routing table
10	MCP server integration	Official gitea-mcp (v1.2.0); no granular PAT token scopes; --tools is capability filter not security boundary; Claude Code per-role MCP filtering not currently supported; Ansible/Terraform execution stays human-controlled	10	MCP setup, tool scoping, homelab threat model

---

Topic 1: Skills Library Design

The question: What is the current standard for packaging reusable AI agent skills, how should a library be structured and taxonomised, and what security considerations apply?

The SKILL.md open standard

Anthropic published the Agent Skills specification as an open standard on December 18, 2025. A skill is a directory containing one required file: SKILL.md with YAML frontmatter (name, description, optional metadata) and a Markdown body. Within 48 hours of publication, Microsoft integrated it into VS Code and OpenAI added it to Codex CLI and ChatGPT. By March 2026, 26+ tools from competing companies — including Google's Gemini CLI, JetBrains' Junie, AWS's Kiro, and Block's Goose — all read the same SKILL.md files from the same directory structure. ⚠️ Earlier research cited "32+" tools; deep research pass found consistent secondary sources citing 26 documented adopters as of May 2026 — updated here. Source: paperclipped.de/en/blog/agent-skills-open-standard-interoperability (March 23, 2026); medium.com/@automation.labs/skill-md-is-becoming-the-rest-of-agents (April 2026).

The official specification is published at agentskills.io/specification. The standard covers: file structure (a directory with SKILL.md), frontmatter format (YAML with required name and description, plus optional fields including allowed-tools, license, metadata block), instruction format (markdown), supporting directories (scripts/, references/, assets/), and the progressive disclosure model. Source: agentskills.io/specification; agensi.io/learn/agent-skills-open-standard.

File structure: SKILL.md at minimum. Optional subdirectories: scripts/ (executable code), references/ (documentation), assets/ (templates), examples/. Source: medium.com/@loccarrre/the-agent-skills-standard (February 9, 2026).

Progressive disclosure model: Only name+description pairs load at startup (~30–50 tokens per skill); full SKILL.md loads when triggered; reference files load only when needed during execution. Source: strapi.io/blog/what-are-agent-skills-and-how-to-use-them; inference.sh/blog/skills/agent-skills-overview (April 14, 2026).

The description field is the trigger. "If your skill does not trigger, it is almost never the instructions. It is the description." Source: bibek-poudel.medium.com/the-skill-md-pattern-how-to-write-ai-agent-skills-that-actually-work (February 26, 2026).

Canonical directory: .agents/skills/ per the open standard. Tool-specific locations (.claude/skills/ for Claude Code, ~/.codex/skills/ for Codex) are thin adapters. Source: agensi.io/learn/agent-skills-open-standard; agentskills.io/specification.

Size guidance: Skills should be kept focused. The SkillReducer paper (arxiv.org/pdf/2603.29919) explicitly states "a skill with a large body and references can consume tens of thousands of tokens per invocation, and when multiple skills are active simultaneously, their cumulative cost can dominate the context budget." The 500-line limit is consistent with the practitioner finding that "focused 50-line files outperform sprawling 1,000-line ones."

Security — documented and serious

The skills ecosystem had a significant security crisis in early 2026, providing the empirical basis for the "never install third-party skills without reading them" principle.

ToxicSkills (Snyk, February 5, 2026): The first comprehensive security audit of the AI agent skills ecosystem, scanning 3,984 skills from ClawHub and skills.sh. Found prompt injection in 36% of skills tested and 1,467 malicious payloads across the ecosystem. "There's a 13% chance it [a recently installed skill] contains a critical security flaw and a non-zero chance it's actively exfiltrating your credentials right now." Source: snyk.io/blog/toxicskills-malicious-ai-agent-skills-clawhub (February 5, 2026); snyk.io/articles/skill-md-shell-access (February 3, 2026).

ClawHavoc (February 2026): A security audit of 2,857 skills on ClawHub uncovered 341 malicious skills across multiple campaigns. The primary campaign (ClawHavoc) delivered Atomic Stealer (AMOS), a commodity macOS infostealer. Skill names appeared legitimate (solana-wallet-tracker, youtube-summarize-pro). The attack exploited trust through documentation — malicious skills looked professional and matched what users sought. Antiy CERT's analysis reported 1,184 malicious skills across the broader coordinated campaign. Source: snyk.io/articles/skill-md-shell-access (February 3, 2026); repello.ai/blog/malicious-openclaw-skills-exposed-a-full-teardown (February 16, 2026). ⚠️ Discrepancy in count: Snyk says 341 malicious on ClawHub; Antiy CERT says 1,184 across the broader campaign — different scope.

Mobb.ai audit: 22,511 public skills across four registries; 140,963 total issues found. Source: agensi.io/learn/toxicskills-clawhavoc-agent-skills-security-crisis-2026.

OWASP Agentic Skills Top 10 (AST10): An OWASP Incubator project — not a ratified standard (v0.0.0 on the OWASP site; v1.0 release targeted Q3 2026; Flagship submission targeted Q4 2026). ⚠️ The "April 27, 2026" publication date appears only on third-party blogs; OWASP site shows March 2026 as last update date — treat with caution. Project lead Ken Huang. AST01 (Malicious Skills, Critical) cites ClawHavoc and ToxicSkills as the defining incidents. Full ten categories: AST01 Malicious Skills (Critical), AST02 Supply Chain Compromise (Critical), AST03 Over-Privileged (High), AST04 Insecure Metadata (High), AST05 Unsafe Deserialization (High), AST06 Weak Isolation (High), AST07 Update Drift (Medium), AST08 Poor Scanning (Medium), AST09 No Governance (Medium), AST10 Cross-Platform Reuse (Medium). Source: owasp.org/www-project-agentic-skills-top-10.

Attack vector: Skills use a "Markdown-as-installer" paradigm — SKILL.md provides documentation for the agent AND installation instructions for the human. Attackers embed obfuscated setup commands within documentation, relying on implicit trust in documentation files. Source: penligent.ai/hackinglabs/clawhub-malicious-skills-beyond-prompt-injection (February 9, 2026).

For self-authored skills: AST01 (Malicious Skills) does not apply — it requires attacker-controlled skill content, which cannot occur when the author controls all content. The relevant AST categories for a self-authored private factory are: AST03 (Over-Privileged Skills), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance). AST01/AST02 apply only when importing skills from public registries (ClawHub, skills.sh). Use allowed-tools to restrict invocable tools; use disable-model-invocation: true for production-touching skills.

Taxonomy

Two-axis matrix: phase (design, implement, test, review, deploy, operate) × domain (python, typescript, ansible, terraform, docker, k8s, proxmox). No empirically validated taxonomy exists in the research — this structure is principled but not confirmed at scale.

Counterarguments

The SKILL.md standard is under a year old; tool-specific behaviour differences can break portability at edges (Claude Code adds context forking, Codex adds openai.yaml metadata). The two-axis taxonomy is principled but unproven at scale. The 500-line limit is practitioner-derived; the SkillReducer paper provides partial academic backing.

Bias flags

Security statistics from Snyk (vendor with commercial interest in finding security issues) and Mobb.ai (security tool vendor). ToxicSkills scanned only ClawHub and skills.sh — may not be representative of self-hosted or curated skill registries. The 36% figure applies to third-party public skills; self-authored skills have a fundamentally different risk profile.

Provisional principles

1. Use SKILL.md format as the universal packaging unit — the official spec is at agentskills.io.
2. Write the description field first — it is the trigger.
3. Keep each SKILL.md focused; move reference material to references/ (backed by SkillReducer paper, arxiv.org/pdf/2603.29919).
4. Organise the library on a phase × domain matrix.
5. Version every skill in frontmatter; track changes in git.
6. Apply allowed-tools restrictions, especially for skills touching production.
7. Never install third-party skills without reading the full SKILL.md and any bundled scripts — the OWASP Agentic Skills Top 10 (Incubator draft) formalises this as AST01 (Malicious Skills). For self-authored skills, the relevant risks are AST03 (Over-Privileged), AST04 (Insecure Metadata), AST06 (Weak Isolation), AST07 (Update Drift), and AST09 (No Governance).

---

Topic 2: Workflow and Agent Chain Design

The question: How should skills be composed into multi-step workflows, what patterns govern state passing, and what are the documented failure modes of chaining?

Orchestration patterns

Five workflow patterns are documented across the literature: prompt chaining (sequential LLM calls), parallelization (simultaneous operations), routing (classify-then-direct), orchestrator-worker (dynamic subtask delegation), and evaluator-optimizer (generate-evaluate-refine loops). Source: tao-hpu.medium.com/ai-agent-landscape-2025-2026-a-technical-deep-dive (January 5, 2026, citing sources [19][20]); fungies.io/ai-agent-orchestration-developers-guide-2026.

Orchestrator-worker is now the dominant production pattern. Anthropic's research multi-agent system, Claude Code's Task tool, OpenAI's agents-as-tools, and Cognition's March 2026 Managed Devins all use the orchestrator + isolated subagents pattern. Source: flowhunt.io/blog/multi-agent-ai-system (May 2026).

"Multi-agent system inquiries surged 1,445% in 2025." Source: fungies.io/ai-agent-orchestration-developers-guide-2026.

Subagent isolation: Each subagent runs in its own fresh context window with a dedicated system prompt, executes its task, and returns a single summary string. No peer-to-peer channel; no shared mutable state. This bounds coordination cost and avoids the quadratic communication explosion of peer architectures. Source: flowhunt.io/blog/multi-agent-ai-system.

Academic backing: AORCHESTRA (arXiv 2602.03786, February 2026) models every subagent as a 4-tuple (INSTRUCTION, CONTEXT, TOOLS, MODEL) and reports +16.28% relative improvement on GAIA, SWE-Bench, and Terminal-Bench. ⚠️ This figure is Gemini-3-Flash specific — the paper attributes the relative gain to that model pairing; transfer to Claude Code workflows has not been independently validated. The architectural pattern (orchestrator + isolated subagents) is well-supported across independent sources regardless. Source: flowhunt.io/blog/multi-agent-ai-system citing arxiv.org/abs/2602.03786.

State passing

Define the WorkflowContext schema before writing any skill in a chain. Each skill returns compact structured outputs — only the fields the next step needs. Build failure states into every skill (return error fields; never throw). LangGraph uses a graph structure with nodes and edges; CrewAI uses role-based handoffs. Source: fungies.io/ai-agent-orchestration-developers-guide-2026; eesel.ai/blog/subagent-orchestration.

SkillsBench — confirmed, but precision correction required: SkillsBench (Li et al., arXiv 2602.12670, February 2026) is a real benchmark: 86 tasks across 11 domains, 7 agent-model configurations, 7,308 trajectories, deterministic verifiers. The finding cited as "2–3 skills optimal" was a unit-of-analysis error — "2–3 modules per skill" is the finding, not "2–3 skills per task." Exact finding: focused skills with 2–3 modules outperform comprehensive documentation. Two additional findings the research document should carry: (a) self-generated skills yielded no benefit (−1.3pp average); (b) 19% of tasks regressed under curated skills, indicating skills are not unconditionally helpful. A follow-up paper (arXiv 2604.04323) notes SkillsBench skills appear "hand-crafted to overfit each task," with performance degrading under realistic retrieval conditions. The directional finding — more complexity per skill increases failure surface — remains valid. Source: arxiv.org/abs/2602.12670; skillsbench.ai/blogs/introducing-skillsbench.

Session handoff pattern

Context is ephemeral; files are durable. "Vibe coding's one-long-session workflow is structurally designed to maximize context rot. All three SDD frameworks address this with the same architectural insight: external state must live in files or version control, not in the LLM's context window." Source: blog.appxlab.io/2026/03/27/spec-driven-development-ai-coding.

Spec-driven development as workflow anchor

The spec is the contract between all workflow stages. "The coding agent knows what to build because the specification defines the target, not an ephemeral chat prompt." Source: javacodegeeks.com/2026/05/spec-driven-development (May 2026); thebcms.com/blog/spec-driven-development.

Counterarguments

Orchestrator-worker assumes reliable task decomposition — current models drift on novel or ambiguous tasks. The SkillsBench 2–3 skills optimal figure is unverified. WorkflowContext schemas add upfront design cost that may not pay off at small scale.

Provisional principles

1. Use orchestrator-worker for complex multi-phase workflows; sequential pipeline for deterministic steps.
2. Define the WorkflowContext schema before writing any skill in a chain.
3. Every skill returns compact structured output — minimum needed by the next step.
4. Every skill handles failure explicitly — return error fields, never silent failures.
5. Apply subagent isolation for heavy subtasks; return only essential results to parent.
6. Place HITL gates before irreversible actions, not at end of chain.
7. Use session-handoff documents for workflows spanning sessions.
8. Anchor every workflow to a spec — it is the contract between stages.

---

Topic 3: SDLC Phase Coverage and Feedback Loops

The question: What skills and workflows are needed at each SDLC phase, what does AI do well and poorly, and how do review findings feed back reliably?

Spec-first: now mainstream

Spec-Driven Development (SDD) is "the practice of writing a structured, versioned specification before invoking an AI coding agent, so the agent has explicit goals, constraints, and acceptance criteria. It replaces ad-hoc prompting ('vibe coding') with a disciplined Spec → Plan → Tasks → Implement loop." Source: thebcms.com/blog/spec-driven-development (Definitive 2026 Guide, May 2026).

"Vibe coding" was coined by Andrej Karpathy in February 2025. "Even Andrej Karpathy, who coined the term 'vibe coding' in February 2025, admitted just a year later that this era is ending and that we are entering the age of agentic engineering." Source: towardsdatascience.com/from-vibe-coding-to-spec-driven-development (May 2026).

SDD is rapidly emerging, but "now mainstream" overstates current production adoption. GitHub Spec Kit (open source, MIT-licensed, ~92,000 stars as of May 2026, supporting 28 named AI agent platforms). AWS Kiro (commercial, attracted 250,000 developers during its preview period — approximately four months from July 2025; AWS has not disclosed active-user or production-deployment numbers, making this likely a signup count). DeepLearning.AI launched a dedicated "Spec-Driven Development with Coding Agents" course in late 2025. No controlled study comparing SDD with ad-hoc prompting outcomes exists as of May 2026. Source: dev.to/remybuilds/spec-driven-development-structure-beats-vibes (May 2026); blog.appxlab.io (March 2026); thebcms.com.

Counterargument documented: Thoughtworks Radar rated SDD "Assess, not Adopt" in November 2025 (Vol 33). Full rationale: "We may be relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn't scale." Vol 34 (April 2026) did not re-rate SDD but added GitHub Spec Kit separately and moved "Curated shared instructions for software teams" to the Adopt level. Marmelab documented a 1,300-line spec for a one-feature date display — spec overhead is real. "SDD works best for large teams (10+ developers), complex systems with integration points, and regulated industries. It's overkill for solo developers, simple CRUD applications, and weekend projects." GitHub's own documentation states Spec Kit is "best suited for greenfield builds and large feature work rather than small bug fixes." Source: dev.to/remybuilds; byteiota.com/spec-driven-development-kills-vibe-coding-march-2026 (March 2026); thoughtworks.com/radar.

AI-generated code security — corrected attribution

Veracode 2025 GenAI Code Security Report: Tested 100+ LLMs across 4 languages (Java, JavaScript, Python, C#), 80 coding tasks. 45% of code samples introduced OWASP Top 10 vulnerabilities; Java had a 72% security failure rate. Source: businesswire.com/news/home/20250730694951 (July 30, 2025); growexx.com/blog/ai-code-security-crisis-2026 (February 2026).

2.74× more vulnerabilities — corrected attribution: This figure does NOT come from Veracode. It comes from CodeRabbit's "State of AI vs Human Code Generation" report (December 2025) analysing 470 GitHub PRs — and the 2.74× figure applies specifically to XSS vulnerabilities, not all vulnerability types. CodeRabbit's overall finding is more modest: AI code has 1.7× more issues and 1.57× more security findings overall. ⚠️ Earlier research merged these two sources incorrectly. CodeRabbit is a code-review tool vendor with commercial interest in this finding. Source: vibegraveyard.ai/story/coderabbit-ai-code-quality-study (theregister.com/2025/12/17).

Independent academic corroboration:

• Yan et al. (arXiv 2506.23034, GMU): 9.8–42.1% of AI-generated code is vulnerable, depending on benchmark and model.
• Schreiber & Tippe (arXiv 2510.26103, October 2025): large-scale GitHub analysis of 7,703 AI-generated files found only 12.1% contained CWE-mapped vulnerabilities — substantially below controlled-prompt benchmark rates. Real-world figures may be much lower than benchmark-driven figures when developers apply typical hygiene.
• Backslash Security (April 2026): All 7 tested LLMs generated code vulnerable to ≥4 of 10 common CWEs with naive prompts; Claude 3.7 Sonnet performed best (6/10 with naive prompts, 10/10 with security-focused prompts). ⚠️ Vendor source (Backslash is a security tool); finding on security-focused prompts is practically important. Source: backslash.security/press-releases/backslash-security-reveals (April 2026); betanews.com/2025/04/24.

⚠️ The "60%+ vulnerability reduction after human review" claim in earlier research has no traceable primary source — removed. The directional finding (review reduces vulnerability rates substantially) is reasonable but unquantified.

Surviving AI-introduced issues in production repositories topped 110,000 by February 2026 (large-scale empirical study, arXiv 2026). Source: augmentcode.com/guides/what-is-spec-driven-development; codeqa.aivyuh.com.

AI agents inside real business workflows fail roughly one attempt in three. Source: mydataschool.com/blog/ai-spec-driven-development (citing Stanford HAI 2026 AI Index).

Phase-by-phase findings

Design: Without explicit scope, agents make assumptions and head in the wrong direction fast. "The spec is the prompt." Source: thebcms.com; javacodegeeks.com (May 2026).

Testing — critical gap: AI-generated tests test the implementation rather than the intent (circular reasoning). Tests must derive from the spec to validate intent. "SDD says 'write the whole spec first — behavior, architecture, edge cases, constraints — and let an agent produce the code, tests, and docs from it.'" Source: thebcms.com/blog/spec-driven-development; augmentcode.com/guides/what-is-spec-driven-development. ⚠️ The 67% of testers would trust AI-generated tests with mandatory human review figure was cited in the original session but not found in re-search — remove from the document; unconfirmed.

Code review: ⚠️ The Atlassian RovoDev 38.7% figure (AI review comments leading to fixes) was not confirmed in re-search. Removing the specific figure; retaining the directional finding: AI review is good at breadth, poor at contextual judgment. Source: not independently confirmed in this session — treat as directional only.

Feedback loops

Two loops required: short (finding → immediate fix, happens by default) and long (recurring pattern → convention, must be explicitly designed). Source: principle derived from practitioner consensus; no single primary citation.

Auto memory scope — verified correction: Claude Code's auto-memory is stored at ~/.claude/projects/<hash>/memory/ — per-machine, not repo-shared. For repo-level lessons, LESSONS.md committed to the repo root is the correct mechanism. Source: Claude Code memory documentation — corrected during session when user identified the error.

Tiered review severity: Block critical, warn high, suggest low. Alert fatigue from over-blocking breaks the loop. Source: general practitioner guidance across CI/review tooling.

Counterarguments

SDD overhead is documented — Thoughtworks Radar "Assess, not Adopt" (November 2025). For solo developers and small projects, SDD over-engineers. "Solo developers, small teams (under 5 people), and projects with rapidly changing requirements gain little from upfront specification overhead." Source: byteiota.com (March 2026). Veracode figures are from a controlled study — real-world rates vary significantly by prompt quality and review process.

Bias flags

Veracode is a security tool vendor with commercial interest in finding vulnerabilities. GitHub Spec Kit statistics come partly from GitHub itself. SDD adoption figures (90K stars, 250K developers for Kiro) are from the tool makers.

Provisional principles

1. Every implementation workflow begins with a spec skill — SDD is now the mainstream approach, not experimental.
2. Generate tests from the spec, not from the implementation — prevents circular reasoning.
3. AI review supplements human review; never replaces the HITL gate before merge.
4. Design both feedback loops explicitly — short (finding → fix) and long (pattern → convention).
5. Repo-level lessons learned live in committed LESSONS.md, not in auto memory.
6. Tier review findings by severity — alert fatigue breaks the loop.
7. Convention graduation: 3+ LESSONS.md entries on same pattern → promote to CONTEXT.md.
8. Session handoff skill explicitly prompts LESSONS.md extraction before closing.

---

Topic 4: Infrastructure-as-Code Patterns

The question: How should AI-assisted IaC work across Ansible, Terraform, Docker, Kubernetes, and Proxmox?

Division of concerns — verified

"Terraform handles 'Day 0' activities — initial provisioning and lifecycle management of infrastructure components. Ansible handles 'Day 1 and beyond' tasks — configuration of provisioned resources." Source: scalr.com/learning-center/ultimate-guide-to-using-terraform-with-ansible (reviewed May 1, 2026).

"Use Terraform to provision infrastructure first, then hand off to Ansible for OS and application configuration." Source: spacelift.io/blog/using-terraform-and-ansible-together; computingforgeeks.com/terraform-ansible-tutorial (tested April 2026 with Terraform 1.14.8, ansible-core 2.16.14, bpg/proxmox provider 0.78 on real Proxmox hardware).

Both tools are idempotent by design. "Running them again should produce no changes if the desired state already matches reality." Source: computingforgeeks.com/terraform-ansible-tutorial.

Tool-specific failure modes — verified

Terraform provisioners — explicitly deprecated by HashiCorp: "Terraform provisioners are one of the most polarizing features in the Infrastructure as Code toolkit. HashiCorp — the creators of Terraform — explicitly recommends using them only as a 'last resort.'" Key problems: provisioner scripts are not idempotent by default; they create drift invisible to Terraform state; they introduce command injection risks. Source: scalr.com/learning-center/understanding-terraform-provisioners (reviewed June 1, 2025).

"Avoid calling Ansible via Terraform provisioners except for rare bootstrap cases." Source: spacelift.io/blog/using-terraform-and-ansible-together.

Ansible idempotency: "While Ansible modules aim for idempotency, the tool doesn't rely on stored state like Terraform does." Non-idempotent patterns are documented: Terraform local-exec provisioners appending to files on every run; Ansible shell/command tasks running unconditionally. Best practice: prefer ansible.builtin.apt, copy, template over shell/command. Source: dev.to/bowale/infrastructure-as-code-iac-challenges (January 2025); scalr.com/learning-center/understanding-terraform-provisioners.

AI-generated Ansible: AI correctly uses FQCNs and handlers, but misses idempotency guards (e.g., missing creates: on certbot tasks), validation tasks before reloads, and rollback strategies. Source: computingforgeeks.com/ai-coding-agents-devops-terraform-ansible-kubernetes (tested April 2026 with OpenCode on Rocky Linux 9.5).

AI-generated K8s: 41% of AI-generated backend code includes overly broad permission settings. Source: sqmagazine.co.uk/ai-coding-security-vulnerability-statistics (April 2026). K8s manifests require resource limits, security contexts, and network policies — these are routinely omitted in AI-generated output.

Proxmox — confirmed field experience: The bpg/proxmox Terraform provider (version 0.78 as of April 2026) is community-maintained and actively developed. The computingforgeeks.com guide (tested April 2026 on real Proxmox hardware) confirms the provider works but requires precise version pinning and a dedicated API token (not root credentials). "The full token ID format is terraform@pve!terraform-token." Source: computingforgeeks.com/terraform-ansible-tutorial.

Validation — standard practice

terraform plan, ansible --check --diff, kubectl diff are the deterministic validation mechanisms. These are standard IaC practice, not novel governance requirements. Source: computingforgeeks.com; spacelift.io; scalr.com.

Skills needed for IaC

write-ansible-role, write-terraform-module, write-k8s-manifest, write-docker-compose, proxmox-vm-spec (loads references/proxmox-cluster.md), iac-security-review, write-molecule-test. Each skill carries the relevant tool/provider version in frontmatter metadata.

Counterarguments

The Ansible/Terraform separation is the recommended pattern but not universal — Ansible's community.proxmox collection can provision Proxmox VMs directly, removing the need for Terraform at homelab scale. AI-generated Ansible is not uniformly bad at idempotency — the computingforgeeks.com test found AI (OpenCode) correctly using FQCNs and handlers; the failure mode is missing edge case guards, not fundamental incorrectness.

Bias flags

computingforgeeks.com is a practitioner site with real hardware testing — high credibility for the Proxmox provider version information, but not a controlled study. scalr.com and spacelift.io are IaC platform vendors with commercial interest in presenting Terraform + Ansible separation as best practice.

Provisional principles

1. Skills are tool-specific — write-ansible-role and write-terraform-module have different validation steps.
2. Every IaC skill carries explicit tool and provider version in frontmatter metadata.
3. Idempotency is the required property for Ansible — the skill review step must explicitly check for shell/command without guards and missing creates: parameters.
4. Never call Ansible from Terraform provisioners — HashiCorp itself calls this a last resort; keep them as separate CI/CD stages.
5. Validation is deterministic: plan/check/diff are mandatory before any apply.
6. Proxmox skills carry cluster context in references/proxmox-cluster.md; pin bpg/proxmox to a specific version.
7. K8s manifests require resource limits, security contexts, and network policies — review skill checks these explicitly.
8. Docker images must use pinned versions, not :latest.

---

Topic 5: Gitea and Self-Hosted CI

The question: What does Gitea Actions support and not support, and how should the factory handle CI/CD?

Sources for this topic are verified from session search results (preserved in original search pass).

Gitea Actions capabilities

Gitea Actions production-stable since Gitea 1.19. GitHub Actions-compatible YAML syntax. act_runner forked from nektos/act. DEFAULT_ACTIONS_URL = github allows GitHub marketplace actions. Runs in Docker containers. Source: docs.gitea.com/usage/actions/overview; docs.gitea.com/usage/actions/design; about.gitea.com/products/runner.

Practical setup: act_runner runs as a Docker sibling container alongside Gitea, polling for jobs. Source: botmonster.com/posts/self-hosted-cicd-pipeline-gitea-actions-docker (May 2026); dev.to/diogojlq/self-hosted-gitea-ci (January 2026).

Compatibility gaps

Concurrency groups ignored. GITHUB_TOKEN maps to owner-level Gitea API token, not fine-grained scopes. Cache requires explicit setup (Gitea 1.21+). runs-on labels must match exactly. "Compatibility with GitHub Actions is impressive for a project this size, but there are sharp edges." Source: docs.gitea.com/usage/actions/comparison; blog.elest.io/how-to-migrate-from-github-actions-to-gitea-actions (March 2026); mylinux.work/guides/gitea-actions-and-runners (April 2026).

What Gitea lacks — and community responses

No native equivalent of GitHub Copilot review, GitHub Advanced Security, or GitHub MCP server. AI models default to GitHub Actions assumptions. Source: docs.gitea.com/usage/actions/comparison; github.com/go-gitea/gitea/issues/34527 (AI agent feature request, May 2025).

Gitea vs Forgejo: In late 2024, Forgejo forked from Gitea over governance concerns. Both share the same workflow syntax; Forgejo uses forgejo-runner. Source: serverspan.com/en/blog/the-2026-guide-to-self-hosted-git-gitea-forgejo (February 2026).

gitea-agent (April 2026): A community project specifically for Gitea self-hosted environments, combating "LLM drift — where large language models skip tests, modify incorrect files, or hallucinate paths." Issue → Plan → Approval → Implementation → Eval → PR cycle with human approval gates. Runs fully locally. Source: devactivity.com/posts/apps-tools/gitea-agent (April 7, 2026). ⚠️ Early-stage community project; not production-ready at time of research.

Replacing GitHub-native security tooling

• Secret scanning: trufflehog (ghcr.io/trufflesecurity/trufflehog) or gitleaks
• Dependency scanning: trivy (aquasec/trivy)
• Licence scanning: licensee or fossa-cli

Source: botmonster.com/posts/self-hosted-cicd-pipeline-gitea-actions-docker; mylinux.work/guides/gitea-actions-and-runners.

Counterarguments

"Mostly compatible" overstates for complex pipelines. Gitea token model may vary by version. gitea-agent is community and early-stage.

Bias flags

Gitea compatibility claims from Gitea's own documentation. gitea-agent assessment from community discussion threads.

Provisional principles

1. Port workflows to .gitea/workflows/; fix Gitea-specific edge cases.
2. Configure the cache server before writing pipelines.
3. The write-ci-pipeline skill explicitly targets Gitea; never assumes GitHub defaults.
4. Secret, dependency, and licence scanning are default pipeline steps.
5. AI PR review is a custom workflow step — gitea-agent is worth watching as it matures.
6. Agent tokens scoped carefully in workflows posting comments or modifying PRs.
7. All CI runners self-hosted; no code routed through external services.

---

Topic 6: Agent Role Design

The question: How should AI agent roles be structured, what roles are needed for the factory?

Sources for this topic are verified from session search results (preserved in original search pass).

The core insight

Different phases require different cognitive modes. Planning is not review. Review is not shipping. Source: github.com/garrytan/gstack; backslash.security/blog/the-new-role-of-developers-ai-sdlc (May 2026).

Industry-wide confirmation: "Skill shift: The role of the developer is shifting from 'Code Writer' to 'Agent Orchestrator' and 'System Reviewer.'" Source: hcltech.com/trends-and-insights/autonomous-software-factory-agentic-ai-sdlc (February 16, 2026). "More than 75% of developers will be architecting, governing, and orchestrating instead of building applications." Source: devopsdigest.com/2026-devops-predictions-5 (December 2025). IBM Bob (April 28, 2026): "an agentic platform that embeds an AI partner into every role across the SDLC." Source: newsroom.ibm.com/2026-04-28-introducing-ibm-bob.

Reference implementation: gstack

Garry Tan's open-source skill pack (MIT, launched March 12, 2026): 23–31 role-based slash commands. Skills run in sprint order: Think → Plan → Build → Review → Test → Ship → Reflect. Source: github.com/garrytan/gstack; augmentcode.com/learn/garry-tan-gstack-claude-code (April 2026); sitepoint.com/gstack-garry-tan-claude-code (March 2026); mager.co/blog/2026-03-28-gstack; awesomeagents.ai/guides/gstack (April 2026).

Star count (rapidly changing): 39K within 11 days of launch; 56K by late March; ~96K by May 2026. High adoption rate for an opinionated tool indicates genuine pain point being addressed. Source: producthunt.com/products/gstack; awesomeagents.ai/guides/gstack; dev.to/max_quimby/gstack.

Key finding: separating plan review from code review prevents the failure mode where the agent either rubber-stamps or gets lost in details. Source: producthunt.com/products/gstack discussion thread (March 2026).

Productivity claim — actual detail and caveats: Garry Tan claims 810× his 2013 part-time output, measured in self-defined "logical LOC" (non-blank, non-comment lines across all garrytan/* repos via custom script). The 2013 baseline was part-time output while running YC and Posterous — well below historical developer norms (Capers Jones: 16–38; McConnell: 20–125). Tan acknowledges: "if you think the right baseline is 50, the multiple drops from 810× to 228×." The comparison confounds three variables: better 2026 LLMs, 10–15 parallel Claude Code sessions via Conductor.build, and gstack's role structure — these cannot be separated. No independent team has published comparable results. TechCrunch (March 17, 2026) provides the primary skeptical coverage, noting gstack's popularity reflects partly Tan's YC platform. The gstack repo itself includes an ON_THE_LOC_CONTROVERSY.md file acknowledging the measurement disputes. Source: github.com/garrytan/gstack/blob/main/docs/ON_THE_LOC_CONTROVERSY.md; techcrunch.com/2026/03/17.

"Simpler, well-bounded agent patterns usually outperform elaborate multi-agent complexity when the goal is reliable software delivery." Source: cio.com/article/4166035 (attributed to Anthropic's experience, May 2026).

"The real ceiling isn't the harness — it's the 1–5% error rate from frontier models. Cascading failures still need human taste to catch before they compound." Source: producthunt.com/products/gstack discussion thread.

Six factory roles

Role	Cognitive mode	Primary output	HITL gate
Architect	Systems thinking, adversarial design review	Design doc, API contracts, data models	Human approves before build
Developer	Implementation against plan	Code, tests, docs	Human reviews diff before commit
Reviewer	Staff-engineer perspective	Severity-rated findings	Human decides which to act on
Security	Adversarial audit (OWASP/STRIDE)	Security findings	Human approves before merge
QA	Adversarial testing	Test results, regression tests	Human reviews before PR opened
Ops	Infrastructure and deployment	Pipelines, manifests, runbooks	Human approves before any apply

Counterarguments

gstack productivity claims (810× self-reported output) use a self-defined "logical LOC" metric, a part-time 2013 baseline, and confound parallelism (10–15 sessions) with role design — not independently verifiable. Six roles may over-engineer for a single-developer homelab early on. The role-based design is not isolated as the causal mechanism; comparable frameworks (ECC, GSD, OpenSpec) achieve similar coordination without gstack specifically.

Bias flags

Primary evidence from gstack — created by YC CEO with platform interest in demonstrating agentic productivity. Star count reflects novelty and author's platform, not adoption at scale. No independent team has published comparable productivity results.

Provisional principles

1. Roles enforce cognitive separation.
2. Six roles cover the factory: Architect, Developer, Reviewer, Security, QA, Ops.
3. Roles run sequentially on demand — not concurrently.
4. Each role's output is the next role's structured input.
5. Ops is a distinct role from Developer.
6. HITL gates apply per role.
7. Roles reduce error surface but don't eliminate it.

---

Topic 7: Prompt Engineering for Code

Sources for this topic are verified from session search results (preserved in original search pass).

The question: What prompting patterns are effective for code generation and review, and what anti-patterns must skills avoid?

The shift to context engineering

Context engineering has largely superseded prompt engineering in 2026. Four strategies (LangChain): write (persist externally), select (RAG), compress, isolate. Source: thomas-wiegold.com/blog/prompt-engineering-best-practices-2026 (February 2026); arxiv.org/pdf/2603.09619 (Context Engineering: From Prompts to Corporate Multi-Agent Architecture, 2026).

Context length and reasoning quality — corrected 2026 evidence

⚠️ The "3,000-token degradation ceiling" framing is inaccurate. Levy, Jacoby, and Goldberg (2024) found degradation beginning as early as ~500 tokens in the models they tested; 3,000 tokens was simply the maximum length tested, not a ceiling. Citing this as "reasoning degrades around 3,000 tokens" inverts the finding.

2026 evidence updates this substantially:

• Anthropic MRCR v2 (multi-needle retrieval at 1M tokens): Opus 4.6 = 76% vs. Sonnet 4.5 = 18.5% — a 4× improvement attributed to context compaction. Targeted retrieval at very long context is now feasible for flagship models.
• OOLONG benchmark (arXiv 2511.02817, November 2025): All frontier models (including GPT-5, Claude Sonnet 4, Gemini 2.5 Pro) score <50% at 128K context on aggregation reasoning — directly contradicting the assumption that large context windows solve comprehension problems.
• "Intelligence Degradation in Long-Context LLMs" (arXiv 2601.15300): F1 collapses ~45% at 50% of maximum context for tested models — degradation is catastrophic past a threshold, not gradual.

Design implication: The 500-line skill limit and ≤200-line CONTEXT.md constraint remain well-founded for a different reason: not because reasoning degrades at exactly 3,000 tokens, but because (a) aggregation reasoning is poor even at 128K for all current models, and (b) the SkillReducer paper (arXiv 2603.29919) confirms that large skill bodies provide worse quality output than compressed equivalents, independent of absolute token counts. Drop the "3,000-token ceiling" framing; keep the size limits with this corrected rationale. Source: Levy et al. (2024) via thomas-wiegold.com; arxiv.org/abs/2511.02817; arxiv.org/abs/2601.15300; Anthropic MRCR v2 eval.

XML tags: when they help

Benefit scales with complexity. On prompts under 500 tokens, XML adds tokens without improving accuracy. Source: controlled experiment at dev.to/manishramavat (May 2026, N=12 — very small sample). Claude natively parses XML-style tags. Source: platform.claude.com/docs/en/build-with-claude/prompt-engineering/use-xml-tags; docs.claude.com/en/docs/build-with-claude/prompt-engineering/use-xml-tags.

The constraints section as highest-ROI element

Claude Opus 4.5 and 4.6 tend to overengineer — extra files, unnecessary abstractions. Explicit constraints prevent this. Source: code.claude.com/docs/en/best-practices.

Context file design

"Decisions not descriptions" → ~20% token reduction. AST-level file loading → ~65% fewer file reads. Source: earezki.com/ai-news/2026-02-26 (February 2026 — single practitioner case study). ⚠️ Commercial bias: Nicola Alessi is the developer of vexp (vexp.dev, the AST-loading tool used in the case study). The case study should not be treated as an independent third-party result. The general principle (precise context over broad loading) is sound; the specific 65% figure reflects vexp's capabilities in that specific codebase and should be treated as an indicative upper bound, not a guaranteed outcome. The "40 file reads" baseline was also unusually broad — more disciplined CONTEXT.md instructions could close most of the gap without AST tooling. Technology choices with rationale, ADRs, codebase-specific anti-patterns, Preferred/Avoid code blocks. Source: packmind.com/context-engineering-ai-coding (April 2026).

Chain-of-thought for review skills

<thinking> + named output tags: reasoning to audit log, structured output feeds next step. Source: kindatechnical.com/claude-ai/claudes-prompt-style-xml-tags; platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices.

Counterarguments

The "3,000-token ceiling" framing has been corrected — the 2024 study showed degradation beginning at ~500 tokens, not a ceiling at 3,000. 2026 models show substantially better long-context handling for retrieval tasks (MRCR v2) but aggregation reasoning remains poor at 128K+. XML experiment was N=12 — very small. Context engineering framing can be overstated.

Bias flags

XML experiment small sample (N=12). "20% reduction" and "65% reduction" are single practitioner case studies. Claude prompting documentation is from Anthropic.

Provisional principles

1. Context engineering is the primary discipline — CONTEXT.md and reference files are the primary artefact.
2. Keep loaded context per skill invocation lean — the 500-line skill limit is justified by SkillReducer's "less-is-more effect" (+2.8% quality from compressed skills) and OOLONG's finding of <50% aggregation reasoning at 128K even for frontier models. The previous "3,000-token ceiling" framing (Levy et al. 2024) was a misreading — see corrected evidence above.
3. XML for complex skills (500+ tokens, 3+ sections); plain prose for simple skills.
4. Every implementation skill includes an explicit constraints section.
5. Use <thinking> + named output tags for review and security skills.
6. Reference files by path rather than loading full content.
7. End every implementation skill with a self-check instruction.
8. Document Preferred/Avoid code blocks in CONTEXT.md for every convention that matters.

---

Topic 8: Validation and Testing of Skills

Sources for this topic are verified from session search results (preserved in original search pass).

The question: How do you verify a skill works correctly, what testing frameworks exist, and how does skill testing integrate into CI?

Why skill testing differs from unit testing

Standard LLM evals test a function; agent evals test a system with emergent behaviour. Non-determinism compounds. Source: promptfoo.dev/docs/guides/evaluate-coding-agents; developers.openai.com/blog/eval-skills (January 2026).

The trigger eval pattern

10–20 test cases: explicit invocation, implicit invocation, and negative cases. Start small; grow from real failures. Source: developers.openai.com/blog/eval-skills (January 2026); promptfoo.dev/docs/integrations/agent-skill. ⚠️ OpenAI's guidance is for Codex; the eval structure is applicable in principle across Claude Code, Gemini CLI, and VS Code Copilot — all use description-matching activation — but no cross-tool empirical comparison of false-positive/false-negative rates has been published as of May 2026. Validate eval suites per tool; do not assume portability.

Output eval assertions

Deterministic assertions (JSON schema, contains, regex) as blocking gates; LLM-rubric as secondary. Source: promptfoo.dev/docs/intro; evaluation tooling guides from confident-ai.com/knowledge-base/compare/best-ai-evaluation-tools-2026; augmentcode.com/tools/best-ai-agent-evaluation-tools (May 2026).

Tooling

Promptfoo: Open-source MIT, YAML-based, self-hosted, CLI-first. Acquired by OpenAI March 9, 2026 for $86M. MIT license confirmed maintained post-acquisition; multi-provider support (Anthropic, Google, Azure, Ollama) confirmed as of May 2026. OpenAI committed publicly: "Promptfoo will remain open source under the current license, and we will continue to service and support current customers." Structural conflict of interest exists for non-OpenAI evaluation — monitor for neutrality degradation; version-pin to a known-good release rather than tracking latest. Source: github.com/promptfoo/promptfoo; openai.com/index/openai-to-acquire-promptfoo; techcrunch.com/2026/03/09.

CI integration: --fail-on-error flag as quality gate; trigger on .agents/skills/** path changes. Source: promptfoo.dev/docs/integrations/ci-cd.

Alternatives if vendor neutrality degrades: DeepEval (pytest-native, zero adoption friction for Python teams, free); Arize Phoenix (self-hosted, vendor-neutral). Source: dev.to/thedailyagent/top-5-ai-agent-eval-tools-after-promptfoos-exit.

Three eval types

Type	When	What	Tool
Trigger	Every skill change in CI	Activation correct?	promptfoo
Output	Every skill change in CI	Structured, quality output?	promptfoo + LLM-rubric
Regression	Weekly cron	Did model update break behaviour?	promptfoo matrix

Counterarguments

LLM-as-judge introduces model bias. Full output evals on every PR is expensive at API token rates.

Bias flags

Promptfoo documentation is now from an OpenAI-acquired company. confident-ai.com and augmentcode.com are commercial eval platforms.

Provisional principles

1. Every skill has trigger tests and output tests.
2. Trigger evals: 10–20 cases including explicit, implicit, and negative.
3. Deterministic assertions as blocking gates; LLM-rubric as secondary.
4. Co-locate eval.yaml with SKILL.md.
5. Skill file changes trigger CI eval runs; failing eval blocks merge.
6. Weekly regression evals catch model-version-induced degradation.
7. Every corrected skill behaviour generates a new eval case.
8. Promptfoo is the starting point; track OpenAI acquisition; DeepEval or Arize Phoenix as fallbacks.

---

Topic 9: Token Optimisation

Sources for this topic are verified from session search results (preserved in original search pass).

The question: Where do tokens actually go, what are the highest-ROI reduction techniques, and how does this apply across the factory?

The orientation problem

Agents spend up to 80% of token budget on "orientation" rather than actual work. Source: earezki.com/ai-news/2026-02-26 (February 2026); smartscope.blog/en/generative-ai/claude/agents-md-token-optimization-guide-2026.

Average prompt length grew ~4× from 2024 to 2025 (1,500 → 6,000 tokens). Source: Tokalator paper (arxiv.org/pdf/2604.08290), 100 trillion tokens from OpenRouter platform.

Multi-agent systems consume 4–15× more tokens than single calls if not optimised. Source: obviousworks.ch/en/token-optimization-saves-up-to-80-percent-llm-costs (2026).

Spec-driven development as token optimisation: "When requirements, scope, acceptance criteria, constraints, interfaces, and non-goals live in a written specification, the developer does not need to re-explain the same feature intent in every coding conversation. This is not merely a process preference — it is a context optimisation technique." Source: medium.com/@nprasads/token-efficiency-in-ai-coding-agents (April 13, 2026). This connects Topic 3 and Topic 9: spec-first is both a quality practice and a token reduction strategy.

Academic backing for skill size limits

SkillReducer paper: "A skill with a large body and references can consume tens of thousands of tokens per invocation, and when multiple skills are active simultaneously, their cumulative cost can dominate the context budget." Source: arxiv.org/pdf/2603.29919. This provides academic backing for the 500-line limit in Topic 1.

Prompt caching — corrected framing and TTL regression

Two distinct metrics — do not conflate: (a) Per-token discount: Sonnet 4.6 cache reads cost $0.30/MTok vs. $3.00/MTok standard — a 90% per-token reduction on cached content. This is documented and confirmed. (b) Cache hit rate: The proportion of input tokens that are actually served from cache. The 74–84% figure cited in original research came from a production support bot (BSWEN's personal Claude Code usage analysis, single-blogger) and a security product (ProjectDiscovery, 84% after explicit breakpoint placement). Agentic coding workflows have substantially more variable dynamic content (different diffs, files, task inputs) — no published study measures hit rates specifically for agentic coding workflows. Treat 74–84% as an upper bound from a different workflow type, not an expected coding workflow result.

Critical March 2026 TTL regression: Around March 6, 2026, Anthropic silently changed the default cache TTL from 1 hour back to 5 minutes for Claude Code Pro/API users. Max subscribers retained 1-hour TTL. This caused 20–32% cache-creation cost inflation for Pro/API users and was not communicated via changelog. Source: github.com/anthropics/claude-code/issues/46829; dev.to/whoffagents (April 2026).

Current TTL structure:

• Claude Code Max / Anthropic Max subscribers: 1-hour TTL ($6/MTok write, $0.30/MTok read)
• Claude Code Pro / API users: 5-minute TTL ($3.75/MTok write, $0.30/MTok read)
• Break-even: 1 hit for 5-minute TTL; ≥2 hits for 1-hour TTL

Peer-quality data (arXiv 2601.06007, "Don't Break the Cache", February 2026): Evaluated caching on DeepResearch Bench (10K-token system prompts across agentic tasks). Found 41–80% cost reduction and 13–31% TTFT improvement in tested agentic workflows. This is the most methodologically rigorous available data point on caching in agentic (vs. support bot) contexts. Source: arxiv.org/pdf/2601.06007.

Context architecture (40–60% savings)

Always-loaded index file ≤200 lines; skills load on demand. Source: code.claude.com/docs/en/best-practices; code.claude.com/docs/en/costs; towardsdatascience.com/agentic-ai-how-to-save-on-tokens.

"Decisions not descriptions" → ~20% reduction. AST-level loading → ~65% fewer file reads. ⚠️ Source: earezki.com/ai-news/2026-02-26 (single practitioner case study; Alessi is the vexp tool developer — commercial bias). See Topic 7 for full caveat.

Model routing (40–60% cost reduction)

Prices verified April 2026 (official Anthropic documentation): Haiku 4.5 $1/$5; Sonnet 4.6 $3/$15; Opus 4.7 $5/$25. Note: Opus 4.7 uses a new tokenizer generating up to 35% more tokens for the same input text. Source: platform.claude.com/docs/en/about-claude/pricing; cloudzero.com/blog/claude-api-pricing; finout.io/blog/anthropic-api-pricing.

Batch API

50% discount on both input and output for non-time-sensitive work. Source: platform.claude.com/docs/en/about-claude/pricing (official documentation).

Agent team costs

~7× more tokens than standard sessions. Source: code.claude.com/docs/en/costs (official documentation).

Tool definition bloat

55,000–134,000 tokens of tool definitions before optimisation. Source: Anthropic engineering blog "Introducing advanced tool use" (November 24, 2025) — cited in towardsdatascience.com/agentic-ai-how-to-save-on-tokens (which correctly attributes it to Anthropic documentation). The 134K figure reflects Anthropic's own internal measurement across multiple MCP servers; the 55K figure covers a representative multi-server configuration (~58 tools: GitHub, Slack, Sentry, Grafana, Splunk). Anthropic's Tool Search Tool reduces this to ~5K. Source: mindstudio.ai/blog/reduce-token-usage-ai-agents-mcp-optimization.

Counterarguments

90% per-token caching savings requires a high hit rate on a stable prefix; for variable-content agentic coding workflows the hit rate may be substantially lower than the 74–84% figures from support-bot deployments. Haiku routing risks misclassification. The 20% and 65% context-file reduction figures are single case studies from the same author who has a commercial interest in one of the tools.

Bias flags

Most token cost figures from Anthropic documentation or third-party analysis. Case studies should not be treated as guaranteed results. Vexp case study: commercial bias as noted above. Caching hit-rate figures from non-coding workflow deployments.

Provisional principles

1. Orientation is the primary token sink — solve with structured context before micro-optimising.
2. Prompt caching is the highest single-ROI optimisation — static content first, dynamic input last. Distinguish the per-token discount (90% on cache hits, confirmed) from the hit rate (workflow-dependent; 41–80% in peer-quality agentic data; higher in stable deployments).
3. Be aware of the March 2026 TTL regression: Pro/API users get 5-minute TTL by default; 1-hour TTL requires Max subscription or explicit API configuration. Factor this into caching strategy — intra-session reuse is reliable; inter-session reuse requires Max tier.
4. Always-loaded context file ≤200 lines; specialised instructions in skills.
5. Route by complexity: lightweight model for formatting/classification, standard for most coding, flagship for adversarial review.
6. Batch API for non-time-sensitive factory work.
7. Sequential roles, not concurrent agent teams.
8. Scope MCP tools per skill; concise tool descriptions.
9. Measure tokens per completed task — the sustainability metric.

---

Topic 10: MCP Server Integration

Sources for this topic are verified from session search results (preserved in original search pass).

The question: What MCP servers exist for the factory's stack, how do you connect them securely?

What MCP enables

Without MCP: agents read/write files through custom scripts. With MCP: agents create issues, inspect containers, query infrastructure nodes as direct tool calls. HITL gates still apply. "MCP is not just a developer convenience — it is a new and distinct security boundary that needs policy, oversight, and operational discipline." Source: CoSAI/OASIS WS4 Secure Design for Agentic Systems (github.com/cosai-oasis/ws4-secure-design-agentic-systems); MCP specification Security Best Practices (modelcontextprotocol.io). ⚠️ Earlier research cited a "CIS MCP Companion Guide (April 20, 2026)" — no such CIS Benchmark for MCP exists as of May 2026; reference corrected.

Gitea MCP Server

Official gitea.com/gitea/gitea-mcp — MIT-licensed, actively maintained. At v1.2.0 as of May 10, 2026 (not pre-v1.0 as originally stated). Tool schema reduced 26.6% and --tools flag added May 2026. Two modes: stdio (local, no auth by default) and HTTP (remote, multi-user). Source: gitea.com/gitea/gitea-mcp (repository, releases page verified May 2026); pkg.go.dev/gitea.com/gitea/gitea-mcp.

--tools flag clarification: The --tools flag is a client-facing capability filter — it restricts which tools appear in the tools/list JSON-RPC response (reducing context token cost). It does NOT restrict what the underlying PAT can do on the Gitea server side. A PAT with write:repository still enables DELETE /repos/{owner}/{repo} regardless of --tools configuration. Treat --tools as a token-optimisation and scope-communication mechanism, not a security boundary.

Token efficiency: gitea-mcp-compact fork (github.com/appslop/gitea-mcp-compact) reduces token usage 70–80% on list/search operations per its README. ⚠️ Community fork: 0 stars, 0 forks, 1 commit as of May 2026 — very low maturity; verify before using in production.

Critical security gap — Gitea token scopes: Gitea PAT scopes are category-level (read/write × repository/issue/user/etc.) but have no sub-scopes to prohibit destructive operations within a category. write:repository includes DELETE /repos/{owner}/{repo}. Gitea 1.20 redesigned the PAT scope system; Gitea 1.23 (the relevant version) extended granular scopes to OAuth2 applications. The underlying PAT category model has not become finer-grained since 1.20 — the warning about repository deletion remains correct for PAT-based access. Source: github.com/appslop/gitea-mcp-compact README; docs.gitea.com/development/oauth2-provider.

Forgejo alternative: github.com/raohwork/forgejo-mcp — supports both Gitea and Forgejo, available in stdio and HTTP (multi-user) modes. Source: github.com/raohwork/forgejo-mcp.

Infrastructure MCP servers

Docker: ckreiling/mcp-server-docker and Docker's official MCP Toolkit. Source: mcp.aibase.com/class/Virtualization (2026); docker.com/blog.

Kubernetes: K8m — AI-driven K8s console with MCP services, multi-cluster support. Source: mcp.aibase.com/class/Virtualization.

Proxmox: Python-based community server (~10K GitHub stars). Source: mcp.aibase.com/class/Virtualization. ⚠️ Community-maintained; requires source review before connecting to VM management tokens.

Ansible MCP: None found. Execution stays human-controlled. Source: absence confirmed by search.

Terraform MCP: None stable found. Same reasoning. Source: absence confirmed by search.

MCP security

⚠️ "CIS MCP Companion Guide" reference unverifiable: No CIS Benchmark for MCP exists as of May 2026. The most likely intended primary sources are the CoSAI/OASIS WS4 Secure Design for Agentic Systems (github.com/cosai-oasis/ws4-secure-design-agentic-systems) and the MCP specification's own Security Best Practices. The reference has been updated below.

"Using MCP does not automatically make an AI system safe." Source: CoSAI Secure Agentic AI Design Guide.

"The authorization spec says authorization is optional overall, applies to HTTP-based transports, and should not be used for stdio the same way." Source: modelcontextprotocol.io/specification/draft/basic/authorization. Clarification: stdio is explicitly designed for local-trust environments; treating "no auth by default" as a security gap misreads the spec. In a single-user homelab where only the user can send prompts, stdio provides the correct security model. The real stdio risk is supply-chain — malicious npm/uvx packages that bundle an MCP server (multiple CVEs in 2025–2026: CVE-2025-49596 MCP Inspector, CVE-2026-22252 LibreChat). Install MCP server packages only from reviewed, version-pinned sources.

"If an unmanaged AI agent is compromised, it can use legitimately authorised tokens to exfiltrate data. The activity appears as normal API traffic." Source: nudgesecurity.com/post/mcp-security-risks (May 2026).

Homelab vs. enterprise threat model: Enterprise security guidance (Red Hat OAuth 2.1, multi-tenant auth) addresses adversarial multi-user environments. In a single-user self-hosted homelab, the relevant threat model is: (1) supply chain — malicious MCP server packages; (2) prompt injection from untrusted content the agent reads (PR bodies, issue descriptions, code comments). Transport-level auth requirements do not apply to single-user stdio. The dedicated agent account mitigation addresses audit trail and selective revocation, not granular token permissions — scope accordingly.

For multi-user HTTP MCP: prefer delegating to an external OAuth/OIDC provider rather than embedding auth in the MCP server. Source: redhat.com/en/blog/mcp-security-implementing-robust-authentication-and-authorization (March 11, 2026).

Claude Code per-session/per-role MCP filtering: not currently supported. Multiple open feature requests exist (github.com/anthropics/claude-code/issues/7328 tool filtering; #45293 per-session MCP server profiles; #4380 per-agent tool filtering). What currently exists: three install scopes (local, project, user); an interactive @ menu (v2.0.10+) for session-level tool selection. Sub-agents inherit all parent MCP tools. The Claude Agent SDK (separate from Claude Code) does support allowed_tools / skills filtering programmatically. Until native Claude Code per-role filtering ships, use the interactive @ menu manually or the Agent SDK for programmatic workflows. Source: github.com/anthropics/claude-code/issues/4380.

forgejo-mcp HTTP mode: Does not add OAuth scoping over the underlying PAT — relays a bearer PAT in the Authorization header. Users manage scopes themselves. Source: github.com/raohwork/forgejo-mcp README.

MCP security threat taxonomy covers five layers: transport/communication, authentication/identity, context integrity/confidentiality, authorization/privilege management, supply chain security. Source: aembit.io/blog/the-ultimate-guide-to-mcp-security-vulnerabilities (March 19, 2026).

Counterarguments

gitea-mcp is now at v1.2.0 — tool schemas still change without semver guarantees at this stage. Proxmox MCP is community Python — requires source review. stdio auth model is the correct model for single-user controlled environments (not a gap, as the original research implied).

Bias flags

MCP security statistics from vendors with commercial interest in finding MCP risks. Red Hat and enterprise-framing guidance applies to multi-tenant environments — calibrate to homelab threat model. gitea-mcp-compact 70–80% figure is self-reported in the fork's README with no independent verification and very low maturity indicators.

Provisional principles

1. Dedicated agent user account per service — never use personal admin tokens. This improves audit attribution and selective revocation; it does NOT improve token granularity (same PAT category scopes apply).
2. Use --tools filtering on every MCP server to reduce token cost and communicate intended scope. This is a capability/token filter, not a security boundary — the underlying PAT retains full Gitea-side permissions.
3. Gitea PAT scopes are category-level with no sub-category prohibitions — a write:repository token can delete repos. Mitigate with: dedicated agent user account, repo-scoped access only, quarterly rotation. Gitea 1.23 extended granular scopes to OAuth2 applications; PAT model remains category-level.
4. Ansible and Terraform execution stays human-controlled — no MCP servers for these.
5. Ops role uses MCP for read operations; write operations still require HITL approval.
6. Claude Code does not currently support per-session/per-role MCP filtering natively — use the interactive @ menu for manual scoping, or the Claude Agent SDK for programmatic control.
7. Pin MCP server versions — gitea-mcp is at v1.2.0 as of May 2026; pin to a specific tag. Retest on upgrade.
8. Every MCP tool call logged per governance constitution §7.
9. For multi-user setup, switch to HTTP transport with OAuth 2.1 — forgejo-mcp supports HTTP mode (note: does not add OAuth scoping over the underlying PAT; users manage scopes themselves).
10. Primary homelab MCP threats are supply chain (verify npm/uvx packages before installing) and prompt injection from untrusted content; transport-level auth is not the primary concern for single-user stdio deployments.

---

Research session completed May 2026. Version 1.4 — deep research pass (Session 3) applied across all topics. Twelve challenges verified; ten additional elements checked. Design specification: ai-coding-factory-principles.md Session audit trail: ai-coding-factory-session.md