holocron/docs/research/ai-coding-factory/ai-coding-factory-session.md

AI Coding Factory — Session Audit Trail

Purpose: Auditability of the artifact creation process. Documents what was done, how, why, and what decisions were made or deferred. Not a task list — a process record.

Project: AI coding skills, workflows, and templates research for building a personal AI coding factory spanning software development and infrastructure management. Sessions: Three sessions, May 2026. Artifacts produced: See artifact registry below.

---

Project Overview

The goal was to produce a research foundation and governance principles before building an AI-assisted software development and infrastructure management setup. The author wanted practices grounded in current research — not opinion — that would govern skill design, workflow structure, and agent behaviour across a homelab and development context.

The work was deliberately sequenced: research first, then distil into principles. The research must be independently challengeable; the principles must be applicable at both homelab and team scale.

---

Methodology

Research approach

• Web search from reliable sources per topic. Training data avoided as a source; sourced findings preferred throughout. Whether this was fully achieved cannot be independently verified — some analytical and synthesis sections may reflect training data rather than sourced research, particularly where citations are sparse.
• Research conducted topic by topic so findings can be challenged before they harden into design decisions.
• Provisional design decisions established during scoping were treated as hypotheses to be tested, not assumptions to be confirmed.
• Bias flags reference limitations within the sourced research itself (vendor-funded studies, single-study findings, practitioner observation rather than controlled research) — not the AI assistant's training data. However, the line between "synthesising sourced findings" and "drawing on training data" is not always clear, and some analytical conclusions may have training data influence that went undetected.

Output format per topic

Each topic in the research document follows: question being researched → findings (with sources) → counterarguments → bias flags → provisional principles.

Distillation logic

Research document = full sourced reasoning (human reference, never in agent context). Principles document = operative design principles derived from research (agent-readable, repo artifact). Skills index = temporary pre-implementation build reference for skill trigger descriptions and constraints. Session document = this file — audit trail of the creation process.

Standing integrity caveat

The instruction to use web sources was given and followed to the best of the session's ability. However, AI-generated research cannot self-audit for training data influence with certainty. Analytical framing, the choice of which findings to emphasise, and the synthesis of sources into conclusions all carry some risk of undetected training data influence. Where citations are absent or thin, treat findings with additional scepticism and verify independently before relying on them for high-stakes decisions.

---

Scope Decisions

These decisions were made during the initial scoping interview before research began. Each shaped all subsequent work.

Decision	Conclusion	Rationale
Output form	Research doc + principles doc + skills index (temporary)	Separates reasoning from operative principles; skills index preserves implementation-level detail not appropriate for principles doc
Audience	Solo now, team-inheritable later	Every design choice must be legible without the author present
Tool approach	Tool-agnostic with thin adapters	Will be used across Claude Code, Copilot, Cursor, and others; cannot be tool-locked
Stack scope	Python + TypeScript (app), Ansible, Terraform (future), Docker, Proxmox, K8s (infra)	Reflects actual current and near-future tooling
SDLC scope	Full cycle — design through operations	Factory that only covers implementation is incomplete
Research depth	Sourced, with tradeoffs	Design decisions are only as strong as the evidence behind them
Gitea assumption	Self-hosted, Gitea Actions, no GitHub	Changes CI skill design significantly; established before research began
Research sequencing	Topic by topic, user challenges before proceeding	Prevents hypotheses hardening before they are tested

---

Topics Researched

#	Topic	Key finding	Status
1	Skills library design	SKILL.md open standard published Dec 2025; description field is the trigger; security risks real even for self-authored skills	Complete
2	Workflow and agent chain design	WorkflowContext schema before any skill; compact structured outputs; context pollution is the primary failure mode	Complete
3	SDLC phase coverage	Spec-first prevents drift; AI-generated tests must derive from spec not implementation; circular reasoning is a documented failure mode	Complete
3b	Feedback loops	Two loops required (short + long); auto memory is machine-local not repo; LESSONS.md must be repo-committed	Complete — merged with Topic 3 in research doc
4	IaC patterns	Provider version drift is the primary AI IaC failure mode; Ansible idempotency must be explicit; Proxmox has thin training data coverage	Complete
5	Gitea and self-hosted CI	Mostly GitHub Actions compatible; AI defaults to GitHub assumptions; no native AI integration; must build AI review as custom workflow	Complete
6	Agent role design	Cognitive separation by role outperforms generalist; gstack is the reference implementation; six roles cover the factory	Complete
7	Prompt engineering for code	Context engineering superseded prompt engineering; 3,000-token ceiling framing corrected (see Session 3); constraints section is highest-ROI element	Complete — corrected in v1.4
8	Validation and testing of skills	Two test types required; trigger tests most commonly missing; promptfoo acquired by OpenAI March 2026 — track vendor neutrality; version-pin	Complete
9	Token optimisation	Added by user request; orientation problem is the primary sink; prompt caching per-token savings confirmed; TTL regression documented (March 2026); model routing 40–60% savings	Complete — corrected in v1.4
10	MCP server integration	Added per recommendation; official gitea-mcp (v1.2.0 as of May 2026); no granular Gitea PAT token scopes; Claude Code per-role MCP filtering not supported natively; Ansible/Terraform execution stays human-controlled	Complete — corrected in v1.4

---

Key Decisions Log

Decision	Why
SKILL.md as the packaging format	Closest thing to a universal standard across Claude Code, Codex CLI, Gemini CLI, Copilot, Cursor. Published by Anthropic Dec 2025, MIT-licensed.
.agents/skills/ as canonical directory	Open standard path. Tool-specific directories (.claude/skills/, etc.) are thin adapters. Single source of truth per governance pattern.
AGENTS.md as primary operative file	Consistent with existing governance foundation. Tool adapters reference it; content not duplicated per tool.
Spec-first as mandatory workflow step	Research confirmed vibe coding as the documented failure mode. Spec is the contract between all workflow stages.
LESSONS.md committed to repo	Auto memory (~/.claude/projects/.../memory/) is machine-local, not shared. Repo-level lessons require an explicit committed file.
Two feedback loops explicitly designed	Only the short loop (finding → fix) happens by default. The long loop (pattern → convention) must be deliberately built.
Gitea MCP dedicated agent user	Gitea PAT scopes are category-level; a write:repository token can still delete repos regardless of account type. Dedicated account improves audit attribution and selective revocation — it does not improve token granularity. Corrected framing in Session 3.
Ansible/Terraform: no MCP execution	Both make infrastructure changes; human-controlled execution aligns with the governance constitution. Agents generate; humans apply.
Sequential roles not concurrent agents	Agent teams use ~7× more tokens. Solo homelab context; parallelism overhead not justified.
Six roles	Covers the full SDLC with distinct cognitive modes. Based on gstack (~96K stars, May 2026) as the primary reference implementation. The 810× productivity claim uses a self-defined metric and part-time baseline — the cognitive-separation insight stands independently.
Token routing: Haiku/Sonnet/Opus	40–60% cost reduction from proper routing documented. Haiku for formatting; Sonnet for most coding; Opus for architecture and adversarial review.
Topic 9 (token optimisation) added	User request during research. Justified by existing governance constitution's sustainability metric: tokens per unit of value.
Topic 10 (MCP integration) added	Recommended based on gap analysis. Without MCP research, the factory would produce agents that generate code about infrastructure but cannot act on it.
Topics 3 and 3b merged in research doc	Feedback loops are part of the SDLC cycle, not a separate topic. Editorial decision to improve document coherence.
Design doc reframed as principles doc	Original design doc was positioned as a blueprint/implementation guide — templates, CI YAML, build order, repo structure diagrams. The appropriate role is operative design principles, analogous to the governance constitution. Implementation artifacts belong in the repo, not in a documentation artifact.
Agent notes merged into principles	The notes were created by analogy with the governance project's ai-agent-instructions-notes.md. But the factory has no single short operative file equivalent to AGENTS.md — the notes' rationale purpose collapses into the principles doc. Two documents answering the same question with no clear boundary between them.
Skills index as temporary artifact	Skill trigger descriptions and key constraints (27 skills) are not appropriate for the principles doc (too implementation-specific) and not in the research doc (too specific). Preserved as a pre-implementation build reference. Delete once actual SKILL.md files exist in the repo.
Name: ai-coding-factory-principles.md	Preferred over FACTORY.md. Consistent with research doc naming convention; explicit about content (principles, not configuration).

---

Corrections and Integrity Notes

What was wrong	When caught	How corrected
MEMORY.md described as a repo-level shared file	During Topic 3b	User caught it. Corrected: MEMORY.md / auto memory is machine-local, per ~/.claude/projects/<hash>/memory/. LESSONS.md (repo-committed) is the correct mechanism for shared lessons. Design document updated.
Design document used Claude Code-specific paths and syntax	Post-artifact review	User caught it: .claude/skills/, CLAUDE.md, @filename syntax, ccusage are Claude Code-specific. Corrected: design document updated to use .agents/skills/, tool-agnostic context file references, generic adapter pattern.
Research document lacked source citations	Post-artifact review	User caught it. Research doc updated to add source URLs per topic (v1.1), then source audit applied (v1.2): Topics 1–4 flagged ⚠️ UNVERIFIED URL because search results were cleared from context; Topics 5–10 checked against preserved search results and missing citations added.
Topics 1–4 re-searched (v1.3)	User-requested re-search pass	All four topics re-searched with new web searches. Key findings added: Agent Skills spec confirmed at agentskills.io (December 18, 2025); ToxicSkills security figures now sourced to snyk.io/blog (February 5, 2026), ClawHavoc to snyk.io/articles (February 3, 2026), Mobb.ai 22,511 skills audit to agensi.io, OWASP Agentic Skills Top 10 to owasp.org (April 27, 2026); SDD now confirmed mainstream with verified sources (thebcms.com, augmentcode.com, growexx.com); Veracode 2025 GenAI Code Security Report confirmed as primary AI code vulnerability benchmark (2.74× more vulnerabilities, 45% OWASP Top 10 rate); Terraform provisioner deprecation confirmed from scalr.com and spacelift.io; bpg/proxmox provider version 0.78 confirmed from computingforgeeks.com tested on real Proxmox hardware April 2026. The 67% testers figure and Atlassian RovoDev 38.7% figure were NOT found in re-search — removed from research doc.
Multiple found-but-not-cited sources	Source audit (v1.2)	Sources found in preserved search results (Topics 5–10) but not included in v1.1: gitea-agent (devactivity.com); serverspan.com 2026 self-hosted git guide; HCLTech autonomous software factory; IBM Bob; DevOps Digest 2026 predictions; gstack additional sources (mager.co, awesomeagents.ai); SkillReducer paper (arxiv.org/pdf/2603.29919) — most significant miss, provides academic backing for 500-line skill limit; medium.com/@nprasads spec-driven-as-token-optimisation; forgejo-mcp alternative; redhat.com MCP OAuth 2.1 guidance; aembit.io MCP security taxonomy; promptfoo agent-skill integration docs; eval tooling comparison sources. All added in v1.2.
Design document not updated after research re-search (v1.2)	Post-v1.3-research review	Five skill registry entries updated based on finding changes: write-spec now assesses whether spec overhead is warranted before proceeding; security-review now distinguishes OWASP LLM Top 10 / Agentic AI Top 10 / Agentic Skills Top 10; write-ansible-role now names specific idempotency failure modes; write-terraform-module now cites HashiCorp "last resort" authority for provisioner prohibition; iac-security-review now includes non-idempotent Ansible patterns.
Session audit trail not created initially	Post-artifact review	User caught it. This document created as correction.
SkillsBench "2–3 skills optimal" figure unverified	Session 2 review	Removed from session topics table. Flagged ⚠️ unverified in research doc. Not incorporated into principles doc.
SkillsBench unit-of-analysis error	Session 3 deep research	Confirmed as a real benchmark (arXiv 2602.12670) but the finding was misread: "2–3 modules per skill" not "2–3 skills per task." Also: 19% of tasks regressed under curated skills; self-generated skills yielded no benefit. Research doc updated in Topic 2.
"3,000-token degradation ceiling" framing	Session 3 deep research	Misread of Levy et al. (2024): 3,000 tokens was the maximum tested, not a ceiling; degradation begins at ~500 tokens in that study. 2026 evidence (OOLONG, arXiv 2601.15300, Anthropic MRCR v2) supersedes the 2024 study. Research doc Topic 7 rewritten; principles doc skill-limit rationale updated to SkillReducer "less-is-more" finding.
"Veracode 2.74× more vulnerabilities" misattribution	Session 3 deep research	2.74× figure comes from CodeRabbit (December 2025), not Veracode, and applies specifically to XSS vulnerabilities. Veracode's actual finding is 45% OWASP Top 10 rate. Added Schreiber & Tippe (arXiv 2510.26103): 12.1% real-world CWE rate. Research doc Topic 3 corrected; principles doc Section 6 corrected.
SDD "now mainstream"	Session 3 deep research	Overstated. No controlled study of SDD vs ad-hoc prompting outcomes exists. Kiro 250K figure is signups over ~4 months, not active users. Thoughtworks full rationale added. Updated to "rapidly emerging." Research doc Topic 3 and principles doc Section 1 corrected.
"CIS MCP Companion Guide" reference	Session 3 deep research	No CIS Benchmark for MCP exists as of May 2026. Reference replaced with CoSAI/OASIS WS4 and MCP specification security guidance.
stdio MCP framed as a security gap	Session 3 deep research	Misframed. MCP spec explicitly designates stdio as the local-trust transport; "no auth by default" is the correct design for single-user environments. Real homelab threats are supply chain and prompt injection. Research doc Topic 10 and principles doc Section 10 corrected.
Dedicated agent account framed as token-granularity control	Session 3 deep research	Corrected: dedicated account improves audit attribution and selective revocation; it does not improve token granularity (same PAT category scopes apply). Research doc Topic 10 and principles doc Section 10 corrected.
AST10 described as a published OWASP standard	Session 3 deep research	AST10 is an OWASP Incubator project, not a ratified standard (v0.0.0 on OWASP site; v1.0 targeted Q3 2026). "April 27, 2026" publication date appears only on third-party blogs. AST01 corrected to apply only to third-party registry skills; self-authored factory risks are AST03/04/06/07/09. Research doc Topic 1 and principles doc Section 2 corrected.
gstack "80K+ stars" and "10× productivity"	Session 3 deep research	Stars updated to ~96K (May 2026). Actual claim is 810× vs a part-time 2013 baseline — uncontrolled, confounds parallelism with role design, no independent replication. Research doc Topic 6 corrected; principles doc Section 5 corrected.
"32+ tools" adopted Agent Skills standard	Session 3 deep research	Corrected to "26+" — consistent secondary sources as of May 2026 document 26 confirmed adopters. Research doc Topic 1 and principles doc Section 2 corrected.
gitea-mcp described as pre-v1.0	Session 3 deep research	gitea-mcp is at v1.2.0 as of May 10, 2026 — not pre-v1.0. Research doc Topic 10 and principles doc Section 10 corrected.
--tools flag framed as a security boundary	Session 3 deep research	--tools is a client-facing capability filter (token-optimisation, scope communication). The underlying PAT retains full Gitea-side permissions regardless. Research doc Topic 10 and principles doc Section 10 corrected.
Claude Code per-session/per-role MCP filtering	Session 3 deep research	This capability does not exist in Claude Code as of May 2026. Multiple open feature requests. Available workarounds: interactive @ menu, Claude Agent SDK for programmatic control. Research doc Topic 10 and principles doc Section 10 corrected.
Prompt caching TTL	Session 3 deep research	March 2026 silent regression: Anthropic changed default TTL from 1 hour back to 5 minutes for Pro/API users (Max retains 1-hour). This was not communicated via changelog. Research doc Topic 9 and principles doc Section 9 corrected.
Prompt caching hit rate vs. per-token discount conflated	Session 3 deep research	Two distinct metrics treated as one. Hit rate is workflow-dependent (peer data: 41–80% in agentic tasks, arXiv 2601.06007). Per-token discount is 90% on cache reads (confirmed). Research doc Topic 9 and principles doc Section 9 clarified.
Alessi case study figures cited without commercial bias flag	Session 3 deep research	Nicola Alessi is the developer of vexp, the AST-loading tool used in the "40 → 5 file reads" case study. Both figures (20% and 65%) are self-reported by someone with commercial interest in one of the techniques. Research doc Topic 7 and Topic 9 updated with bias flag.
Tool definition bloat source cited as Towards Data Science	Session 3 deep research	Primary source is the Anthropic engineering blog "Introducing advanced tool use" (November 24, 2025). Towards Data Science correctly attributes it to Anthropic but is a secondary source. Research doc Topic 9 corrected.
AORCHESTRA +16.28% improvement presented as unconditional	Session 3 deep research	+16.28% is relative improvement and Gemini-3-Flash specific — not a general claim across models. Research doc Topic 2 and principles doc Section 4 updated with qualifier.
forgejo-mcp HTTP mode adds OAuth scoping	Session 3 deep research	Refuted: forgejo-mcp HTTP mode relays a bearer PAT in the Authorization header; users manage scopes themselves. Research doc Topic 10 corrected.

---

Session 1 — Research and Initial Artifact Creation

What was done

1. Scoping session: Established purpose, audience, methodology, and scope decisions documented above. Output: shared understanding of what to build and why.
2. Topic-by-topic research: Ten governance topics researched in sequence (with Topic 3b on feedback loops merged into Topic 3). Each topic was web-searched, synthesised, and reviewed before proceeding.
3. Source corrections: Multiple rounds of source verification and citation fixes (v1.1 → v1.2 → v1.3). Topics 1–4 fully re-searched after search results were cleared from context.
4. Artifact distillation: Research document → design specification → agent notes → session doc.
5. Post-artifact corrections: Tool-agnostic path fixes, session audit trail created, skill registry entries updated to reflect re-search findings.

---

Session 2 — Artifact Restructuring

What was done

1. Design doc repositioning identified: The design doc (v1.2) was structured as a blueprint — templates, CI YAML, build order, repo structure diagrams. The appropriate role for a permanent artifact is operative principles (what to do and why), not implementation scaffolding (how to implement it). Implementation artifacts belong in the repo.

2. Agent notes overlap identified: The agent notes were created by analogy with the governance project but serve a different need. In the governance project, the notes explain the rationale for AGENTS.md (a single short operative file). The factory has no equivalent single short operative file — the skills library is the operative layer, distributed across many SKILL.md files. Both docs were answering the same question with no clear boundary.

3. Decision: reframe design doc as principles doc, merge agent notes. Strip implementation content; keep and strengthen principles and rationale; absorb agent notes content inline. Name: ai-coding-factory-principles.md (over FACTORY.md — consistency with research doc naming convention).

4. Content at risk identified: The skill registry tables (trigger descriptions and key constraints for all 27 skills) are not appropriate for the principles doc and not in the research doc. Without preservation, anyone building the skills would lose carefully specified trigger language and constraint detail.

5. Skills index created: ai-coding-factory-skills-index.md — temporary build reference containing the full skill registry. Flagged for deletion once actual SKILL.md files exist in the repo.

6. Research doc footer updated: Reference to ai-coding-factory-design.md changed to ai-coding-factory-principles.md.

7. Design doc and agent notes deleted: Both superseded. Content is fully preserved in the principles doc and skills index.

---

Session 3 — Deep Research and Corrections

What was done

1. Deep research session launched: All 12 challenges in ai-coding-factory-challenges.md researched against independent sources, plus 10 additional elements identified by the research team as requiring verification. Every cited paper, URL, and quantitative claim checked.

2. Verdicts assigned: Of 12 challenges: 4 confirmed, 5 partially confirmed / precision-corrected, 3 contain material errors requiring correction. Of 10 additional elements: AORCHESTRA confirmed with qualifier; adoption count corrected (32+ → 26+); Claude Code per-session MCP filtering refuted (does not exist); tool definition bloat source corrected; SkillReducer confirmed with detail.

3. Three architecturally significant corrections identified:

   • The "3,000-token degradation ceiling" is a misreading of Levy et al. 2024 (3,000 was max tested, not a ceiling; degradation begins at ~500 tokens in that study). 2026 evidence now supersedes it.
   • The "2.74× more vulnerabilities" figure was misattributed to Veracode — it comes from CodeRabbit and applies to XSS specifically.
   • Claude Code per-session/per-role MCP filtering does not exist as of May 2026. Multiple open feature requests. Principles doc corrected.

4. All corrections applied: Research doc updated to v1.4; principles doc updated to v1.1. Session doc updated to v3.0. Challenges doc left as-is (reference artifact).

5. Three governance gaps identified and documented: Silent March 2026 prompt caching TTL regression (1h → 5min for Pro/API); MCP supply-chain threat as the primary homelab risk (not transport auth); promptfoo post-acquisition version-pin and fallback guidance.

Deep research verdict summary

Challenge	Verdict	Key correction
1. SkillsBench "2–3 skills optimal"	Partially confirmed — precision error	"2–3 modules per skill," not "per task." 19% regression rate added. Self-generated skills yield no benefit.
2. 3,000-token degradation ceiling	Refuted as stated	Misread of Levy et al.; degradation starts at ~500 tokens in that study. 2026 models covered by OOLONG, arXiv 2601.15300, MRCR v2.
3. Skill trigger reliability across tools	Confirmed — gap acknowledged	All four tools use description matching; no empirical cross-tool false-positive/negative study published.
4. Veracode 2.74× vulnerability finding	Refuted attribution	2.74× is CodeRabbit/XSS-specific. Veracode finding is 45% OWASP Top 10. Added Schreiber & Tippe 12.1% real-world rate.
5. SDD adoption — stars vs. production usage	Partially confirmed	"Mainstream" overstated; no controlled study; Kiro 250K is signups not active users; Thoughtworks full rationale added.
6. Prompt caching hit rate in coding contexts	Partially confirmed — metrics conflated	Hit rate (workflow-dependent) vs. per-token discount (90%, confirmed) are distinct. March 2026 TTL regression documented.
7. gstack productivity claims	Partially confirmed	Stars: ~96K. Claim: 810× vs part-time baseline, no independent replication. Role separation insight stands independently.
8. MCP security posture at homelab scale	Partially confirmed — enterprise framing corrected	stdio is designed for local-trust. Real threats: supply chain + prompt injection. CIS reference unverifiable.
9. OWASP AST10 applies to self-authored skills	Refuted as stated	AST10 is an Incubator draft. AST01 requires attacker-controlled content. Self-authored risks: AST03/04/06/07/09.
10. Token optimisation case study figures	Confirmed — commercial bias flagged	Alessi is the vexp developer. Figures are plausible but self-reported; not independently replicated.
11. gitea-mcp permission model	Partially confirmed — version corrected	v1.2.0 (not pre-v1.0). --tools is a capability filter, not a security boundary. 1.23 extended OAuth2 scopes.
12. Skill eval pattern transfers across tools	Confirmed in principle — gap noted	All tools use description matching; no published cross-tool false-positive/negative data. Version-pin promptfoo.

---

Artifact Registry

Current state of all artifacts as of end of Session 3.

File	Version	Purpose	Status
ai-coding-factory-research.md	1.4	Full research document. Sourced findings, counterarguments, bias flags, provisional principles for all 10 topics.	Complete. Deep research corrections applied.
ai-coding-factory-principles.md	1.1	Operative design principles derived from research. Agent-readable.	Complete. Deep research corrections applied.
ai-coding-factory-skills-index.md	1.0	Temporary build reference. Trigger descriptions and key constraints for all 27 skills. Input for writing individual SKILL.md files.	Delete once skills are written in repo.
ai-coding-factory-challenges.md	1.0	12 structured research challenges. Verdicts documented in Session 3 of this document.	Reference only — no changes needed.
ai-coding-factory-session.md	3.0	This document. Audit trail of the research, restructuring, and correction process.	Current.
ai-coding-factory-design.md	1.2	Factory design specification.	Deleted — superseded by principles doc and skills index.
ai-coding-factory-agent-notes.md	1.0	Design rationale for the skills library.	Deleted — merged into principles doc.