holocron/docs/research/ai-coding-factory/ai-coding-factory-challenges.md

AI Coding Factory — Open Questions & Research Challenges

Purpose: Agenda for the deep research session. Each item identifies a finding, assumption, or claim from the initial factory research that warrants deeper investigation, challenge, or independent quantification. Ordered by design impact — highest-stakes uncertainties first.

How to use: Upload alongside ai-coding-factory-research.md and ai-coding-factory-session.md to continue in a fresh session. Each challenge should be web-searched against independent sources before accepting or correcting the current position. Any finding refuted by the deep research should be corrected in all affected artifacts.

---

1. SkillsBench "2–3 skills optimal" — does this benchmark exist?

Current position: The research cites SkillsBench (2026) as finding the optimal range for skills per task is 2–3, with more skills per step increasing failure surface.

What to challenge:

• Is SkillsBench a real published benchmark or a practitioner label? No primary source URL was confirmed in either the original research session or the re-search pass.
• If it exists, what is the methodology — controlled experiment, user study, or practitioner survey?
• Is the 2–3 figure specific to agent skill invocations or to general multi-step LLM chaining?
• Does the finding generalise across tools (Claude Code, Codex, Gemini CLI) or is it tool-specific?

Design implication: The 2–3 skill optimal range influences workflow chain design. If unverified, the principle has no empirical basis and should be removed or replaced.

---

2. The 3,000-token degradation ceiling — does it hold for 2026 models?

Current position: LLM reasoning performance starts degrading around 3,000 tokens of context. This is the primary justification for the 500-line skill limit, the CONTEXT.md ≤200 line constraint, and most of the token optimisation strategy.

What to challenge:

• The source is Levy, Jacoby, and Goldberg (2024) — a study on models that predate Claude Sonnet 4.6 and Opus 4.7 by at least one generation. Do 2026 models show the same degradation at the same threshold?
• Has Anthropic or any independent lab published updated context length vs. reasoning quality data for Claude 4-series models?
• The SkillReducer paper (arxiv.org/pdf/2603.29919) documents context budget problems but does not validate the 3,000-token specific threshold. Is there a more recent primary source?
• Does the degradation manifest equally for structured code tasks vs. natural language tasks?

Design implication: If the threshold is actually higher for 2026 models, the 500-line skill limit and CONTEXT.md constraint may be over-conservative. If lower, they should tighten.

---

3. Skill trigger reliability across tools — does the description field work equally well everywhere?

Current position: The description field is the trigger — the agent uses it to decide whether a given request activates the skill. The research treats this as a universal cross-tool mechanism.

What to challenge:

• Do Claude Code, Codex CLI, Gemini CLI, and GitHub Copilot implement skill activation identically, or does each tool use different matching logic (semantic similarity, keyword matching, LLM classification)?
• What is the empirically observed false-positive and false-negative activation rate per tool for a typical skill description?
• Does the negative case in trigger evals actually prevent activation reliably, or do tools still fire on adjacent prompts?
• The OpenAI eval guidance (developers.openai.com/blog/eval-skills) is specifically for Codex. Does the 10–20 test case structure produce reliable coverage for Claude Code specifically?

Design implication: If trigger reliability differs significantly across tools, the tool-agnostic factory design may require tool-specific description tuning — undermining the single-source-of-truth principle.

---

4. Veracode vulnerable code finding — independent validation

Current position: Veracode 2025 GenAI Code Security Report found AI-generated code has 2.74× more vulnerabilities than human-written code, with 45% of code samples introducing OWASP Top 10 vulnerabilities across 100+ LLMs and 4 languages.

What to challenge:

• Veracode is a security tool vendor with commercial interest in finding vulnerabilities. Do independent academic benchmarks confirm the 2.74× figure and 45% OWASP rate?
• The Yan et al. (2025) range (9.8–42.1%) is from a different study. How do these reconcile — different metrics, different populations?
• Does the vulnerability rate change significantly when AI-generated code is reviewed by a human before commit? The research implies review closes the gap but does not quantify by how much.
• The Backslash Security finding (all seven tested LLMs generated code vulnerable to at least 4 of 10 common CWEs with "naive" prompts) — is this replicable with security-focused prompts?

Design implication: The security-review skill is positioned as mandatory pre-merge because of the vulnerability rate evidence. If review substantially closes the gap, the framing remains correct; if the rate with review is much lower, the mandatory-gate urgency may need calibration.

---

5. SDD adoption claims — stars vs. actual production usage

Current position: GitHub Spec Kit has ~90,000 stars (April 2026). AWS Kiro attracted 250,000 developers in its first three months. Spec-Driven Development is described as "now mainstream."

What to challenge:

• GitHub stars measure visibility, not usage. What evidence exists that teams are shipping production software with SDD workflows rather than experimenting briefly?
• The 250,000 figure for AWS Kiro — is this active users, signups, or downloads? AWS has a strong incentive to publicise adoption figures favourably.
• Thoughtworks Radar rated SDD "Assess, not Adopt" in November 2025. What is their full rationale?
• Does SDD overhead scale with team size such that it's genuinely counterproductive for solo developers? The counterargument evidence suggests yes — what does the evidence say for a solo homelab context specifically?

Design implication: The write-spec skill is the factory's highest-priority foundation skill. If SDD's evidence base is thinner than presented for solo use, a lighter-weight spec approach may be appropriate and the build order priority may need adjusting.

---

6. Prompt caching hit rate — achievable in coding contexts?

Current position: A 74–84% cache hit rate is achievable on "stable agent workloads," yielding 90% savings on cached input tokens.

What to challenge:

• The 74–84% figure is from production support bot and stable document processing workflows (Vellum and Helicone reports via tokenmix.ai). Coding workflows have significantly more variable dynamic content — different diffs, files, and task inputs per session. Is this hit rate achievable for the factory's actual use patterns?
• How does the 5-minute Sonnet TTL interact with typical development sessions where a developer pauses between tool calls? If sessions regularly exceed the TTL, cache savings may be substantially lower.
• Is there published data on prompt caching performance specifically in agentic coding workflows?
• What is the cache write overhead at low hit rates — does it add net cost at low reuse patterns?

Design implication: Prompt caching is the single highest-ROI optimisation recommended. If actual hit rates for coding workflows are 20–30% rather than 74–84%, the expected savings are dramatically lower and the static-prefix-first architecture design may need revisiting.

---

7. gstack productivity claims — what does "logical LOC" actually mean?

Current position: Garry Tan claims 600,000 lines of production code in 60 days (10,000–20,000/day), an 810× improvement over his 2013 baseline. gstack is cited as the primary evidence that role-based design works in practice.

What to challenge:

• "Logical LOC" is not a standard metric. What does Tan's own methodology document define as a logical line — is it meaningful-change lines excluding whitespace, comments, and generated boilerplate?
• The comparison baseline (2013, Bookface at YC) may not represent a typical developer's productivity — it may reflect his output at a specific early-career stage on a specific project type.
• Are there independent accounts of other teams using gstack achieving comparable productivity improvements?
• Does the role-based design pattern specifically drive the gain, or is it Claude Code + any structured workflow?

Design implication: The six-role factory design is substantially influenced by gstack as a reference implementation. If the productivity evidence is thin, the role design still holds on cognitive mode separation logic — but the specific gstack role mapping should be treated as one implementation, not the validated approach.

---

8. MCP security posture at homelab scale — is enterprise framing appropriate?

Current position: CIS guide, Red Hat OAuth 2.1 guidance, and vendor security research establish the MCP security requirements. Primary mitigations: dedicated agent accounts, token rotation, --tools filtering, HTTP over stdio for multi-user.

What to challenge:

• The CIS guide, Red Hat article, and vendor research are all written for enterprise contexts with multiple users and adversarial threat models. What is the appropriate security posture for a single-user, self-hosted homelab with no external users?
• The dedicated agent account mitigation for Gitea's lack of granular scopes — is this meaningfully different from a personal token in a single-user context where the account owner controls all repos anyway?
• Is stdio MCP (no auth by default) a meaningful security gap in a single-user context where only the user can send prompts?
• What does the threat model actually look like at homelab scale? Primary risks cited (token exfiltration via compromised agents, rogue MCP servers) require an adversary with machine access — at which point most mitigations are moot.

Design implication: The MCP security requirements may be calibrated for a threat model that doesn't apply to the current single-user homelab context. Some mitigations are still good practice; others may add friction without proportionate protection.

---

9. OWASP Agentic Skills Top 10 — does AST01 apply to self-authored skills?

Current position: The OWASP Agentic Skills Top 10 (April 27, 2026) is referenced in the security-review skill. AST01 (Malicious Skills) is the critical risk category.

What to challenge:

• AST01 and the ToxicSkills/ClawHavoc research concern skills installed from public registries (ClawHub, skills.sh). The factory produces and uses self-authored skills only. Does AST01 apply to self-authored private skills in any meaningful way?
• The attack vectors documented (staged payload delivery via Markdown-as-installer, prompt injection in SKILL.md) require the attacker to control the skill content. In a self-authored factory, the author controls all content. What residual risks remain?
• Are there any AST10 risk categories beyond AST01 that apply to self-authored private skills rather than public marketplace skills?
• The AST10 was published 18 days before the end of this research session. How mature is it — provisional community draft or reviewed standard?

Design implication: If AST01 primarily concerns third-party public skills, the security-review skill's AST10 reference may be misleading for a self-authored factory. The constraint should either be removed or replaced with relevant self-authored skill risks.

---

10. Single practitioner case studies — are the token optimisation figures replicable?

Current position: "Decisions not descriptions" yields 20% token reduction; AST-level file loading reduced relevant file reads from 40 to 5 (65% reduction). Both sourced to a single Nicola Alessi case study (earezki.com, February 2026).

What to challenge:

• What was the baseline in the case study — 40 file reads suggests an unusually broad grep pattern that may not reflect typical agent behaviour.
• Has any other practitioner independently documented similar reductions from the "decisions not descriptions" context file approach?
• The 65% file read reduction from AST-level loading requires implementing AST tooling (vexp, Rust-based). What is the implementation cost relative to the savings? Is it warranted at homelab scale?
• Are there published benchmarks or controlled studies of context file design approaches, or is this field entirely practitioner case studies?

Design implication: If these figures are not replicable, the token optimisation strategy rests on anecdote. The general principle (precise context over broad loading) is sound regardless, but the specific figures should not be cited as expected outcomes.

---

11. gitea-mcp permission model — verified for current Gitea versions?

Current position: Gitea has no granular token scopes — the agent can do anything including deleting repositories. The dedicated agent account is the primary mitigation.

What to challenge:

• Has Gitea introduced token scope improvements in versions 1.20, 1.21, or 1.22? The warning comes from gitea-mcp-compact (community fork README), not official Gitea documentation.
• Does the --tools flag in the official gitea-mcp actually restrict what actions the underlying token can perform, or does it only filter what the agent knows it can invoke?
• The forgejo-mcp HTTP mode with multi-user support — does it address the token granularity problem through OAuth scoping, or does it still use broad Gitea tokens underneath?
• Is the blast radius of an unintended action with a full-access token on a single-user homelab Gitea instance meaningfully different from one with a repo-scoped token?

Design implication: If Gitea has added scope support in recent versions, or if --tools provides real enforcement rather than just advertised scope, the dedicated-account-as-mitigation framing needs updating.

---

12. Skill eval reliability — does the OpenAI pattern transfer across tools?

Current position: The trigger eval pattern (10–20 test cases: explicit, implicit, negative) comes from developers.openai.com/blog/eval-skills, which documents evaluation for OpenAI Codex skills. Promptfoo is the recommended testing tool.

What to challenge:

• Does skill activation in Claude Code use the same matching mechanism as Codex, or does it differ enough that the eval structure needs adaptation?
• What is the observed false-negative rate (skill should trigger, doesn't) for well-written descriptions across tools?
• Has Promptfoo's approach to non-OpenAI model evaluation changed since the March 2026 OpenAI acquisition? Specifically: has cross-model eval parity been maintained?
• Do DeepEval or Arize Phoenix have skill-specific eval guidance, or do they require custom configuration to test skill trigger behaviour?

Design implication: If the OpenAI eval pattern doesn't transfer to Claude Code, the factory's CI eval pipeline may produce false confidence. If promptfoo's non-OpenAI coverage has degraded, the tooling recommendation needs to change.

---

Deep Research Session Objectives

1. Confirm or refute the SkillsBench 2–3 skills finding — if unconfirmed, remove from research doc.
2. Find updated context length vs. reasoning quality data for 2026 models — update the 3,000-token threshold if evidence warrants.
3. Find independent, non-vendor data on AI-generated code vulnerability rates — reconcile with or replace the Veracode finding.
4. Find practitioner or experimental data on SDD outcomes at solo/small team scale — calibrate the write-spec skill's priority if warranted.
5. Find data on prompt caching performance in agentic coding workflows specifically — update expected savings if warranted.
6. Determine whether Gitea has introduced token scope improvements — update MCP security posture if so.
7. Assess OWASP AST10 applicability to self-authored private skills — adjust security-review constraints accordingly.
8. Find cross-tool skill activation reliability data — assess whether tool-specific trigger testing is needed.

---

Prepared after initial factory research session, May 2026. Upload alongside ai-coding-factory-research.md and ai-coding-factory-session.md to continue.