docs: add ai-coding-factory research, gap analysis, and grill-me scope

Add research exploration notes (principles, challenges, implementation
guidance, skills index, session log) and gap/conflict analysis against the
current repo vision and roadmap. Sharpen the roadmap housekeeping item with
the grill-me intent, central scope question, and expected output (ADR +
updated chunk scope for 2, 3, and 4).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-05-16 20:07:00 +00:00
parent 15a4387eb2
commit ace8b53abf
8 changed files with 2263 additions and 0 deletions

View File

@@ -0,0 +1,110 @@
# Exploration Note: AI Coding Factory Research — Gaps and Conflicts with Current Roadmap
**Date:** 2026-05-16
**Status:** Pre-grill input. Not authoritative — decisions come from the grill session.
**Context:** Before starting Chunk 4 (workflows), the roadmap requires grilling the AI coding factory research against the existing repo. This note identifies where the research diverges from the current vision/roadmap/ADRs to frame that session.
---
## 1. The Central Scope Question (must resolve first)
The factory research describes a **project-level development environment** — a repo where you build software with AI assistance. This repo is a **global config repo** that provides skills, instructions, and conventions to projects.
**The question:** Is this repo *building* the factory pattern for others to use in their projects? Or is this repo *itself* structured as a factory?
This changes almost every downstream decision. Until it's resolved, individual gap items below may be pointing at the wrong layer.
---
## 2. True Conflicts (one has to win)
### 2.1 Where coding conventions live
**Factory research:** Preferred/Avoid code blocks belong in `CONTEXT.md`. Convention lives in a single shared-vocabulary file the agent always loads.
**Chunk 2 plan:** Coding conventions go in a separate `core/instructions/coding.md` file, loaded on-demand.
One of these wins. If conventions go in `CONTEXT.md`, `coding.md` becomes much thinner or unnecessary. If they stay in `coding.md`, the factory's "CONTEXT.md is the primary context artefact" principle weakens.
### 2.2 Skill path structure: flat vs. nested
**Factory research:** 34 skills in 9 nested categories — `design/grill-me/SKILL.md`, `implement/tdd/SKILL.md`, `cross-cutting/session-handoff/SKILL.md`, etc. (phase × domain matrix)
**Current state:** 12 skills in a flat structure — `grill-me/SKILL.md`, `tdd/SKILL.md`, etc.
Changing to nested paths breaks any tool that discovers skills by path. The nested structure also implies a different deployment model from `install.sh`. This is a structural decision for Chunk 3 — if we're adopting the nested taxonomy, it needs to be decided before writing any more skill files.
### 2.3 Governance file naming
**Factory research:** `AGENTS.md` at repo root as the single governance file for agents.
**Current repo:** `core/instructions/governance.md` loaded via `@import` in `providers/claude-code/CLAUDE.md`.
These serve the same purpose. The factory's naming is cleaner (one file, obvious name), but the current structure fits the provider-agnostic model (governance isn't provider-specific). Not a blocking conflict, but naming inconsistency will cause confusion in grill sessions if not resolved.
---
## 3. Gaps (factory has it; this repo doesn't plan it yet)
### 3.1 LESSONS.md (impacts Chunk 2)
Factory treats `LESSONS.md` as a required committed file — the mechanism for long-loop feedback and convention graduation. Without it, there is no defined path from "I noticed a pattern" to "this is now a standing rule in CONTEXT.md". Not in the Chunk 2 scope. Question: is this a Chunk 2 addition or deferred?
### 3.2 `docs/spec/` living spec layer
Factory requires `docs/spec/overview.md` and `docs/spec/architecture.md` — a spec that is updated in the same PR as any behaviour change. The current docs structure has no equivalent layer; workflow artifacts go in `docs/prd/`, `docs/ard/`, etc. Adding `docs/spec/` is additive (not conflicting), but it changes the docs convention and needs to be decided before Chunk 4 (workflows) defines workflow artifacts.
### 3.3 Session handoff skill (impacts Chunk 3)
Factory treats `cross-cutting/session-handoff` as a core cross-cutting skill — it explicitly prompts LESSONS.md extraction before closing a session. Not in the current skills roadmap. If LESSONS.md is adopted (see 3.1), session handoff becomes necessary. These are linked.
### 3.4 Eval infrastructure timing
Factory requires `eval.yaml` co-located with every skill. Current roadmap puts tooling in Chunk 6. Every skill built in Chunk 3 without an eval accumulates tech debt against this requirement. Chunk 6 either has to back-fill evals for all existing skills, or the decision is made now that evals are deferred deliberately.
### 3.5 `factory/` meta-skills category
Factory plans 7 meta-skills (write-skill, write-eval, write-adr, write-workflow, validate-skill, upgrade-skill, write-issue-spec) that let the factory build and verify itself. Not in the current roadmap at all. These would logically fit in Chunk 3 (skills), but they require the nested taxonomy decision (2.2) first.
### 3.6 References directory and stack versions registry
Factory requires a `references/` directory with `stack-versions.md` (authoritative version registry) and context files like `references/proxmox-cluster.md`. Not planned in the current structure. Low urgency unless IaC skills are being built.
### 3.7 Model routing
Factory defines an explicit model routing table (Haiku for classification/formatting, Sonnet for most coding, Opus for adversarial review). Not in current vision or chunk scope. The factory treats this as both a cost decision and a sustainability decision — using frontier models for tasks a smaller model can do is waste, and the governance constitution calls this out explicitly. Where does this decision live — in `governance.md`, in CONTEXT.md, or per-skill?
---
## 4. Questions the Grill Needs to Resolve
These don't have a "right answer" yet — they need explicit decisions.
**Q1:** Does this repo adopt the factory's nested skill taxonomy now (before Chunk 3 adds more skills), or does it stay flat and reorganize later? What does `install.sh` need to change to support nested paths?
**Q2:** Should LESSONS.md be added as part of Chunk 2? If yes, what's the graduation threshold and mechanism for this homelab/solo context?
**Q3:** What parts of the factory's proposed structure belong in the *global config repo* vs. in *project repos that inherit from it*? Specifically: IaC skills, Gitea integration skills, `docs/spec/`, and role skills (Architect, Developer, Reviewer, etc.) — are these global primitives or project-specific?
**Q4:** The factory's implementation guidance lists 6 grill-me sessions before building. Sessions 1 and 3 (repo vision and roadmap) are roughly done. Sessions 4 (documentation structure), 5 (governance file structure), and 6 (skill library scope) have not been formally grilled — these map directly to the decisions above. Should this grill session cover all three, or one at a time?
**Q5:** Where does the write-a-skill skill (already deployed as `write-a-skill/SKILL.md`) map in the factory taxonomy? Is it the same as `factory/write-skill`, or is it a different thing?
**Q6:** The factory says every skill needs trigger tests before production use. The current skills have no evals. Is this a hard blocker on declaring Chunk 3 "done", or is it consciously deferred to Chunk 6 with a known backlog?
---
## 5. What Is Already Aligned (don't re-litigate)
- `.agents/skills/` as canonical skill path — consistent with the Agent Skills standard
- SKILL.md format with YAML frontmatter — consistent
- `CONTEXT.md` exists and is ≤200 lines — consistent with factory's context architecture
- Pull-based distribution model — consistent with factory's "no runtime dependency" principle
- Governance.md loaded via `@import` — matches factory's "always-on" governance pattern
- No third-party skill installs without reading them — consistent with OWASP AST01 guidance
- ADRs in `docs/adr/` — consistent
- Conventional commits — consistent with factory's git-commit-message skill intent
- Multi-file skill structure (tdd/ is already multi-file) — consistent with factory's best practice
- Progressive disclosure (names load at startup, bodies on demand) — already how install.sh works
- HITL before irreversible actions — consistent with governance layer
---
*Input for the grill-me session. These are framing questions, not conclusions. The grill should surface what was missed here and challenge the framing of the conflicts before any decisions are locked.*