diff --git a/.agents/evals/factory/write-eval/eval.yaml b/.agents/evals/factory/write-eval/eval.yaml index bcdb894..4eeeae9 100644 --- a/.agents/evals/factory/write-eval/eval.yaml +++ b/.agents/evals/factory/write-eval/eval.yaml @@ -58,9 +58,9 @@ output_tests: name: "Deterministic — test plan presented before file is written" type: deterministic prompt: "write evals for the diagnose skill" - expected_output: "Skill presents proposed test cases for review and requests confirmation before writing any file" + expected_output: "Skill presents each proposed test case with its id, type, and query before writing any file, then requests confirmation" assertions: - - "Response includes a proposed test plan or list of test cases before any file is written" + - "Response presents each proposed test case individually — showing at minimum the query and test type — before any file is written" - "Response requests confirmation before proceeding to write" - id: deterministic-merge-conflict-flagged diff --git a/.agents/evals/factory/write-skill/eval.yaml b/.agents/evals/factory/write-skill/eval.yaml index 4193518..1fee481 100644 --- a/.agents/evals/factory/write-skill/eval.yaml +++ b/.agents/evals/factory/write-skill/eval.yaml @@ -33,10 +33,10 @@ trigger_tests: output_tests: - id: output-has-all-sections - name: All 8 body sections present + name: All 8 body sections present in order type: deterministic prompt: "Write a new skill for linting markdown files, category: implement" - expected_output: A complete SKILL.md containing all 8 required body sections in order. + expected_output: A complete SKILL.md containing all 8 required body sections in the prescribed order. assertions: - "Output contains '## Role'" - "Output contains '## When to use / When not to use'" @@ -46,11 +46,12 @@ output_tests: - "Output contains '## Output format'" - "Output contains '## Failure handling'" - "Output contains '## Self-check'" + - "Sections appear in this order: ## Role, ## When to use / When not to use, ## Required inputs, ## Constraints, ## Process, ## Output format, ## Failure handling, ## Self-check" - id: output-path-correct name: Output path and frontmatter fields correct type: deterministic - prompt: "Write a new skill for linting markdown files, category: implement" + prompt: "Write a new skill for sending Slack notifications on deploy events, category: deploy" expected_output: A SKILL.md with correct output path stated and all required frontmatter fields present. assertions: - "Output contains '.agents/skills/' in the stated output path" diff --git a/CONTEXT.md b/CONTEXT.md index a89781b..97ab4ca 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -64,9 +64,6 @@ This repo is built by a junior developer as a homelab tool intended to scale to ### Management Application A separate product (separate repo) for browsing, editing, and configuring AI development configs through a proper product UI. Git is the persistence layer, invisible to the user. The app is repo-agnostic — it works with any git repo that follows these conventions. This repo is the canonical default content (the official starter). See `docs/VISION.md` for the phased roadmap. -### Fleet -The set of machines and projects under management in Phase 2 of the Management Application. A fleet member is any machine that has the config deployed and can be reached by the runtime orchestration layer. - ### Skills Reusable slash commands for AI coding tools, defined as `SKILL.md` files following the [Agent Skills open standard](https://agentskills.io). Canonical location: `.agents/skills//SKILL.md` in this repo; deployed to `~/.agents/skills/` on install. Providers that don't read `~/.agents/skills/` natively get a symlink adapter declared in `providers//provider-manifest.sh` (e.g. Claude Code: `~/.claude/skills/ → ~/.agents/skills/`). @@ -82,21 +79,6 @@ Agent pauses before a consequential action; human approves before execution. Req ### HOTL (human-on-the-loop) Agent acts; human monitors and can intervene after the fact. Acceptable for low-stakes, bounded, reversible actions where the cost of pausing for approval exceeds the blast radius of an error. The distinction between HITL and HOTL must be explicit and documented — defaulting to HOTL for convenience is not acceptable. -### Symbolic oversight -Oversight implemented as a gesture (assigning a reviewer) rather than a functional safeguard. A reviewer without the information, time, agency, or intent to evaluate is not oversight — it is the appearance of oversight. The documented failure mode: symbolic oversight passes audits but does not catch errors. Genuine oversight requires: the reviewer has access to what was produced, time to evaluate it meaningfully, authority to reject it, and the intent to do so. - -### Data classification tiers -The four-tier framework governing what data may enter AI context. Apply the tier of the most sensitive element in any dataset or prompt. - -| Tier | Examples | AI Rule | -|---|---|---| -| **Public** | Publicly available info | No restrictions | -| **Internal** | Operational data, anonymised logs | Enterprise AI tools only; not consumer/free-tier | -| **Confidential** | Source code, architecture, personal data, IP | Enterprise AI + contractual data-not-trained guarantee | -| **Restricted** | GDPR Article 9 health/biometrics, credentials, regulated financial data | Never enters AI context — hard architectural prohibition | - -Defined in full in `docs/ai-constitution.md` Section 3. Agent-actionable rules in `core/instructions/governance.md`. - ### Sycophancy The failure mode where RLHF-trained models prioritise approval over accuracy. Treated as a first-class reliability risk: models change correct answers to wrong ones under user pressure in a majority of observed cases, then persist in the wrong answer. Designing against sycophancy is an explicit obligation, not a quality-of-life concern. Countermeasures: explicit pushback resistance instructions, prompting for dissent, cross-validating against independent sources. Never interpret AI agreement as AI accuracy. diff --git a/tests/test-governance-layer.sh b/tests/test-governance-layer.sh index 783621e..fa79931 100755 --- a/tests/test-governance-layer.sh +++ b/tests/test-governance-layer.sh @@ -111,14 +111,6 @@ contains "HOTL" "$CONTEXT" \ && pass "CONTEXT.md: HOTL term defined" \ || fail "CONTEXT.md: HOTL definition missing" -contains "[Ss]ymbolic oversight" "$CONTEXT" \ - && pass "CONTEXT.md: Symbolic oversight defined" \ - || fail "CONTEXT.md: Symbolic oversight definition missing" - -contains "[Dd]ata classification" "$CONTEXT" \ - && pass "CONTEXT.md: Data classification tiers defined" \ - || fail "CONTEXT.md: Data classification tiers missing" - contains "[Ss]ycophancy" "$CONTEXT" \ && pass "CONTEXT.md: Sycophancy defined" \ || fail "CONTEXT.md: Sycophancy definition missing"