Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix: architecture review corrections — CONTEXT.md trim and eval backtrack verification

Browse Source 
- Remove Fleet, Symbolic oversight, and Data classification tiers from
  CONTEXT.md glossary: Fleet is Phase 2 only; the other two duplicate
  always-on governance.md content
- Drop now-stale governance test assertions for the removed terms
- write-skill eval: add section-ordering assertion; fix duplicate prompt
  across two deterministic tests
- write-eval eval: tighten plan-shown assertion to require per-case
  detail (query + type), not just "a list of test cases"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Defame1297
2026-05-17 19:30:39 +00:00
parent 3dcd0f08fb
commit 2631142a0e
4 changed files with 6 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.agents/evals/factory/write-eval/eval.yaml
View File

@@ -58,9 +58,9 @@ output_tests:
name: "Deterministic — test plan presented before file is written"
type: deterministic
prompt: "write evals for the diagnose skill"
expected_output: "Skill presents proposed test cases for review and requests confirmation before writing any file"
expected_output: "Skill presents each proposed test case with its id, type, and query before writing any file, then requests confirmation"
assertions:
- "Response includes a proposed test plan or list of test cases before any file is written"
- "Response presents each proposed test case individually — showing at minimum the query and test type — before any file is written"
- "Response requests confirmation before proceeding to write"
- id: deterministic-merge-conflict-flagged

7
.agents/evals/factory/write-skill/eval.yaml
View File

@@ -33,10 +33,10 @@ trigger_tests:
output_tests:
- id: output-has-all-sections
name: All 8 body sections present
name: All 8 body sections present in order
type: deterministic
prompt: "Write a new skill for linting markdown files, category: implement"
expected_output: A complete SKILL.md containing all 8 required body sections in order.
expected_output: A complete SKILL.md containing all 8 required body sections in the prescribed order.
assertions:
- "Output contains '## Role'"
- "Output contains '## When to use / When not to use'"
@@ -46,11 +46,12 @@ output_tests:
- "Output contains '## Output format'"
- "Output contains '## Failure handling'"
- "Output contains '## Self-check'"
- "Sections appear in this order: ## Role, ## When to use / When not to use, ## Required inputs, ## Constraints, ## Process, ## Output format, ## Failure handling, ## Self-check"
- id: output-path-correct
name: Output path and frontmatter fields correct
type: deterministic
prompt: "Write a new skill for linting markdown files, category: implement"
prompt: "Write a new skill for sending Slack notifications on deploy events, category: deploy"
expected_output: A SKILL.md with correct output path stated and all required frontmatter fields present.
assertions:
- "Output contains '.agents/skills/' in the stated output path"