docs(kyberforge): add agentskillsio, agentsmd research docs and skill-write examples

- Add agentskillsio/ reference docs (8 topic files, agentskills- prefix stripped) - Add agentsmd/ reference docs (4 topic files) - Add skill-write examples: skill-creator (Anthropic), writing-great-skills (mattpocock), writing-skills (obra/superpowers) with canonical sources.md files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-22 17:42:52 +00:00
parent 345f80438e
commit 41c4da31cf
42 changed files with 10059 additions and 0 deletions
--- a/plugins/kyberforge/docs/agentskillsio/evaluating-skills.md
+++ b/plugins/kyberforge/docs/agentskillsio/evaluating-skills.md
@@ -0,0 +1,188 @@
+---
+topic: agentskills-evaluating-skills
+source_keys:
+  - agentskills-evaluating-skills
+---
+
+## Designing test cases
+
+A test case has three parts: a **prompt** (realistic user message), **expected output** (human-readable success description), and optional **input files**.
+
+Store test cases in `evals/evals.json` inside the skill directory:
+
+```json
+{
+  "skill_name": "csv-analyzer",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "I have a CSV of monthly sales data in data/sales_2025.csv. Can you find the top 3 months by revenue and make a bar chart?",
+      "expected_output": "A bar chart showing the top 3 months by revenue, with labeled axes.",
+      "files": ["evals/files/sales_2025.csv"]
+    }
+  ]
+}
+```
+
+**Tips for test prompts:**
+- Start with 2–3 test cases. Expand after seeing first results.
+- Vary phrasing, formality, and detail level.
+- Cover at least one edge case (malformed input, ambiguous request).
+- Use realistic context: file paths, column names, personal context. "Process this data" tests nothing useful.
+
+Don't define assertions yet — add them after seeing what the first run produces.
+
+## Workspace structure
+
+Each iteration gets its own directory. Each test case gets `with_skill/` and `without_skill/` subdirectories:
+
+```
+csv-analyzer/
+├── SKILL.md
+└── evals/
+    └── evals.json
+csv-analyzer-workspace/
+└── iteration-1/
+    ├── eval-top-months-chart/
+    │   ├── with_skill/
+    │   │   ├── outputs/
+    │   │   ├── timing.json
+    │   │   └── grading.json
+    │   └── without_skill/
+    │       ├── outputs/
+    │       ├── timing.json
+    │       └── grading.json
+    └── benchmark.json
+```
+
+The only file you author by hand is `evals/evals.json`. Other JSON files are produced during the eval process.
+
+## Running evals
+
+Each run starts with a clean context — no leftover state. Run each eval **with the skill** and **without it** (or against a previous version as baseline).
+
+When improving an existing skill, snapshot it before editing:
+```bash
+cp -r <skill-path> <workspace>/skill-snapshot/
+```
+
+Capture timing data when each run completes:
+```json
+{
+  "total_tokens": 84852,
+  "duration_ms": 23332
+}
+```
+
+In Claude Code, the task completion notification includes `total_tokens` and `duration_ms` — save immediately.
+
+## Writing assertions
+
+Add assertions after seeing first-round outputs. Assertions are verifiable statements about what the output should contain:
+
+Good assertions:
+- `"The output file is valid JSON"` — programmatically verifiable
+- `"The bar chart has labeled axes"` — specific and observable
+- `"The report includes at least 3 recommendations"` — countable
+
+Weak assertions:
+- `"The output is good"` — too vague
+- `"The output uses exactly the phrase 'Total Revenue: $X'"` — too brittle
+
+Add assertions to `evals.json`:
+
+```json
+"assertions": [
+  "The output includes a bar chart image file",
+  "The chart shows exactly 3 months",
+  "Both axes are labeled",
+  "The chart title or caption mentions revenue"
+]
+```
+
+## Grading outputs
+
+Grade each assertion against actual outputs: PASS or FAIL with specific evidence. Evidence should quote or reference the output, not state an opinion.
+
+```json
+{
+  "assertion_results": [
+    {
+      "text": "Both axes are labeled",
+      "passed": false,
+      "evidence": "Y-axis is labeled 'Revenue ($)' but X-axis has no label"
+    }
+  ],
+  "summary": { "passed": 3, "failed": 1, "total": 4, "pass_rate": 0.75 }
+}
+```
+
+Grading principles:
+- **Require concrete evidence for PASS.** Don't give the benefit of the doubt.
+- **Review the assertions themselves** while grading — notice when assertions are always passing (too easy) or always failing (broken/too hard).
+
+For holistic quality: try blind comparison — present both outputs to an LLM judge without revealing which came from which version. Complements assertion grading by catching differences that don't map to specific assertions.
+
+## Aggregating results
+
+```json
+{
+  "run_summary": {
+    "with_skill": {
+      "pass_rate": { "mean": 0.83, "stddev": 0.06 },
+      "time_seconds": { "mean": 45.0 },
+      "tokens": { "mean": 3800 }
+    },
+    "without_skill": {
+      "pass_rate": { "mean": 0.33, "stddev": 0.10 },
+      "time_seconds": { "mean": 32.0 },
+      "tokens": { "mean": 2100 }
+    },
+    "delta": { "pass_rate": 0.50, "time_seconds": 13.0, "tokens": 1700 }
+  }
+}
+```
+
+The `delta` tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate).
+
+## Analyzing patterns
+
+- **Remove assertions that always pass in both configurations.** They inflate the with-skill pass rate without reflecting skill value.
+- **Investigate assertions that always fail in both.** Either the assertion is broken or the task is too hard.
+- **Study assertions that pass with-skill but fail without.** This is where the skill adds value — understand why.
+- **Tighten instructions when results are inconsistent.** High `stddev` means flaky evals or ambiguous instructions. Add examples or more specific guidance.
+- **Check time and token outliers.** If one eval takes 3× longer, read its execution transcript.
+
+## Human review
+
+After grading, review actual outputs. Assertion grading only checks what you thought to write assertions for. Record specific feedback:
+
+```json
+{
+  "eval-top-months-chart": "Chart is missing axis labels and months are alphabetical not chronological.",
+  "eval-clean-missing-emails": ""
+}
+```
+
+"Missing axis labels" is actionable; "looks bad" is not. Empty feedback means the output looked fine.
+
+## Iterating on the skill
+
+Three sources of signal:
+- **Failed assertions** — specific gaps: missing step, unclear instruction, unhandled case
+- **Human feedback** — broader quality issues: wrong approach, poorly structured output
+- **Execution transcripts** — why things went wrong: ambiguous instructions, unproductive steps
+
+Feed all three — with the current `SKILL.md` — to an LLM to propose changes. Guidelines for the LLM:
+- Generalize from feedback (don't add narrow patches for specific examples)
+- Keep the skill lean (fewer, better instructions outperform exhaustive rules)
+- Explain the why (reasoning-based instructions work better than rigid directives)
+- Bundle repeated work into `scripts/` when agents reinvent the same logic each run
+
+### The loop
+
+1. Give eval signals + current `SKILL.md` to an LLM; ask for improvements.
+2. Review and apply the changes.
+3. Rerun all test cases in a new `iteration-<N+1>/` directory.
+4. Grade, aggregate, and review with a human.
+5. Repeat until satisfied, feedback is consistently empty, or improvement stops.