feat(skills): add research skill for web-sourced reference file generation
Standalone /research skill that scans the codebase, discovers canonical sources via websearch, reads and deepens in parallel via subagents, and writes structured topic files + sources.md to an explicit output path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
95
.agents/evals/research/research/eval.yaml
Normal file
95
.agents/evals/research/research/eval.yaml
Normal file
@@ -0,0 +1,95 @@
|
||||
skill_name: research
|
||||
|
||||
trigger_tests:
|
||||
- id: explicit-basic
|
||||
name: "Explicit trigger — basic invocation with topic and path"
|
||||
query: "/research stripe .agents/skills/stripe/references/"
|
||||
should_trigger: true
|
||||
|
||||
- id: explicit-with-urls
|
||||
name: "Explicit trigger — topic, path, and starting URLs provided"
|
||||
query: "Research the FastMCP library and put refs in .agents/skills/mcp/references/ — start here: https://fastmcp.dev/llms.txt"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-pre-skill
|
||||
name: "Implicit trigger — looking up docs before writing a skill"
|
||||
query: "Before we write the GitHub Actions skill, can you look up the API and put the docs somewhere useful?"
|
||||
should_trigger: true
|
||||
|
||||
- id: negative-write-docs
|
||||
name: "Negative — documenting existing code goes to write-docs"
|
||||
query: "Document the auth module for me"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-diagnose
|
||||
name: "Negative — debugging investigation goes to diagnose"
|
||||
query: "Research why these integration tests are failing"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-neuledge
|
||||
name: "Negative — MCP server setup goes to neuledge-context"
|
||||
query: "Install the neuledge context server and set it up"
|
||||
should_trigger: false
|
||||
|
||||
output_tests:
|
||||
- id: sources-always-written
|
||||
name: "sources.md is always produced at the output path"
|
||||
type: deterministic
|
||||
prompt: "/research redis .agents/skills/redis/references/"
|
||||
expected_output: >-
|
||||
A sources.md file written to .agents/skills/redis/references/sources.md
|
||||
containing at least one source section with a URL and status field.
|
||||
assertions:
|
||||
- "Output path .agents/skills/redis/references/sources.md is created"
|
||||
- "sources.md contains at least one ## <slug> section"
|
||||
- "Each section in sources.md includes a URL: line and a Status: line"
|
||||
|
||||
- id: topic-frontmatter
|
||||
name: "Every topic file has required frontmatter fields"
|
||||
type: deterministic
|
||||
prompt: "/research stripe .agents/skills/stripe/references/"
|
||||
expected_output: >-
|
||||
One or more topic .md files written to .agents/skills/stripe/references/,
|
||||
each with YAML frontmatter containing topic: and source_keys: fields.
|
||||
assertions:
|
||||
- "Every .md file except sources.md contains a YAML frontmatter block delimited by ---"
|
||||
- "Every topic file frontmatter contains a topic: field"
|
||||
- "Every topic file frontmatter contains a source_keys: field with at least one entry"
|
||||
|
||||
- id: no-empty-files
|
||||
name: "No topic files written when no content is found"
|
||||
type: deterministic
|
||||
prompt: "/research xyzzy-nonexistent-tool-404 /tmp/refs-test/"
|
||||
expected_output: >-
|
||||
No topic .md files written. Skill reports what was searched and asks user
|
||||
to provide starting URLs. sources.md may still be written if sources were attempted.
|
||||
assertions:
|
||||
- "Skill does not write any topic .md files to the output path"
|
||||
- "Skill reports the search terms or URLs it tried"
|
||||
- "Skill asks the user to provide starting URLs or confirm the topic name"
|
||||
|
||||
- id: output-path-respected
|
||||
name: "All files written inside the provided output path only"
|
||||
type: deterministic
|
||||
prompt: "/research openai .agents/skills/openai/references/"
|
||||
expected_output: >-
|
||||
All written files are under .agents/skills/openai/references/. No files
|
||||
created outside that directory.
|
||||
assertions:
|
||||
- "Every file path in the write operations starts with .agents/skills/openai/references/"
|
||||
- "No files are written to the current working directory root or any other path"
|
||||
|
||||
- id: content-quality
|
||||
name: "Extracted content is relevant, concise, and well-organised"
|
||||
type: llm-rubric
|
||||
prompt: "/research fastapi .agents/skills/fastapi/references/"
|
||||
expected_output: >-
|
||||
A set of topic files covering FastAPI concepts relevant to skill authoring:
|
||||
installation, API reference, configuration, and examples at minimum. Content
|
||||
is concise summaries, not verbatim page dumps. sources.md accurately maps
|
||||
each source to the files it contributed to.
|
||||
assertions:
|
||||
- "Topic files cover concepts relevant to FastAPI (routing, request handling, dependency injection, or similar) — not generic web framework content"
|
||||
- "Body prose in each topic file is a synthesis or extraction, not a verbatim copy of the source page"
|
||||
- "source_keys in each topic file match slugs present in sources.md"
|
||||
- "sources.md contributing files list accurately reflects which topic files each source contributed to"
|
||||
Reference in New Issue
Block a user