feat(research): integrate Context7 MCP as primary source channel
Adds Context7 resolution before websearch for library/framework/API topics, reducing reliance on web crawling for well-indexed libraries. Falls back to websearch for unresolved libraries, concept topics, or when the user provides starting URLs. Subagents are explicitly prohibited from calling Context7 to prevent tool inheritance from producing duplicate or conflicting summaries. Includes trigger and output evals for the Context7 path (resolves, fallback, skipped for non-library topics), stale description and constraint fixes, and a concrete "sufficient content" threshold. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,26 @@ trigger_tests:
|
||||
query: "Install the neuledge context server and set it up"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-context7-direct-question
|
||||
name: "Negative — direct doc question goes to context7-mcp, not research"
|
||||
query: "What are the Next.js middleware options?"
|
||||
should_trigger: false
|
||||
|
||||
- id: negative-starting-urls-skips-context7
|
||||
name: "Explicit starting URLs — Context7 step skipped, user source respected"
|
||||
query: "Research fastapi at docs/refs/fastapi/ — start here: https://fastapi.tiangolo.com/llms.txt"
|
||||
should_trigger: true
|
||||
|
||||
- id: implicit-library-context7-attempted
|
||||
name: "Implicit library research — skill triggers and should attempt Context7 before websearch"
|
||||
query: "Before we write the Redis skill, grab the docs and put them in .agents/skills/redis/references/"
|
||||
should_trigger: true
|
||||
|
||||
- id: concept-topic-still-triggers-research
|
||||
name: "Concept topic — research triggers but Context7 step is skipped (not a named library)"
|
||||
query: "/research 'JWT authentication concepts' docs/refs/jwt/"
|
||||
should_trigger: true
|
||||
|
||||
output_tests:
|
||||
- id: sources-always-written
|
||||
name: "sources.md is always produced at the output path"
|
||||
@@ -79,6 +99,40 @@ output_tests:
|
||||
- "Every file path in the write operations starts with .agents/skills/openai/references/"
|
||||
- "No files are written to the current working directory root or any other path"
|
||||
|
||||
- id: context7-resolves-source-recorded
|
||||
name: "Context7 source recorded in sources.md with context7: URL when library resolves"
|
||||
type: deterministic
|
||||
prompt: "/research next.js docs/refs/nextjs/"
|
||||
expected_output: >-
|
||||
sources.md contains at least one entry with a URL starting with context7:
|
||||
(e.g. context7:/vercel/next.js) and a slug matching context7-*.
|
||||
assertions:
|
||||
- "sources.md contains a ## context7-* section"
|
||||
- "The URL: line for that section starts with context7:"
|
||||
- "At least one topic file lists the context7-* slug in its source_keys frontmatter"
|
||||
|
||||
- id: context7-fallback-to-web
|
||||
name: "Fallback to websearch when Context7 does not resolve the library"
|
||||
type: deterministic
|
||||
prompt: "/research xyzzy-internal-tool-abc docs/refs/xyzzy/"
|
||||
expected_output: >-
|
||||
No context7-* source in sources.md. Skill falls back to websearch.
|
||||
If no web results found, skill reports failure and does not write topic files.
|
||||
assertions:
|
||||
- "sources.md does not contain a ## context7-* section"
|
||||
- "Skill either writes topic files sourced from web URLs or reports no sources found"
|
||||
|
||||
- id: context7-skipped-for-non-library
|
||||
name: "Context7 step skipped when topic is not a library/framework/API"
|
||||
type: deterministic
|
||||
prompt: "/research 'JWT authentication concepts' docs/refs/jwt/"
|
||||
expected_output: >-
|
||||
No resolve-library-id call made. Skill proceeds directly to websearch.
|
||||
sources.md contains only web-sourced slugs.
|
||||
assertions:
|
||||
- "sources.md does not contain a ## context7-* section"
|
||||
- "Topic files are sourced from web URLs, not Context7"
|
||||
|
||||
- id: content-quality
|
||||
name: "Extracted content is relevant, concise, and well-organised"
|
||||
type: llm-rubric
|
||||
|
||||
Reference in New Issue
Block a user