diff --git a/.agents/evals/research/research/eval.yaml b/.agents/evals/research/research/eval.yaml index 58d6ebe..1f402f1 100644 --- a/.agents/evals/research/research/eval.yaml +++ b/.agents/evals/research/research/eval.yaml @@ -31,6 +31,26 @@ trigger_tests: query: "Install the neuledge context server and set it up" should_trigger: false + - id: negative-context7-direct-question + name: "Negative — direct doc question goes to context7-mcp, not research" + query: "What are the Next.js middleware options?" + should_trigger: false + + - id: negative-starting-urls-skips-context7 + name: "Explicit starting URLs — Context7 step skipped, user source respected" + query: "Research fastapi at docs/refs/fastapi/ — start here: https://fastapi.tiangolo.com/llms.txt" + should_trigger: true + + - id: implicit-library-context7-attempted + name: "Implicit library research — skill triggers and should attempt Context7 before websearch" + query: "Before we write the Redis skill, grab the docs and put them in .agents/skills/redis/references/" + should_trigger: true + + - id: concept-topic-still-triggers-research + name: "Concept topic — research triggers but Context7 step is skipped (not a named library)" + query: "/research 'JWT authentication concepts' docs/refs/jwt/" + should_trigger: true + output_tests: - id: sources-always-written name: "sources.md is always produced at the output path" @@ -79,6 +99,40 @@ output_tests: - "Every file path in the write operations starts with .agents/skills/openai/references/" - "No files are written to the current working directory root or any other path" + - id: context7-resolves-source-recorded + name: "Context7 source recorded in sources.md with context7: URL when library resolves" + type: deterministic + prompt: "/research next.js docs/refs/nextjs/" + expected_output: >- + sources.md contains at least one entry with a URL starting with context7: + (e.g. context7:/vercel/next.js) and a slug matching context7-*. + assertions: + - "sources.md contains a ## context7-* section" + - "The URL: line for that section starts with context7:" + - "At least one topic file lists the context7-* slug in its source_keys frontmatter" + + - id: context7-fallback-to-web + name: "Fallback to websearch when Context7 does not resolve the library" + type: deterministic + prompt: "/research xyzzy-internal-tool-abc docs/refs/xyzzy/" + expected_output: >- + No context7-* source in sources.md. Skill falls back to websearch. + If no web results found, skill reports failure and does not write topic files. + assertions: + - "sources.md does not contain a ## context7-* section" + - "Skill either writes topic files sourced from web URLs or reports no sources found" + + - id: context7-skipped-for-non-library + name: "Context7 step skipped when topic is not a library/framework/API" + type: deterministic + prompt: "/research 'JWT authentication concepts' docs/refs/jwt/" + expected_output: >- + No resolve-library-id call made. Skill proceeds directly to websearch. + sources.md contains only web-sourced slugs. + assertions: + - "sources.md does not contain a ## context7-* section" + - "Topic files are sourced from web URLs, not Context7" + - id: content-quality name: "Extracted content is relevant, concise, and well-organised" type: llm-rubric diff --git a/.agents/skills/research/META.md b/.agents/skills/research/META.md index cb35c08..824f4c3 100644 --- a/.agents/skills/research/META.md +++ b/.agents/skills/research/META.md @@ -1,11 +1,15 @@ ```yaml -version: "1.0" +version: "1.1" updated: 2026-06-21 when: >- Invoked when the user wants to gather structured reference documentation for a - tool, library, or API from web sources. Typically run before writing a new skill - that wraps an external tool, or any time reference files are needed for a topic. - Triggered explicitly ("/research ") or implicitly when the user - asks to look up, gather, or pull docs for a topic before implementing something. + tool, library, or API from MCP documentation indexes or web sources. Typically + run before writing a new skill that wraps an external tool, or any time + reference files are needed for a topic. Triggered explicitly + ("/research ") or implicitly when the user asks to look up, + gather, or pull docs for a topic before implementing something. + +references: + - .agents/skills/context7-mcp/SKILL.md # context7-mcp — MCP source channel integrated at step 2 ``` diff --git a/.agents/skills/research/SKILL.md b/.agents/skills/research/SKILL.md index 0ff8ff7..1453ddb 100644 --- a/.agents/skills/research/SKILL.md +++ b/.agents/skills/research/SKILL.md @@ -2,13 +2,13 @@ name: research description: >- Use when the user wants to research a topic and generate structured reference - markdown files from web sources. Handles: finding canonical docs for a - tool/library/API, reading and deepening into linked pages, organizing extracted - content into topic files (overview, installation, configuration, cli-reference, - api-reference, examples, troubleshooting). Do NOT use when the user wants to - write documentation from existing code or specs (use write-docs), install or - query a specific documentation MCP server (use neuledge-context), or research - a bug/incident (use diagnose). + markdown files. Handles: finding canonical docs for a tool/library/API via + Context7 MCP or web sources, reading and deepening into linked pages, + organizing extracted content into topic files (overview, installation, + configuration, cli-reference, api-reference, examples, troubleshooting). Do + NOT use when the user wants to write documentation from existing code or specs + (use write-docs), install or manage the neuledge-context MCP server (use + neuledge-context), or research a bug/incident (use diagnose). metadata: category: research allowed-tools: @@ -16,6 +16,8 @@ allowed-tools: - WebFetch - Read - Write + - mcp__context7__resolve-library-id + - mcp__context7__query-docs model: sonnet --- @@ -33,9 +35,10 @@ model: sonnet - Skip any default topic file if no relevant content is found for it — do not create empty files - Create additional topic files beyond the default list when content warrants it (e.g. `webhooks.md`, `rate-limits.md`) - Subagents handle parallel source reading and link deepening — the orchestrator writes all files; subagents return summaries only, never write directly +- Context7 MCP calls (`resolve-library-id`, `query-docs`) are made only by the orchestrator at step 2 — subagents must not call them - `sources.md` is always written, even if only one source was read - Each topic file must have frontmatter with `topic` and `source_keys`; body is prose only — no inline URLs -- Source keys in `sources.md` must be kebab-case slugs derived from the source domain or page title +- Source keys in `sources.md` must be kebab-case slugs: derived from the source domain or page title for web sources; for Context7 sources use `context7-` (e.g. `context7-vercel-next-js`) - Default topic list and file format spec live in `references/` sub-files — read them at step 1 @@ -46,17 +49,19 @@ model: sonnet 1. **Scan codebase.** Search the working directory for existing usage of the topic — imports, config files, version pins, existing reference files. Use findings to narrow research scope (e.g. target the version already in use, skip topics already documented). Read `references/topics.md` for the default topic list and `references/file-format.md` for the output file format spec. -2. **Discover sources.** If no starting URLs were provided, websearch for canonical documentation (prefer `llms.txt`, developer docs, official API references over tutorials or blog posts). Collect 3–5 candidate URLs before reading any. +2. **Try Context7.** If the topic is a library, framework, or API and no starting URLs were provided, call `resolve-library-id` with the topic name and the user's question. If a match resolves, call `query-docs` once per default topic area (see `references/topics.md`). Treat each response as a source summary with slug `context7-` (e.g. `context7-vercel-next-js`). A topic area has sufficient content when the Context7 response contains at least one substantive paragraph — not a "no results" message, redirect notice, or header-only boilerplate. Mark covered topic areas — skip their subagent web reads in step 4. If the library does not resolve, or starting URLs were provided (explicit source choice by the user), skip this step entirely. -3. **Read sources in parallel.** Spawn one subagent per source URL. Each subagent fetches the page, extracts relevant content, identifies links worth deepening, and returns a structured summary (content by topic area + links to follow). Subagents do not write files. +3. **Discover sources.** For topics not covered by Context7 (or when no starting URLs were provided and Context7 did not resolve), websearch for canonical documentation (prefer `llms.txt`, developer docs, official API references over tutorials or blog posts). Collect 3–5 candidate URLs before reading any. -4. **Deepen.** For each subagent that returned links worth following, spawn child subagents per branch. Continue until content becomes repetitive or out of scope. Cap at ~10 additional pages total across all branches. +4. **Read sources in parallel.** Spawn one subagent per source URL. Each subagent fetches the page, extracts relevant content, identifies links worth deepening, and returns a structured summary (content by topic area + links to follow). Subagents do not write files. -5. **Consolidate.** Merge all subagent summaries by topic area. Identify which default topics have sufficient content and which custom topics emerged. +5. **Deepen.** For each subagent that returned links worth following, spawn child subagents per branch. Continue until content becomes repetitive or out of scope. Cap at ~10 additional pages total across all branches. -6. **Write topic files.** For each topic with content, write `/.md` using the format in `references/file-format.md`. Orchestrator writes all files — never delegate file writing to a subagent. +6. **Consolidate.** Merge all subagent summaries (Context7 and web) by topic area. Identify which default topics have sufficient content and which custom topics emerged. -7. **Write `sources.md`.** Write `/sources.md` mapping each source slug to its URL, description, and list of topic files it contributed to. Include sources that yielded no content, marked `no content extracted`. +7. **Write topic files.** For each topic with content, write `/.md` using the format in `references/file-format.md`. Orchestrator writes all files — never delegate file writing to a subagent. + +8. **Write `sources.md`.** Write `/sources.md` mapping each source slug to its URL (use `context7:` as the URL for Context7 sources), description, and list of topic files it contributed to. Include sources that yielded no content, marked `no content extracted`. ## Output format @@ -79,6 +84,9 @@ model: sonnet - [ ] Codebase scanned before any websearch was performed - [ ] Output path was explicitly provided — not inferred - [ ] `references/topics.md` and `references/file-format.md` read at step 1 +- [ ] Context7 resolution attempted before websearch when topic is a library/framework/API +- [ ] Context7 calls made only at orchestrator step 2 — no subagent called `resolve-library-id` or `query-docs` +- [ ] Context7 sources recorded in `sources.md` with `context7:` as URL - [ ] No topic file written without content - [ ] `sources.md` written with all sources read (including those with no content extracted) - [ ] All file writes performed by the orchestrator, not subagents diff --git a/.agents/skills/research/references/file-format.md b/.agents/skills/research/references/file-format.md index ad3b18b..58e3e7d 100644 --- a/.agents/skills/research/references/file-format.md +++ b/.agents/skills/research/references/file-format.md @@ -36,4 +36,4 @@ No inline URLs in the body — all source traceability lives in `sources.md` via - **Status:** `extracted` | `no content extracted` ``` -Use one `##` section per source. Slugs are kebab-case derived from the domain or page title (e.g. `stripe-api-docs`, `openai-python-sdk-readme`). +Use one `##` section per source. Slugs are kebab-case derived from the domain or page title (e.g. `stripe-api-docs`, `openai-python-sdk-readme`). For Context7 sources, use the slug `context7-` (e.g. `context7-vercel-next-js`) and set **URL** to `context7:` (e.g. `context7:/vercel/next.js`).