diff --git a/CONTEXT.md b/CONTEXT.md index 54b4c69..af6efa6 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -60,5 +60,11 @@ The three-stage traceability record linking a skill back to its research inputs: ### Bidirectional reference principle Files that reference other files should declare those references explicitly. The referencing file carries the forward reference (e.g. content index in `CLAUDE.md`, `references:` in frontmatter). The referenced file carries a `when:` field describing when it is loaded. Both sides should agree — divergence signals staleness. The reverse map ("what files reference this file?") is derived by a reference scanner script, not maintained manually. This principle applies to instruction files, skills, and workflow documents. +### agentsmd-author / agentsmd-audit +A skill pair in the `core` plugin for writing, updating, and reviewing a target repo's `AGENTS.md` file(s) (the generic open-standard file — see the `AGENTS.md` entry above — not this repo's own). `agentsmd-author` creates/updates AGENTS.md content, supports nested monorepo placement (per the standard's nearest-file-wins precedence), and closes out by invoking `agentsmd-audit` inline. `agentsmd-audit` runs a single combined pass checking three mandatory baselines: secrets/credentials (governance.md hard prohibition — AGENTS.md is committed content), structural completeness (common-sections checklist from the agents.md spec), and accuracy/drift (do referenced commands and paths actually resolve against the repo). `agentsmd-audit` never inspects provider adapter files (see `provider-adapter-author`) — its scope is AGENTS.md content only. Chosen over folding this into `kyberforge` because kyberforge's scope is meta-tooling for the holocron marketplace itself, not generic target-repo documentation; `core` is the intended home for cross-cutting, repo-agnostic utility skills. + +### provider-adapter-author +A companion skill (`core` plugin) that detects a target repo's provider-specific instruction file (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) and, where it duplicates content AGENTS.md should own, converts it into a thin adapter that imports AGENTS.md — mirroring this repo's own ADR-0012 pattern. Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`: checks for an import reference, no duplicated headings, size threshold) rather than a separate paired audit skill — the check is mechanical, so a script suffices per governance.md's "prefer deterministic code for repeatable tasks." `agentsmd-author` calls this skill via skill composition when it detects an existing provider file with overlapping content. + ### LESSONS.md Long-loop feedback log for patterns observed across sessions. Three or more entries on the same pattern graduate to the relevant standing file (e.g. a coding convention, a governance rule). Updated by the session-handoff skill or directly by the human. Lives at the repo root. diff --git a/docs/adr/0012-agentsmd-tooling-in-core-plugin.md b/docs/adr/0012-agentsmd-tooling-in-core-plugin.md new file mode 100644 index 0000000..eac8b90 --- /dev/null +++ b/docs/adr/0012-agentsmd-tooling-in-core-plugin.md @@ -0,0 +1,16 @@ +# AGENTS.md tooling lives in `core`, split into three skills + +`kyberforge` is scoped to meta-tooling for building and maintaining the holocron marketplace itself (skills, agents, plugins, marketplace entries) — not to generic capabilities for an arbitrary target repo. Authoring and reviewing a target repo's `AGENTS.md` file is repo-agnostic documentation tooling, closer in kind to `bin:write-docs` or `bin:init` than to `skill-author`/`plugin-author`. Research for this topic was initially placed under `plugins/kyberforge/docs/research/docs/agentsmd/` but has moved to `plugins/core/docs/research/docs/agentsmd/` to keep the provenance chain consistent with the plugin the resulting skills live in. + +## Decision + +Three skills in the `core` plugin (`core`'s first active skills): + +- **`agentsmd-author`** — creates/updates a target repo's `AGENTS.md`, including nested monorepo placement (nearest-file-wins). Closes out by invoking `agentsmd-audit` inline, mirroring the `skill-author`/`skill-audit` pattern. When it detects an existing provider-specific file (`CLAUDE.md`, etc.) with content that duplicates what AGENTS.md should own, it calls `provider-adapter-author` via skill composition. +- **`agentsmd-audit`** — a single combined pass checking three mandatory baselines against `AGENTS.md` only: secrets/credentials (governance.md hard prohibition), structural completeness (common-sections checklist from the agents.md spec), and accuracy/drift (do referenced commands/paths resolve against the repo). Never inspects provider adapter files. +- **`provider-adapter-author`** — detects and converts a provider-specific instruction file into a thin adapter that imports `AGENTS.md` (mirroring this repo's own two-tier `CLAUDE.md` pattern). Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`) rather than a separate paired audit skill, since the check (import present, no duplicated headings, size threshold) is mechanical. + +## Consequences + +- `core`'s plugin.json/README will list real skills for the first time. +- `plugins/kyberforge/docs/research/docs/agentsmd/` moves to `plugins/core/docs/research/docs/agentsmd/` before authoring begins. diff --git a/plugins/core/skills/agentsmd-audit/README.md b/plugins/core/skills/agentsmd-audit/README.md new file mode 100644 index 0000000..8dfb3e6 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/README.md @@ -0,0 +1,28 @@ +# agentsmd-audit + +Audit a target repo's AGENTS.md file(s) for embedded secrets, structural completeness, and drift. + +## What it does + +Runs a single combined pass across every AGENTS.md file in a repo (root and any nested monorepo files): flags embedded secrets/credentials, checks structure against the agents.md common-sections checklist, and resolves referenced commands/paths against the actual repo to catch stale documentation. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix. Never inspects provider-specific adapter files (CLAUDE.md, etc.) and never writes or fixes anything. + +## Usage + +``` +/agentsmd-audit +``` + +Provide the path to the repo root to audit when invoking. + +## Files + +| File | Purpose | +|------|---------| +| `SKILL.md` | Skill instructions for agents | +| `scripts/validate-secrets.sh` | Scans AGENTS.md files for embedded secrets, API keys, tokens, connection strings | +| `scripts/validate-structure.sh` | Checks for empty/placeholder content, common-sections checklist, nested-vs-root duplication | +| `scripts/validate-drift.sh` | Resolves referenced npm/make commands and file paths against the repo | +| `references/sources.md` | Provenance record — sources that informed this skill and which files each contributed to | +| `tests/validate-secrets.bats` | Bats test suite for `scripts/validate-secrets.sh` | +| `tests/validate-structure.bats` | Bats test suite for `scripts/validate-structure.sh` | +| `tests/validate-drift.bats` | Bats test suite for `scripts/validate-drift.sh` | diff --git a/plugins/core/skills/agentsmd-audit/SKILL.md b/plugins/core/skills/agentsmd-audit/SKILL.md new file mode 100644 index 0000000..c873582 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/SKILL.md @@ -0,0 +1,68 @@ +--- +name: agentsmd-audit +description: > + Use when the user wants to review a repo's AGENTS.md file, says "audit this + AGENTS.md", "check my AGENTS.md", "is this AGENTS.md any good", or wants to + know if AGENTS.md is safe to commit — even if they don't use the word + "audit". Also invoke proactively after agentsmd-author creates or updates + AGENTS.md, or after a hand-edit made outside agentsmd-author. Audits a + target repo's AGENTS.md file(s) — root and any nested monorepo files — for + embedded secrets/credentials, structural completeness against the + agents.md common-sections checklist, and drift (referenced commands or + paths that no longer resolve against the repo). Produces a compact + findings report (findings only, no PASS noise) with Why and Fix per + finding. Do not use to audit CLAUDE.md, .cursor/rules, or other + provider-specific adapter files — that's provider-adapter-author's + self-contained concern. Do not use to fix or write AGENTS.md content — use + agentsmd-author instead. +allowed-tools: Bash Read +metadata: + category: docs + source_keys: + - agents-md-official + - context7-websites-agents-md + - context7-agentsmd-agents-md + - governance-secrets-hard-prohibition + version: "0.1.0" +--- + +## Gotchas + +- Always run all three checks — this skill does a single combined pass, not staged/gated passes. Don't skip structure or drift checks just because a secrets FAIL was found. +- Never inspect or mention provider-specific adapter files (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) — that's out of scope. If one exists and duplicates AGENTS.md content, that's `provider-adapter-author`'s concern, not this skill's. +- A missing common section (e.g. no "Security" heading) is informational, not a failure — not every repo needs every section from the checklist. Only flag a FAIL when the file is empty, entirely unfilled placeholder text, or contains a real embedded secret/stale reference. +- Gather findings internally; don't narrate PASS/FAIL per check as you go — surface them only in the final report. + +## Step 1 — Run the validators + +```bash +bash scripts/validate-secrets.sh +bash scripts/validate-structure.sh +bash scripts/validate-drift.sh +``` + +Each script walks the repo for every `AGENTS.md` file (root and nested, excluding `.git`, `node_modules`, `vendor`, and similar) and prints `FAIL`/`INFO`/`SUGGESTION` lines with `Why`/`Fix` (or `Note`) per finding. A nonzero exit means at least one FAIL was found in that dimension. If a script cannot execute (`python3` unavailable, Bash denied), fall back to manual review: scan for real-looking credentials, check common sections are present, and spot-check a few referenced commands/paths by hand. + +## Step 2 — Report + +Open with a coverage line: + +```text +Checked: secrets · structure · drift +``` + +Then output only findings that were found, in this order within a repo: `### Secrets`, `### Structure`, `### Drift`. Omit a dimension heading entirely if it produced nothing — its absence confirms it passed. Report each finding verbatim as emitted by the scripts (they already carry file:line, Why/Fix or Note). + +Close with a result block: + +```text +## Result + +PASS +PASS · P info +PASS (N suggestions) · P info +FAIL (N fails) +FAIL (N fails) · P info +``` + +INFO and SUGGESTION findings are observational — they never flip PASS to FAIL. Do not fix anything — this skill reports and proposes only. Point the user to `agentsmd-author` to apply fixes. diff --git a/plugins/core/skills/agentsmd-audit/references/sources.md b/plugins/core/skills/agentsmd-audit/references/sources.md new file mode 100644 index 0000000..dfd3d9a --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/references/sources.md @@ -0,0 +1,33 @@ +# Sources + +## agents-md-official + +- **URL:** https://agents.md/ +- **Description:** Official agents.md website — format spec, common-sections checklist, precedence rules (nearest-file-wins, no merge across files), monorepo nesting patterns +- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md +- **Contributing files:** SKILL.md +- **Status:** `extracted` + +## context7-websites-agents-md + +- **URL:** context7:/websites/agents_md +- **Description:** Context7 index of the official agents.md website — overview, governance, cross-tool compatibility, configuration examples +- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md +- **Contributing files:** SKILL.md +- **Status:** `extracted` + +## context7-agentsmd-agents-md + +- **URL:** context7:/agentsmd/agents.md +- **Description:** Context7 index of the agentsmd/agents.md repository — format spec, nested monorepo patterns, file structure examples +- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md +- **Contributing files:** SKILL.md +- **Status:** `extracted` + +## governance-secrets-hard-prohibition + +- **URL:** (org convention — not a plugin research corpus entry) +- **Description:** Hard prohibition on placing secrets, API keys, tokens, or credentials in code, config, prompts, or any output. Grounds the secrets/credentials check in `scripts/validate-secrets.sh` and Step 1 of SKILL.md — AGENTS.md is committed content, so an embedded real secret is a hard-prohibition violation, not a style nit. +- **Research doc:** core/instructions/governance.md (org convention file, not a plugin research corpus entry; content is inlined here since plugins must be self-contained and this file may not exist wherever the plugin is installed) +- **Contributing files:** SKILL.md +- **Status:** `extracted` diff --git a/plugins/core/skills/agentsmd-audit/scripts/README.md b/plugins/core/skills/agentsmd-audit/scripts/README.md new file mode 100644 index 0000000..b05c3e7 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/scripts/README.md @@ -0,0 +1,11 @@ +# scripts/ + +Deterministic validators this skill shells out to instead of relying on LLM judgment for mechanical checks. + +| File | Purpose | +|------|---------| +| `validate-secrets.sh` | Scans every AGENTS.md file (root + nested) for embedded secrets, API keys, tokens, and connection strings | +| `validate-structure.sh` | Checks for empty/placeholder content, the common-sections checklist, and nested-vs-root duplication | +| `validate-drift.sh` | Resolves referenced npm/make commands and file paths against the actual repo state | + +All three take a single `` argument, print `FAIL`/`INFO`/`SUGGESTION` findings to stdout, and exit non-zero only on FAIL. diff --git a/plugins/core/skills/agentsmd-audit/scripts/validate-drift.sh b/plugins/core/skills/agentsmd-audit/scripts/validate-drift.sh new file mode 100755 index 0000000..14a0864 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/scripts/validate-drift.sh @@ -0,0 +1,137 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat < + +Check every AGENTS.md file in a repo (root and nested) for drift: package +manager scripts and file paths referenced in the text that no longer exist +in the repo. Catches the failure mode that matters most in practice — an +agent running a documented command that was renamed or deleted. + +Arguments: + repo-root Path to the repository root to scan. + +Exit codes: + 0 No FAIL findings (INFO may still be printed, e.g. no package.json found) + 1 One or more FAIL findings +EOF +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage + exit 0 +fi + +if [[ $# -lt 1 ]]; then + echo "Error: repo-root is required." >&2 + echo "" >&2 + usage >&2 + exit 1 +fi + +python3 -u - "$1" <<'PYTHON' +import sys +import os +import re +import json + +repo_root = os.path.abspath(sys.argv[1]) +if not os.path.isdir(repo_root): + print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr) + sys.exit(1) + +EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"} + +def find_agents_md(root): + results = [] + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")] + for fname in filenames: + if fname == "AGENTS.md": + results.append(os.path.join(dirpath, fname)) + return sorted(results) + +def load_package_scripts(root): + pkg_path = os.path.join(root, "package.json") + if not os.path.isfile(pkg_path): + return None + try: + with open(pkg_path, encoding="utf-8") as f: + data = json.load(f) + except (json.JSONDecodeError, OSError): + return None + return set(data.get("scripts", {}).keys()) + +def load_make_targets(root): + make_path = os.path.join(root, "Makefile") + if not os.path.isfile(make_path): + return None + with open(make_path, encoding="utf-8", errors="replace") as f: + content = f.read() + return set(re.findall(r'(?m)^([a-zA-Z0-9_-]+)\s*:(?!=)', content)) + +NPM_RUN_RE = re.compile(r'\b(?:npm|pnpm|yarn)\s+run\s+([a-zA-Z0-9:_-]+)') +MAKE_RE = re.compile(r'\bmake\s+([a-zA-Z0-9_-]+)') + +# Backticked relative file paths, e.g. `scripts/bootstrap.sh`, `src/index.ts`. +# Requires a path separator and file extension to avoid matching bare commands/words. +PATH_RE = re.compile(r'`([A-Za-z0-9_.\-]+(?:/[A-Za-z0-9_.\-]+)+\.[A-Za-z0-9]+)`') + +has_fail = False + +package_scripts = load_package_scripts(repo_root) +make_targets = load_make_targets(repo_root) + +for fpath in find_agents_md(repo_root): + rel = os.path.relpath(fpath, repo_root) + with open(fpath, encoding="utf-8", errors="replace") as f: + content = f.read() + + for m in NPM_RUN_RE.finditer(content): + script_name = m.group(1) + if package_scripts is None: + print(f"INFO Cannot verify referenced script '{script_name}' — {rel}") + print(f" Note: AGENTS.md references an npm/pnpm/yarn script, but no package.json was found at the repo root to check it against.") + print() + elif script_name not in package_scripts: + has_fail = True + print(f"FAIL Referenced script '{script_name}' not found in package.json — {rel}") + print(f" Why: AGENTS.md tells agents to run '{script_name}', but package.json has no matching \"scripts\" entry — the command will fail.") + print(f" Fix: Update AGENTS.md to reference an existing script, or add '{script_name}' to package.json's scripts.") + print() + + for m in MAKE_RE.finditer(content): + target_name = m.group(1) + if make_targets is None: + print(f"INFO Cannot verify referenced make target '{target_name}' — {rel}") + print(f" Note: AGENTS.md references a make target, but no Makefile was found at the repo root to check it against.") + print() + elif target_name not in make_targets: + has_fail = True + print(f"FAIL Referenced make target '{target_name}' not found in Makefile — {rel}") + print(f" Why: AGENTS.md tells agents to run 'make {target_name}', but the Makefile has no matching target — the command will fail.") + print(f" Fix: Update AGENTS.md to reference an existing target, or add '{target_name}' to the Makefile.") + print() + + file_dir = os.path.dirname(fpath) + for m in PATH_RE.finditer(content): + candidate = m.group(1) + resolved = ( + os.path.isfile(os.path.join(repo_root, candidate)) + or os.path.isfile(os.path.join(file_dir, candidate)) + or os.path.isdir(os.path.join(repo_root, candidate)) + or os.path.isdir(os.path.join(file_dir, candidate)) + ) + if not resolved: + has_fail = True + print(f"FAIL Referenced path '{candidate}' does not exist — {rel}") + print(f" Why: AGENTS.md points agents to '{candidate}', but it isn't present in the repo (checked relative to repo root and to the AGENTS.md's own directory).") + print(f" Fix: Update AGENTS.md to reference the correct path, or restore/create '{candidate}'.") + print() + +if has_fail: + sys.exit(1) +sys.exit(0) +PYTHON diff --git a/plugins/core/skills/agentsmd-audit/scripts/validate-secrets.sh b/plugins/core/skills/agentsmd-audit/scripts/validate-secrets.sh new file mode 100755 index 0000000..c8cb2f0 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/scripts/validate-secrets.sh @@ -0,0 +1,120 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat < + +Scan every AGENTS.md file in a repo (root and nested) for embedded secrets, +API keys, tokens, or connection strings. AGENTS.md is committed content — +real credentials in it are a hard-prohibition violation, not a style nit. +Placeholders (, \$ENV_VAR, YOUR_TOKEN_HERE, example.com, etc.) are +not flagged. + +Arguments: + repo-root Path to the repository root to scan. + +Exit codes: + 0 No findings + 1 One or more FAIL findings +EOF +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage + exit 0 +fi + +if [[ $# -lt 1 ]]; then + echo "Error: repo-root is required." >&2 + echo "" >&2 + usage >&2 + exit 1 +fi + +python3 -u - "$1" <<'PYTHON' +import sys +import os +import re + +repo_root = os.path.abspath(sys.argv[1]) +if not os.path.isdir(repo_root): + print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr) + sys.exit(1) + +EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"} + +def find_agents_md(root): + results = [] + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")] + for fname in filenames: + if fname == "AGENTS.md": + results.append(os.path.join(dirpath, fname)) + return sorted(results) + +PLACEHOLDER_RE = re.compile( + r'(?i)(your[_-]|my[_-]|example|xxx+|placeholder|changeme|<[^>]+>|\$\{|\$[A-Z_][A-Z0-9_]*|\.\.\.|redacted)' +) + +PATTERNS = [ + ("AWS access key ID", re.compile(r'AKIA[0-9A-Z]{16}')), + ("Private key block", re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----')), + ("GitHub token", re.compile(r'gh[pousr]_[A-Za-z0-9]{36,}')), + ("Slack token", re.compile(r'xox[baprs]-[A-Za-z0-9-]{10,}')), + ("GitLab token", re.compile(r'glpat-[A-Za-z0-9_-]{20,}')), + ("Generic API-style secret token", re.compile(r'\bsk-[A-Za-z0-9]{20,}\b')), + ( + "Credential-bearing connection string", + re.compile(r'[a-zA-Z][a-zA-Z0-9+.-]*://[^:@/\s]+:[^@/\s]+@[^\s\'"]+'), + ), + ( + "Assigned secret/password/token literal", + re.compile( + r'(?i)\b(api[_-]?key|secret|token|password|passwd|pwd|access[_-]?key)\b' + r'\s*[:=]\s*[\'"]?([A-Za-z0-9+/_.\-]{12,})[\'"]?' + ), + ), +] + +findings = [] + +def emit_fail(desc, fpath, lineno, why, fix): + findings.append((desc, fpath, lineno, why, fix)) + +for fpath in find_agents_md(repo_root): + rel = os.path.relpath(fpath, repo_root) + with open(fpath, encoding="utf-8", errors="replace") as f: + lines = f.readlines() + for i, line in enumerate(lines, start=1): + if PLACEHOLDER_RE.search(line): + continue + for label, pattern in PATTERNS: + m = pattern.search(line) + if not m: + continue + # Re-check placeholder allowlist against just the matched value, in case + # the placeholder marker sits outside the regex's own match span. + value = m.group(0) + if PLACEHOLDER_RE.search(value): + continue + emit_fail( + f"Possible {label}", + f"{rel}:{i}", + i, + "AGENTS.md is committed content; this line matches a real-looking credential pattern rather than a placeholder.", + "Remove the embedded credential and replace it with an environment variable reference or placeholder (e.g. $API_KEY, ).", + ) + break + +if not findings: + sys.exit(0) + +for desc, fpath, _lineno, why, fix in findings: + print(f"FAIL {desc} — {fpath}") + print(f" Why: {why}") + print(f" Fix: {fix}") + print() + +sys.exit(1) +PYTHON diff --git a/plugins/core/skills/agentsmd-audit/scripts/validate-structure.sh b/plugins/core/skills/agentsmd-audit/scripts/validate-structure.sh new file mode 100755 index 0000000..65c0981 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/scripts/validate-structure.sh @@ -0,0 +1,118 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat < + +Check every AGENTS.md file in a repo (root and nested) for structural +completeness against the agents.md spec's common-sections checklist +(setup/build, code style, testing, security, commit/PR conventions). +Missing individual sections are informational (not every repo needs every +section) — only an empty or entirely unfilled file is a hard failure. + +Arguments: + repo-root Path to the repository root to scan. + +Exit codes: + 0 No FAIL findings (INFO/SUGGESTION may still be printed) + 1 One or more FAIL findings +EOF +} + +if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then + usage + exit 0 +fi + +if [[ $# -lt 1 ]]; then + echo "Error: repo-root is required." >&2 + echo "" >&2 + usage >&2 + exit 1 +fi + +python3 -u - "$1" <<'PYTHON' +import sys +import os +import re + +PLACEHOLDER_RE = re.compile(r'(?i)FILL IN:|TODO:\s*write|lorem ipsum') + +COMMON_SECTIONS = [ + ("setup/build commands", re.compile(r'(?im)^#{1,3}\s*(setup|install|build|getting started)')), + ("code style", re.compile(r'(?im)^#{1,3}\s*(code style|style guide|conventions)')), + ("testing instructions", re.compile(r'(?im)^#{1,3}\s*(test|testing)')), + ("security considerations", re.compile(r'(?im)^#{1,3}\s*security')), + ("commit/PR conventions", re.compile(r'(?im)^#{1,3}\s*(commit|pr|pull request)')), +] + +repo_root = os.path.abspath(sys.argv[1]) +if not os.path.isdir(repo_root): + print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr) + sys.exit(1) + +EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"} + +def find_agents_md(root): + results = [] + for dirpath, dirnames, filenames in os.walk(root): + dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")] + for fname in filenames: + if fname == "AGENTS.md": + results.append(os.path.join(dirpath, fname)) + return sorted(results) + +has_fail = False +file_contents = {} # rel path -> content, for the duplication pass below + +for fpath in find_agents_md(repo_root): + rel = os.path.relpath(fpath, repo_root) + with open(fpath, encoding="utf-8", errors="replace") as f: + content = f.read() + file_contents[rel] = content + + if not content.strip(): + has_fail = True + print(f"FAIL AGENTS.md is empty — {rel}") + print(" Why: An empty file provides no instructions and gives agents nothing to act on.") + print(" Fix: Add at least a project overview and setup/test commands, per the agents.md common-sections checklist.") + print() + continue + + if PLACEHOLDER_RE.search(content): + has_fail = True + print(f"FAIL Unfilled placeholder content — {rel}") + print(" Why: A 'FILL IN:' or template stub left in place means the file has no repo-specific instructions yet.") + print(" Fix: Replace the placeholder with real, repo-specific content.") + print() + continue + + for label, pattern in COMMON_SECTIONS: + if not pattern.search(content): + print(f"INFO No {label} section — {rel}") + print(f" Note: The agents.md common-sections checklist includes {label}; not every repo needs every section, but confirm this omission is deliberate.") + print() + +# --- Nested-vs-root duplication check --- +root_content = file_contents.get("AGENTS.md") +if root_content: + root_lines = {ln.strip() for ln in root_content.splitlines() if ln.strip()} + for rel, content in file_contents.items(): + if rel == "AGENTS.md": + continue + nested_lines = [ln.strip() for ln in content.splitlines() if ln.strip()] + if not nested_lines: + continue + overlap = sum(1 for ln in nested_lines if ln in root_lines) + ratio = overlap / len(nested_lines) + if ratio >= 0.7: + print(f"SUGGESTION Nested AGENTS.md largely duplicates the root file — {rel}") + print(f" Why: {ratio:.0%} of this file's content lines already appear in the root AGENTS.md; per the spec's nearest-file-wins precedence, nested files don't inherit from the root, but they also shouldn't just restate it.") + print(f" Fix: Trim {rel} down to only what's specific to this package/directory.") + print() + +if has_fail: + sys.exit(1) +sys.exit(0) +PYTHON diff --git a/plugins/core/skills/agentsmd-audit/tests/README.md b/plugins/core/skills/agentsmd-audit/tests/README.md new file mode 100644 index 0000000..ddbfcfd --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/tests/README.md @@ -0,0 +1,30 @@ +# tests/ + +Test files for scripts bundled with this skill. + +## Dependencies + +Tests require [bats-support](https://github.com/bats-core/bats-support) and +[bats-assert](https://github.com/bats-core/bats-assert). The test files load +helpers from the repo root's `tests/test_helper/`. + +From the repo root: + +```bash +git clone https://github.com/bats-core/bats-support tests/test_helper/bats-support +git clone https://github.com/bats-core/bats-assert tests/test_helper/bats-assert +``` + +Run all tests for this skill (from the repo root): + +```bash +bats plugins/core/skills/agentsmd-audit/tests/ +``` + +## Files + +| File | Purpose | +|------|---------| +| `validate-secrets.bats` | Bats test suite for `scripts/validate-secrets.sh` | +| `validate-structure.bats` | Bats test suite for `scripts/validate-structure.sh` | +| `validate-drift.bats` | Bats test suite for `scripts/validate-drift.sh` | diff --git a/plugins/core/skills/agentsmd-audit/tests/validate-drift.bats b/plugins/core/skills/agentsmd-audit/tests/validate-drift.bats new file mode 100644 index 0000000..f76f040 --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/tests/validate-drift.bats @@ -0,0 +1,105 @@ +#!/usr/bin/env bats + +setup() { + REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)" + load "$REPO_ROOT/tests/test_helper/bats-support/load" + load "$REPO_ROOT/tests/test_helper/bats-assert/load" + + SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-drift.sh" + TMPDIR="$(mktemp -d)" +} + +teardown() { + rm -rf "$TMPDIR" +} + +@test "fails when AGENTS.md references a stale npm script" { + cat > "$TMPDIR/package.json" <<'EOF' +{ + "scripts": { + "test": "jest" + } +} +EOF + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Testing +- Run `pnpm run e2e` before committing. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "e2e" +} + +@test "passes when the referenced npm script exists" { + cat > "$TMPDIR/package.json" <<'EOF' +{ + "scripts": { + "test": "jest" + } +} +EOF + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Testing +- Run `npm run test` before committing. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success +} + +@test "fails when AGENTS.md references a stale make target" { + cat > "$TMPDIR/Makefile" <<'EOF' +build: + echo building +EOF + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +Run `make deploy` to ship. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "deploy" +} + +@test "emits INFO instead of FAIL when there is no package.json to verify an npm script against" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Testing +Run `pnpm run e2e` before committing. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success + assert_output --partial "INFO" + assert_output --partial "e2e" +} + +@test "fails when a referenced file path does not exist" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +See `scripts/bootstrap.sh` for environment setup. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "scripts/bootstrap.sh" +} + +@test "passes when the referenced file path exists" { + mkdir -p "$TMPDIR/scripts" + : > "$TMPDIR/scripts/bootstrap.sh" + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +See `scripts/bootstrap.sh` for environment setup. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success +} diff --git a/plugins/core/skills/agentsmd-audit/tests/validate-secrets.bats b/plugins/core/skills/agentsmd-audit/tests/validate-secrets.bats new file mode 100644 index 0000000..0fcbccb --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/tests/validate-secrets.bats @@ -0,0 +1,68 @@ +#!/usr/bin/env bats + +setup() { + REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)" + load "$REPO_ROOT/tests/test_helper/bats-support/load" + load "$REPO_ROOT/tests/test_helper/bats-assert/load" + + SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-secrets.sh" + TMPDIR="$(mktemp -d)" +} + +teardown() { + rm -rf "$TMPDIR" +} + +@test "passes on AGENTS.md with no secrets, only placeholders" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +- Set `export API_KEY=$API_KEY` +- Token: +- DB: postgres://user:changeme@localhost/db +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success + assert_output "" +} + +@test "fails on a real-looking AWS access key" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +- AWS_ACCESS_KEY_ID=AKIAABCDEFGHIJKLMNOP # gitleaks:allow (synthetic fixture — this test verifies validate-secrets.sh catches exactly this pattern) +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "AWS access key ID" + assert_output --partial "AGENTS.md:4" +} + +@test "fails on a credential-bearing connection string" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +- DB: postgres://svc_user:h8x2Klm9pQrT@db.internal:5432/prod +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "connection string" +} + +@test "detects secrets in a nested AGENTS.md, not just root" { + mkdir -p "$TMPDIR/packages/api" + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md +Clean root file. +EOF + cat > "$TMPDIR/packages/api/AGENTS.md" <<'EOF' +# API package +- token: ghp_1234567890abcdefghijklmnopqrstuvwxyz01 # gitleaks:allow (synthetic fixture) +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "packages/api/AGENTS.md" +} diff --git a/plugins/core/skills/agentsmd-audit/tests/validate-structure.bats b/plugins/core/skills/agentsmd-audit/tests/validate-structure.bats new file mode 100644 index 0000000..aacb39a --- /dev/null +++ b/plugins/core/skills/agentsmd-audit/tests/validate-structure.bats @@ -0,0 +1,76 @@ +#!/usr/bin/env bats + +setup() { + REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)" + load "$REPO_ROOT/tests/test_helper/bats-support/load" + load "$REPO_ROOT/tests/test_helper/bats-assert/load" + + SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-structure.sh" + TMPDIR="$(mktemp -d)" +} + +teardown() { + rm -rf "$TMPDIR" +} + +@test "fails on an empty AGENTS.md" { + : > "$TMPDIR/AGENTS.md" + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "empty" +} + +@test "fails on an unfilled placeholder AGENTS.md" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup +FILL IN: describe setup commands here. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_failure + assert_output --partial "placeholder" +} + +@test "passes with INFO on real content missing an optional section" { + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup commands +- Install deps: `pnpm install` +- Run tests: `pnpm test` + +## Code style +- TypeScript strict mode, single quotes, no semicolons. +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success + assert_output --partial "INFO" + assert_output --partial "security" +} + +@test "suggests trimming a nested AGENTS.md that duplicates the root file" { + mkdir -p "$TMPDIR/packages/api" + cat > "$TMPDIR/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup commands +- Install deps: `pnpm install` +- Run tests: `pnpm test` +- Lint: `pnpm lint` +- Build: `pnpm build` +EOF + cat > "$TMPDIR/packages/api/AGENTS.md" <<'EOF' +# AGENTS.md + +## Setup commands +- Install deps: `pnpm install` +- Run tests: `pnpm test` +- Lint: `pnpm lint` +- Build: `pnpm build` +EOF + run bash "$SCRIPT" "$TMPDIR" + assert_success + assert_output --partial "SUGGESTION" + assert_output --partial "packages/api/AGENTS.md" +}