feat(core): add agentsmd-audit skill files
The previous commit only landed the research-folder rename — a multi-path git add silently failed and left CONTEXT.md, ADR-0012, and the actual skill files unstaged. This lands them: the agentsmd-audit skill itself (three deterministic validators for secrets, structure, and drift against a target repo's AGENTS.md), its bats test suite, provenance record, and the CONTEXT.md/ADR entries documenting why this lives in core rather than kyberforge. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -60,5 +60,11 @@ The three-stage traceability record linking a skill back to its research inputs:
|
||||
### Bidirectional reference principle
|
||||
Files that reference other files should declare those references explicitly. The referencing file carries the forward reference (e.g. content index in `CLAUDE.md`, `references:` in frontmatter). The referenced file carries a `when:` field describing when it is loaded. Both sides should agree — divergence signals staleness. The reverse map ("what files reference this file?") is derived by a reference scanner script, not maintained manually. This principle applies to instruction files, skills, and workflow documents.
|
||||
|
||||
### agentsmd-author / agentsmd-audit
|
||||
A skill pair in the `core` plugin for writing, updating, and reviewing a target repo's `AGENTS.md` file(s) (the generic open-standard file — see the `AGENTS.md` entry above — not this repo's own). `agentsmd-author` creates/updates AGENTS.md content, supports nested monorepo placement (per the standard's nearest-file-wins precedence), and closes out by invoking `agentsmd-audit` inline. `agentsmd-audit` runs a single combined pass checking three mandatory baselines: secrets/credentials (governance.md hard prohibition — AGENTS.md is committed content), structural completeness (common-sections checklist from the agents.md spec), and accuracy/drift (do referenced commands and paths actually resolve against the repo). `agentsmd-audit` never inspects provider adapter files (see `provider-adapter-author`) — its scope is AGENTS.md content only. Chosen over folding this into `kyberforge` because kyberforge's scope is meta-tooling for the holocron marketplace itself, not generic target-repo documentation; `core` is the intended home for cross-cutting, repo-agnostic utility skills.
|
||||
|
||||
### provider-adapter-author
|
||||
A companion skill (`core` plugin) that detects a target repo's provider-specific instruction file (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) and, where it duplicates content AGENTS.md should own, converts it into a thin adapter that imports AGENTS.md — mirroring this repo's own ADR-0012 pattern. Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`: checks for an import reference, no duplicated headings, size threshold) rather than a separate paired audit skill — the check is mechanical, so a script suffices per governance.md's "prefer deterministic code for repeatable tasks." `agentsmd-author` calls this skill via skill composition when it detects an existing provider file with overlapping content.
|
||||
|
||||
### LESSONS.md
|
||||
Long-loop feedback log for patterns observed across sessions. Three or more entries on the same pattern graduate to the relevant standing file (e.g. a coding convention, a governance rule). Updated by the session-handoff skill or directly by the human. Lives at the repo root.
|
||||
|
||||
16
docs/adr/0012-agentsmd-tooling-in-core-plugin.md
Normal file
16
docs/adr/0012-agentsmd-tooling-in-core-plugin.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# AGENTS.md tooling lives in `core`, split into three skills
|
||||
|
||||
`kyberforge` is scoped to meta-tooling for building and maintaining the holocron marketplace itself (skills, agents, plugins, marketplace entries) — not to generic capabilities for an arbitrary target repo. Authoring and reviewing a target repo's `AGENTS.md` file is repo-agnostic documentation tooling, closer in kind to `bin:write-docs` or `bin:init` than to `skill-author`/`plugin-author`. Research for this topic was initially placed under `plugins/kyberforge/docs/research/docs/agentsmd/` but has moved to `plugins/core/docs/research/docs/agentsmd/` to keep the provenance chain consistent with the plugin the resulting skills live in.
|
||||
|
||||
## Decision
|
||||
|
||||
Three skills in the `core` plugin (`core`'s first active skills):
|
||||
|
||||
- **`agentsmd-author`** — creates/updates a target repo's `AGENTS.md`, including nested monorepo placement (nearest-file-wins). Closes out by invoking `agentsmd-audit` inline, mirroring the `skill-author`/`skill-audit` pattern. When it detects an existing provider-specific file (`CLAUDE.md`, etc.) with content that duplicates what AGENTS.md should own, it calls `provider-adapter-author` via skill composition.
|
||||
- **`agentsmd-audit`** — a single combined pass checking three mandatory baselines against `AGENTS.md` only: secrets/credentials (governance.md hard prohibition), structural completeness (common-sections checklist from the agents.md spec), and accuracy/drift (do referenced commands/paths resolve against the repo). Never inspects provider adapter files.
|
||||
- **`provider-adapter-author`** — detects and converts a provider-specific instruction file into a thin adapter that imports `AGENTS.md` (mirroring this repo's own two-tier `CLAUDE.md` pattern). Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`) rather than a separate paired audit skill, since the check (import present, no duplicated headings, size threshold) is mechanical.
|
||||
|
||||
## Consequences
|
||||
|
||||
- `core`'s plugin.json/README will list real skills for the first time.
|
||||
- `plugins/kyberforge/docs/research/docs/agentsmd/` moves to `plugins/core/docs/research/docs/agentsmd/` before authoring begins.
|
||||
28
plugins/core/skills/agentsmd-audit/README.md
Normal file
28
plugins/core/skills/agentsmd-audit/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# agentsmd-audit
|
||||
|
||||
Audit a target repo's AGENTS.md file(s) for embedded secrets, structural completeness, and drift.
|
||||
|
||||
## What it does
|
||||
|
||||
Runs a single combined pass across every AGENTS.md file in a repo (root and any nested monorepo files): flags embedded secrets/credentials, checks structure against the agents.md common-sections checklist, and resolves referenced commands/paths against the actual repo to catch stale documentation. Outputs a compact findings report — findings only, grouped by dimension, each with Why and Fix. Never inspects provider-specific adapter files (CLAUDE.md, etc.) and never writes or fixes anything.
|
||||
|
||||
## Usage
|
||||
|
||||
```
|
||||
/agentsmd-audit
|
||||
```
|
||||
|
||||
Provide the path to the repo root to audit when invoking.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `SKILL.md` | Skill instructions for agents |
|
||||
| `scripts/validate-secrets.sh` | Scans AGENTS.md files for embedded secrets, API keys, tokens, connection strings |
|
||||
| `scripts/validate-structure.sh` | Checks for empty/placeholder content, common-sections checklist, nested-vs-root duplication |
|
||||
| `scripts/validate-drift.sh` | Resolves referenced npm/make commands and file paths against the repo |
|
||||
| `references/sources.md` | Provenance record — sources that informed this skill and which files each contributed to |
|
||||
| `tests/validate-secrets.bats` | Bats test suite for `scripts/validate-secrets.sh` |
|
||||
| `tests/validate-structure.bats` | Bats test suite for `scripts/validate-structure.sh` |
|
||||
| `tests/validate-drift.bats` | Bats test suite for `scripts/validate-drift.sh` |
|
||||
68
plugins/core/skills/agentsmd-audit/SKILL.md
Normal file
68
plugins/core/skills/agentsmd-audit/SKILL.md
Normal file
@@ -0,0 +1,68 @@
|
||||
---
|
||||
name: agentsmd-audit
|
||||
description: >
|
||||
Use when the user wants to review a repo's AGENTS.md file, says "audit this
|
||||
AGENTS.md", "check my AGENTS.md", "is this AGENTS.md any good", or wants to
|
||||
know if AGENTS.md is safe to commit — even if they don't use the word
|
||||
"audit". Also invoke proactively after agentsmd-author creates or updates
|
||||
AGENTS.md, or after a hand-edit made outside agentsmd-author. Audits a
|
||||
target repo's AGENTS.md file(s) — root and any nested monorepo files — for
|
||||
embedded secrets/credentials, structural completeness against the
|
||||
agents.md common-sections checklist, and drift (referenced commands or
|
||||
paths that no longer resolve against the repo). Produces a compact
|
||||
findings report (findings only, no PASS noise) with Why and Fix per
|
||||
finding. Do not use to audit CLAUDE.md, .cursor/rules, or other
|
||||
provider-specific adapter files — that's provider-adapter-author's
|
||||
self-contained concern. Do not use to fix or write AGENTS.md content — use
|
||||
agentsmd-author instead.
|
||||
allowed-tools: Bash Read
|
||||
metadata:
|
||||
category: docs
|
||||
source_keys:
|
||||
- agents-md-official
|
||||
- context7-websites-agents-md
|
||||
- context7-agentsmd-agents-md
|
||||
- governance-secrets-hard-prohibition
|
||||
version: "0.1.0"
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Always run all three checks — this skill does a single combined pass, not staged/gated passes. Don't skip structure or drift checks just because a secrets FAIL was found.
|
||||
- Never inspect or mention provider-specific adapter files (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) — that's out of scope. If one exists and duplicates AGENTS.md content, that's `provider-adapter-author`'s concern, not this skill's.
|
||||
- A missing common section (e.g. no "Security" heading) is informational, not a failure — not every repo needs every section from the checklist. Only flag a FAIL when the file is empty, entirely unfilled placeholder text, or contains a real embedded secret/stale reference.
|
||||
- Gather findings internally; don't narrate PASS/FAIL per check as you go — surface them only in the final report.
|
||||
|
||||
## Step 1 — Run the validators
|
||||
|
||||
```bash
|
||||
bash scripts/validate-secrets.sh <repo-root>
|
||||
bash scripts/validate-structure.sh <repo-root>
|
||||
bash scripts/validate-drift.sh <repo-root>
|
||||
```
|
||||
|
||||
Each script walks the repo for every `AGENTS.md` file (root and nested, excluding `.git`, `node_modules`, `vendor`, and similar) and prints `FAIL`/`INFO`/`SUGGESTION` lines with `Why`/`Fix` (or `Note`) per finding. A nonzero exit means at least one FAIL was found in that dimension. If a script cannot execute (`python3` unavailable, Bash denied), fall back to manual review: scan for real-looking credentials, check common sections are present, and spot-check a few referenced commands/paths by hand.
|
||||
|
||||
## Step 2 — Report
|
||||
|
||||
Open with a coverage line:
|
||||
|
||||
```text
|
||||
Checked: secrets · structure · drift
|
||||
```
|
||||
|
||||
Then output only findings that were found, in this order within a repo: `### Secrets`, `### Structure`, `### Drift`. Omit a dimension heading entirely if it produced nothing — its absence confirms it passed. Report each finding verbatim as emitted by the scripts (they already carry file:line, Why/Fix or Note).
|
||||
|
||||
Close with a result block:
|
||||
|
||||
```text
|
||||
## Result
|
||||
|
||||
PASS
|
||||
PASS · P info
|
||||
PASS (N suggestions) · P info
|
||||
FAIL (N fails)
|
||||
FAIL (N fails) · P info
|
||||
```
|
||||
|
||||
INFO and SUGGESTION findings are observational — they never flip PASS to FAIL. Do not fix anything — this skill reports and proposes only. Point the user to `agentsmd-author` to apply fixes.
|
||||
33
plugins/core/skills/agentsmd-audit/references/sources.md
Normal file
33
plugins/core/skills/agentsmd-audit/references/sources.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# Sources
|
||||
|
||||
## agents-md-official
|
||||
|
||||
- **URL:** https://agents.md/
|
||||
- **Description:** Official agents.md website — format spec, common-sections checklist, precedence rules (nearest-file-wins, no merge across files), monorepo nesting patterns
|
||||
- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md
|
||||
- **Contributing files:** SKILL.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## context7-websites-agents-md
|
||||
|
||||
- **URL:** context7:/websites/agents_md
|
||||
- **Description:** Context7 index of the official agents.md website — overview, governance, cross-tool compatibility, configuration examples
|
||||
- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md
|
||||
- **Contributing files:** SKILL.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## context7-agentsmd-agents-md
|
||||
|
||||
- **URL:** context7:/agentsmd/agents.md
|
||||
- **Description:** Context7 index of the agentsmd/agents.md repository — format spec, nested monorepo patterns, file structure examples
|
||||
- **Research doc:** plugins/core/docs/research/docs/agentsmd/sources.md
|
||||
- **Contributing files:** SKILL.md
|
||||
- **Status:** `extracted`
|
||||
|
||||
## governance-secrets-hard-prohibition
|
||||
|
||||
- **URL:** (org convention — not a plugin research corpus entry)
|
||||
- **Description:** Hard prohibition on placing secrets, API keys, tokens, or credentials in code, config, prompts, or any output. Grounds the secrets/credentials check in `scripts/validate-secrets.sh` and Step 1 of SKILL.md — AGENTS.md is committed content, so an embedded real secret is a hard-prohibition violation, not a style nit.
|
||||
- **Research doc:** core/instructions/governance.md (org convention file, not a plugin research corpus entry; content is inlined here since plugins must be self-contained and this file may not exist wherever the plugin is installed)
|
||||
- **Contributing files:** SKILL.md
|
||||
- **Status:** `extracted`
|
||||
11
plugins/core/skills/agentsmd-audit/scripts/README.md
Normal file
11
plugins/core/skills/agentsmd-audit/scripts/README.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# scripts/
|
||||
|
||||
Deterministic validators this skill shells out to instead of relying on LLM judgment for mechanical checks.
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `validate-secrets.sh` | Scans every AGENTS.md file (root + nested) for embedded secrets, API keys, tokens, and connection strings |
|
||||
| `validate-structure.sh` | Checks for empty/placeholder content, the common-sections checklist, and nested-vs-root duplication |
|
||||
| `validate-drift.sh` | Resolves referenced npm/make commands and file paths against the actual repo state |
|
||||
|
||||
All three take a single `<repo-root>` argument, print `FAIL`/`INFO`/`SUGGESTION` findings to stdout, and exit non-zero only on FAIL.
|
||||
137
plugins/core/skills/agentsmd-audit/scripts/validate-drift.sh
Executable file
137
plugins/core/skills/agentsmd-audit/scripts/validate-drift.sh
Executable file
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: validate-drift.sh <repo-root>
|
||||
|
||||
Check every AGENTS.md file in a repo (root and nested) for drift: package
|
||||
manager scripts and file paths referenced in the text that no longer exist
|
||||
in the repo. Catches the failure mode that matters most in practice — an
|
||||
agent running a documented command that was renamed or deleted.
|
||||
|
||||
Arguments:
|
||||
repo-root Path to the repository root to scan.
|
||||
|
||||
Exit codes:
|
||||
0 No FAIL findings (INFO may still be printed, e.g. no package.json found)
|
||||
1 One or more FAIL findings
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Error: repo-root is required." >&2
|
||||
echo "" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 -u - "$1" <<'PYTHON'
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
import json
|
||||
|
||||
repo_root = os.path.abspath(sys.argv[1])
|
||||
if not os.path.isdir(repo_root):
|
||||
print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"}
|
||||
|
||||
def find_agents_md(root):
|
||||
results = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")]
|
||||
for fname in filenames:
|
||||
if fname == "AGENTS.md":
|
||||
results.append(os.path.join(dirpath, fname))
|
||||
return sorted(results)
|
||||
|
||||
def load_package_scripts(root):
|
||||
pkg_path = os.path.join(root, "package.json")
|
||||
if not os.path.isfile(pkg_path):
|
||||
return None
|
||||
try:
|
||||
with open(pkg_path, encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return None
|
||||
return set(data.get("scripts", {}).keys())
|
||||
|
||||
def load_make_targets(root):
|
||||
make_path = os.path.join(root, "Makefile")
|
||||
if not os.path.isfile(make_path):
|
||||
return None
|
||||
with open(make_path, encoding="utf-8", errors="replace") as f:
|
||||
content = f.read()
|
||||
return set(re.findall(r'(?m)^([a-zA-Z0-9_-]+)\s*:(?!=)', content))
|
||||
|
||||
NPM_RUN_RE = re.compile(r'\b(?:npm|pnpm|yarn)\s+run\s+([a-zA-Z0-9:_-]+)')
|
||||
MAKE_RE = re.compile(r'\bmake\s+([a-zA-Z0-9_-]+)')
|
||||
|
||||
# Backticked relative file paths, e.g. `scripts/bootstrap.sh`, `src/index.ts`.
|
||||
# Requires a path separator and file extension to avoid matching bare commands/words.
|
||||
PATH_RE = re.compile(r'`([A-Za-z0-9_.\-]+(?:/[A-Za-z0-9_.\-]+)+\.[A-Za-z0-9]+)`')
|
||||
|
||||
has_fail = False
|
||||
|
||||
package_scripts = load_package_scripts(repo_root)
|
||||
make_targets = load_make_targets(repo_root)
|
||||
|
||||
for fpath in find_agents_md(repo_root):
|
||||
rel = os.path.relpath(fpath, repo_root)
|
||||
with open(fpath, encoding="utf-8", errors="replace") as f:
|
||||
content = f.read()
|
||||
|
||||
for m in NPM_RUN_RE.finditer(content):
|
||||
script_name = m.group(1)
|
||||
if package_scripts is None:
|
||||
print(f"INFO Cannot verify referenced script '{script_name}' — {rel}")
|
||||
print(f" Note: AGENTS.md references an npm/pnpm/yarn script, but no package.json was found at the repo root to check it against.")
|
||||
print()
|
||||
elif script_name not in package_scripts:
|
||||
has_fail = True
|
||||
print(f"FAIL Referenced script '{script_name}' not found in package.json — {rel}")
|
||||
print(f" Why: AGENTS.md tells agents to run '{script_name}', but package.json has no matching \"scripts\" entry — the command will fail.")
|
||||
print(f" Fix: Update AGENTS.md to reference an existing script, or add '{script_name}' to package.json's scripts.")
|
||||
print()
|
||||
|
||||
for m in MAKE_RE.finditer(content):
|
||||
target_name = m.group(1)
|
||||
if make_targets is None:
|
||||
print(f"INFO Cannot verify referenced make target '{target_name}' — {rel}")
|
||||
print(f" Note: AGENTS.md references a make target, but no Makefile was found at the repo root to check it against.")
|
||||
print()
|
||||
elif target_name not in make_targets:
|
||||
has_fail = True
|
||||
print(f"FAIL Referenced make target '{target_name}' not found in Makefile — {rel}")
|
||||
print(f" Why: AGENTS.md tells agents to run 'make {target_name}', but the Makefile has no matching target — the command will fail.")
|
||||
print(f" Fix: Update AGENTS.md to reference an existing target, or add '{target_name}' to the Makefile.")
|
||||
print()
|
||||
|
||||
file_dir = os.path.dirname(fpath)
|
||||
for m in PATH_RE.finditer(content):
|
||||
candidate = m.group(1)
|
||||
resolved = (
|
||||
os.path.isfile(os.path.join(repo_root, candidate))
|
||||
or os.path.isfile(os.path.join(file_dir, candidate))
|
||||
or os.path.isdir(os.path.join(repo_root, candidate))
|
||||
or os.path.isdir(os.path.join(file_dir, candidate))
|
||||
)
|
||||
if not resolved:
|
||||
has_fail = True
|
||||
print(f"FAIL Referenced path '{candidate}' does not exist — {rel}")
|
||||
print(f" Why: AGENTS.md points agents to '{candidate}', but it isn't present in the repo (checked relative to repo root and to the AGENTS.md's own directory).")
|
||||
print(f" Fix: Update AGENTS.md to reference the correct path, or restore/create '{candidate}'.")
|
||||
print()
|
||||
|
||||
if has_fail:
|
||||
sys.exit(1)
|
||||
sys.exit(0)
|
||||
PYTHON
|
||||
120
plugins/core/skills/agentsmd-audit/scripts/validate-secrets.sh
Executable file
120
plugins/core/skills/agentsmd-audit/scripts/validate-secrets.sh
Executable file
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: validate-secrets.sh <repo-root>
|
||||
|
||||
Scan every AGENTS.md file in a repo (root and nested) for embedded secrets,
|
||||
API keys, tokens, or connection strings. AGENTS.md is committed content —
|
||||
real credentials in it are a hard-prohibition violation, not a style nit.
|
||||
Placeholders (<your-key>, \$ENV_VAR, YOUR_TOKEN_HERE, example.com, etc.) are
|
||||
not flagged.
|
||||
|
||||
Arguments:
|
||||
repo-root Path to the repository root to scan.
|
||||
|
||||
Exit codes:
|
||||
0 No findings
|
||||
1 One or more FAIL findings
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Error: repo-root is required." >&2
|
||||
echo "" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 -u - "$1" <<'PYTHON'
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
|
||||
repo_root = os.path.abspath(sys.argv[1])
|
||||
if not os.path.isdir(repo_root):
|
||||
print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"}
|
||||
|
||||
def find_agents_md(root):
|
||||
results = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")]
|
||||
for fname in filenames:
|
||||
if fname == "AGENTS.md":
|
||||
results.append(os.path.join(dirpath, fname))
|
||||
return sorted(results)
|
||||
|
||||
PLACEHOLDER_RE = re.compile(
|
||||
r'(?i)(your[_-]|my[_-]|example|xxx+|placeholder|changeme|<[^>]+>|\$\{|\$[A-Z_][A-Z0-9_]*|\.\.\.|redacted)'
|
||||
)
|
||||
|
||||
PATTERNS = [
|
||||
("AWS access key ID", re.compile(r'AKIA[0-9A-Z]{16}')),
|
||||
("Private key block", re.compile(r'-----BEGIN [A-Z ]*PRIVATE KEY-----')),
|
||||
("GitHub token", re.compile(r'gh[pousr]_[A-Za-z0-9]{36,}')),
|
||||
("Slack token", re.compile(r'xox[baprs]-[A-Za-z0-9-]{10,}')),
|
||||
("GitLab token", re.compile(r'glpat-[A-Za-z0-9_-]{20,}')),
|
||||
("Generic API-style secret token", re.compile(r'\bsk-[A-Za-z0-9]{20,}\b')),
|
||||
(
|
||||
"Credential-bearing connection string",
|
||||
re.compile(r'[a-zA-Z][a-zA-Z0-9+.-]*://[^:@/\s]+:[^@/\s]+@[^\s\'"]+'),
|
||||
),
|
||||
(
|
||||
"Assigned secret/password/token literal",
|
||||
re.compile(
|
||||
r'(?i)\b(api[_-]?key|secret|token|password|passwd|pwd|access[_-]?key)\b'
|
||||
r'\s*[:=]\s*[\'"]?([A-Za-z0-9+/_.\-]{12,})[\'"]?'
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
findings = []
|
||||
|
||||
def emit_fail(desc, fpath, lineno, why, fix):
|
||||
findings.append((desc, fpath, lineno, why, fix))
|
||||
|
||||
for fpath in find_agents_md(repo_root):
|
||||
rel = os.path.relpath(fpath, repo_root)
|
||||
with open(fpath, encoding="utf-8", errors="replace") as f:
|
||||
lines = f.readlines()
|
||||
for i, line in enumerate(lines, start=1):
|
||||
if PLACEHOLDER_RE.search(line):
|
||||
continue
|
||||
for label, pattern in PATTERNS:
|
||||
m = pattern.search(line)
|
||||
if not m:
|
||||
continue
|
||||
# Re-check placeholder allowlist against just the matched value, in case
|
||||
# the placeholder marker sits outside the regex's own match span.
|
||||
value = m.group(0)
|
||||
if PLACEHOLDER_RE.search(value):
|
||||
continue
|
||||
emit_fail(
|
||||
f"Possible {label}",
|
||||
f"{rel}:{i}",
|
||||
i,
|
||||
"AGENTS.md is committed content; this line matches a real-looking credential pattern rather than a placeholder.",
|
||||
"Remove the embedded credential and replace it with an environment variable reference or placeholder (e.g. $API_KEY, <your-token>).",
|
||||
)
|
||||
break
|
||||
|
||||
if not findings:
|
||||
sys.exit(0)
|
||||
|
||||
for desc, fpath, _lineno, why, fix in findings:
|
||||
print(f"FAIL {desc} — {fpath}")
|
||||
print(f" Why: {why}")
|
||||
print(f" Fix: {fix}")
|
||||
print()
|
||||
|
||||
sys.exit(1)
|
||||
PYTHON
|
||||
118
plugins/core/skills/agentsmd-audit/scripts/validate-structure.sh
Executable file
118
plugins/core/skills/agentsmd-audit/scripts/validate-structure.sh
Executable file
@@ -0,0 +1,118 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: validate-structure.sh <repo-root>
|
||||
|
||||
Check every AGENTS.md file in a repo (root and nested) for structural
|
||||
completeness against the agents.md spec's common-sections checklist
|
||||
(setup/build, code style, testing, security, commit/PR conventions).
|
||||
Missing individual sections are informational (not every repo needs every
|
||||
section) — only an empty or entirely unfilled file is a hard failure.
|
||||
|
||||
Arguments:
|
||||
repo-root Path to the repository root to scan.
|
||||
|
||||
Exit codes:
|
||||
0 No FAIL findings (INFO/SUGGESTION may still be printed)
|
||||
1 One or more FAIL findings
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "Error: repo-root is required." >&2
|
||||
echo "" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 -u - "$1" <<'PYTHON'
|
||||
import sys
|
||||
import os
|
||||
import re
|
||||
|
||||
PLACEHOLDER_RE = re.compile(r'(?i)FILL IN:|TODO:\s*write|lorem ipsum')
|
||||
|
||||
COMMON_SECTIONS = [
|
||||
("setup/build commands", re.compile(r'(?im)^#{1,3}\s*(setup|install|build|getting started)')),
|
||||
("code style", re.compile(r'(?im)^#{1,3}\s*(code style|style guide|conventions)')),
|
||||
("testing instructions", re.compile(r'(?im)^#{1,3}\s*(test|testing)')),
|
||||
("security considerations", re.compile(r'(?im)^#{1,3}\s*security')),
|
||||
("commit/PR conventions", re.compile(r'(?im)^#{1,3}\s*(commit|pr|pull request)')),
|
||||
]
|
||||
|
||||
repo_root = os.path.abspath(sys.argv[1])
|
||||
if not os.path.isdir(repo_root):
|
||||
print(f"Error: '{repo_root}' is not a directory.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
EXCLUDE_DIRS = {".git", "node_modules", "vendor", ".venv", "venv", "dist", "build"}
|
||||
|
||||
def find_agents_md(root):
|
||||
results = []
|
||||
for dirpath, dirnames, filenames in os.walk(root):
|
||||
dirnames[:] = [d for d in dirnames if d not in EXCLUDE_DIRS and not d.startswith(".")]
|
||||
for fname in filenames:
|
||||
if fname == "AGENTS.md":
|
||||
results.append(os.path.join(dirpath, fname))
|
||||
return sorted(results)
|
||||
|
||||
has_fail = False
|
||||
file_contents = {} # rel path -> content, for the duplication pass below
|
||||
|
||||
for fpath in find_agents_md(repo_root):
|
||||
rel = os.path.relpath(fpath, repo_root)
|
||||
with open(fpath, encoding="utf-8", errors="replace") as f:
|
||||
content = f.read()
|
||||
file_contents[rel] = content
|
||||
|
||||
if not content.strip():
|
||||
has_fail = True
|
||||
print(f"FAIL AGENTS.md is empty — {rel}")
|
||||
print(" Why: An empty file provides no instructions and gives agents nothing to act on.")
|
||||
print(" Fix: Add at least a project overview and setup/test commands, per the agents.md common-sections checklist.")
|
||||
print()
|
||||
continue
|
||||
|
||||
if PLACEHOLDER_RE.search(content):
|
||||
has_fail = True
|
||||
print(f"FAIL Unfilled placeholder content — {rel}")
|
||||
print(" Why: A 'FILL IN:' or template stub left in place means the file has no repo-specific instructions yet.")
|
||||
print(" Fix: Replace the placeholder with real, repo-specific content.")
|
||||
print()
|
||||
continue
|
||||
|
||||
for label, pattern in COMMON_SECTIONS:
|
||||
if not pattern.search(content):
|
||||
print(f"INFO No {label} section — {rel}")
|
||||
print(f" Note: The agents.md common-sections checklist includes {label}; not every repo needs every section, but confirm this omission is deliberate.")
|
||||
print()
|
||||
|
||||
# --- Nested-vs-root duplication check ---
|
||||
root_content = file_contents.get("AGENTS.md")
|
||||
if root_content:
|
||||
root_lines = {ln.strip() for ln in root_content.splitlines() if ln.strip()}
|
||||
for rel, content in file_contents.items():
|
||||
if rel == "AGENTS.md":
|
||||
continue
|
||||
nested_lines = [ln.strip() for ln in content.splitlines() if ln.strip()]
|
||||
if not nested_lines:
|
||||
continue
|
||||
overlap = sum(1 for ln in nested_lines if ln in root_lines)
|
||||
ratio = overlap / len(nested_lines)
|
||||
if ratio >= 0.7:
|
||||
print(f"SUGGESTION Nested AGENTS.md largely duplicates the root file — {rel}")
|
||||
print(f" Why: {ratio:.0%} of this file's content lines already appear in the root AGENTS.md; per the spec's nearest-file-wins precedence, nested files don't inherit from the root, but they also shouldn't just restate it.")
|
||||
print(f" Fix: Trim {rel} down to only what's specific to this package/directory.")
|
||||
print()
|
||||
|
||||
if has_fail:
|
||||
sys.exit(1)
|
||||
sys.exit(0)
|
||||
PYTHON
|
||||
30
plugins/core/skills/agentsmd-audit/tests/README.md
Normal file
30
plugins/core/skills/agentsmd-audit/tests/README.md
Normal file
@@ -0,0 +1,30 @@
|
||||
# tests/
|
||||
|
||||
Test files for scripts bundled with this skill.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Tests require [bats-support](https://github.com/bats-core/bats-support) and
|
||||
[bats-assert](https://github.com/bats-core/bats-assert). The test files load
|
||||
helpers from the repo root's `tests/test_helper/`.
|
||||
|
||||
From the repo root:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/bats-core/bats-support tests/test_helper/bats-support
|
||||
git clone https://github.com/bats-core/bats-assert tests/test_helper/bats-assert
|
||||
```
|
||||
|
||||
Run all tests for this skill (from the repo root):
|
||||
|
||||
```bash
|
||||
bats plugins/core/skills/agentsmd-audit/tests/
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `validate-secrets.bats` | Bats test suite for `scripts/validate-secrets.sh` |
|
||||
| `validate-structure.bats` | Bats test suite for `scripts/validate-structure.sh` |
|
||||
| `validate-drift.bats` | Bats test suite for `scripts/validate-drift.sh` |
|
||||
105
plugins/core/skills/agentsmd-audit/tests/validate-drift.bats
Normal file
105
plugins/core/skills/agentsmd-audit/tests/validate-drift.bats
Normal file
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
setup() {
|
||||
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-drift.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMPDIR"
|
||||
}
|
||||
|
||||
@test "fails when AGENTS.md references a stale npm script" {
|
||||
cat > "$TMPDIR/package.json" <<'EOF'
|
||||
{
|
||||
"scripts": {
|
||||
"test": "jest"
|
||||
}
|
||||
}
|
||||
EOF
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Testing
|
||||
- Run `pnpm run e2e` before committing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "e2e"
|
||||
}
|
||||
|
||||
@test "passes when the referenced npm script exists" {
|
||||
cat > "$TMPDIR/package.json" <<'EOF'
|
||||
{
|
||||
"scripts": {
|
||||
"test": "jest"
|
||||
}
|
||||
}
|
||||
EOF
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Testing
|
||||
- Run `npm run test` before committing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
}
|
||||
|
||||
@test "fails when AGENTS.md references a stale make target" {
|
||||
cat > "$TMPDIR/Makefile" <<'EOF'
|
||||
build:
|
||||
echo building
|
||||
EOF
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
Run `make deploy` to ship.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "deploy"
|
||||
}
|
||||
|
||||
@test "emits INFO instead of FAIL when there is no package.json to verify an npm script against" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Testing
|
||||
Run `pnpm run e2e` before committing.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
assert_output --partial "INFO"
|
||||
assert_output --partial "e2e"
|
||||
}
|
||||
|
||||
@test "fails when a referenced file path does not exist" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
See `scripts/bootstrap.sh` for environment setup.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "scripts/bootstrap.sh"
|
||||
}
|
||||
|
||||
@test "passes when the referenced file path exists" {
|
||||
mkdir -p "$TMPDIR/scripts"
|
||||
: > "$TMPDIR/scripts/bootstrap.sh"
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
See `scripts/bootstrap.sh` for environment setup.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
setup() {
|
||||
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-secrets.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMPDIR"
|
||||
}
|
||||
|
||||
@test "passes on AGENTS.md with no secrets, only placeholders" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
- Set `export API_KEY=$API_KEY`
|
||||
- Token: <your-token-here>
|
||||
- DB: postgres://user:changeme@localhost/db
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
@test "fails on a real-looking AWS access key" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
- AWS_ACCESS_KEY_ID=AKIAABCDEFGHIJKLMNOP # gitleaks:allow (synthetic fixture — this test verifies validate-secrets.sh catches exactly this pattern)
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "AWS access key ID"
|
||||
assert_output --partial "AGENTS.md:4"
|
||||
}
|
||||
|
||||
@test "fails on a credential-bearing connection string" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
- DB: postgres://svc_user:h8x2Klm9pQrT@db.internal:5432/prod
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "connection string"
|
||||
}
|
||||
|
||||
@test "detects secrets in a nested AGENTS.md, not just root" {
|
||||
mkdir -p "$TMPDIR/packages/api"
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
Clean root file.
|
||||
EOF
|
||||
cat > "$TMPDIR/packages/api/AGENTS.md" <<'EOF'
|
||||
# API package
|
||||
- token: ghp_1234567890abcdefghijklmnopqrstuvwxyz01 # gitleaks:allow (synthetic fixture)
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "packages/api/AGENTS.md"
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env bats
|
||||
|
||||
setup() {
|
||||
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../" && pwd)"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-support/load"
|
||||
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
|
||||
|
||||
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-structure.sh"
|
||||
TMPDIR="$(mktemp -d)"
|
||||
}
|
||||
|
||||
teardown() {
|
||||
rm -rf "$TMPDIR"
|
||||
}
|
||||
|
||||
@test "fails on an empty AGENTS.md" {
|
||||
: > "$TMPDIR/AGENTS.md"
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "empty"
|
||||
}
|
||||
|
||||
@test "fails on an unfilled placeholder AGENTS.md" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup
|
||||
FILL IN: describe setup commands here.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_failure
|
||||
assert_output --partial "placeholder"
|
||||
}
|
||||
|
||||
@test "passes with INFO on real content missing an optional section" {
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup commands
|
||||
- Install deps: `pnpm install`
|
||||
- Run tests: `pnpm test`
|
||||
|
||||
## Code style
|
||||
- TypeScript strict mode, single quotes, no semicolons.
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
assert_output --partial "INFO"
|
||||
assert_output --partial "security"
|
||||
}
|
||||
|
||||
@test "suggests trimming a nested AGENTS.md that duplicates the root file" {
|
||||
mkdir -p "$TMPDIR/packages/api"
|
||||
cat > "$TMPDIR/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup commands
|
||||
- Install deps: `pnpm install`
|
||||
- Run tests: `pnpm test`
|
||||
- Lint: `pnpm lint`
|
||||
- Build: `pnpm build`
|
||||
EOF
|
||||
cat > "$TMPDIR/packages/api/AGENTS.md" <<'EOF'
|
||||
# AGENTS.md
|
||||
|
||||
## Setup commands
|
||||
- Install deps: `pnpm install`
|
||||
- Run tests: `pnpm test`
|
||||
- Lint: `pnpm lint`
|
||||
- Build: `pnpm build`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$TMPDIR"
|
||||
assert_success
|
||||
assert_output --partial "SUGGESTION"
|
||||
assert_output --partial "packages/api/AGENTS.md"
|
||||
}
|
||||
Reference in New Issue
Block a user