feat(kyberforge): wire Vale as deterministic prefilter for skill-audit/agent-audit
Adds repo-root .vale.ini plus a custom Kyberforge style (description-opener, vague-wording, and generic reference-pointer padding rules) and a KyberforgeCopilot style scoped to .agent.md files (Use proactively check). skill-audit and agent-audit Step 1 now run vale against the specific file(s) being audited and defer the corresponding Description/Patterns/Body checks to its output instead of re-deriving them by LLM judgment, per the split proposed in issue #84. Closes #84 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FxG5T8EJDgkABXxuneuFfn
This commit is contained in:
11
.vale.ini
Normal file
11
.vale.ini
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
StylesPath = styles
|
||||||
|
MinAlertLevel = suggestion
|
||||||
|
|
||||||
|
[plugins/*/skills/*/SKILL.md]
|
||||||
|
BasedOnStyles = Kyberforge
|
||||||
|
|
||||||
|
[plugins/*/agents/*.md]
|
||||||
|
BasedOnStyles = Kyberforge
|
||||||
|
|
||||||
|
[plugins/*/agents/*.agent.md]
|
||||||
|
BasedOnStyles = Kyberforge, KyberforgeCopilot
|
||||||
@@ -67,7 +67,10 @@ A skill pair in the `core` plugin for writing, updating, and reviewing a target
|
|||||||
A companion skill (`core` plugin) that detects a target repo's provider-specific instruction file (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) and, where it duplicates content AGENTS.md should own, converts it into a thin adapter that imports AGENTS.md — mirroring this repo's own ADR-0002/ADR-0003 two-tier adapter pattern. Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`: checks for an import reference, no duplicated headings, size threshold) rather than a separate paired audit skill — the check is mechanical, so a script suffices per governance.md's "prefer deterministic code for repeatable tasks." `agentsmd-author` calls this skill via skill composition when it detects an existing provider file with overlapping content.
|
A companion skill (`core` plugin) that detects a target repo's provider-specific instruction file (`CLAUDE.md`, `.cursor/rules/*.mdc`, `copilot-instructions.md`, etc.) and, where it duplicates content AGENTS.md should own, converts it into a thin adapter that imports AGENTS.md — mirroring this repo's own ADR-0002/ADR-0003 two-tier adapter pattern. Self-validates via its own bundled deterministic script (`scripts/validate-adapter.sh`: checks for an import reference, no duplicated headings, size threshold) rather than a separate paired audit skill — the check is mechanical, so a script suffices per governance.md's "prefer deterministic code for repeatable tasks." `agentsmd-author` calls this skill via skill composition when it detects an existing provider file with overlapping content.
|
||||||
|
|
||||||
### lint plugin
|
### lint plugin
|
||||||
A standalone, repo-agnostic plugin (`plugins/lint/`) for configuring and running linters — not scoped to kyberforge's own meta-tooling, and not (yet) wired into `skill-audit`/`agent-audit`. First linter is Vale (prose style linting), split into two skills per the git/gitea per-concern pattern: `vale-config` (setup — `.vale.ini`, `StylesPath`, styles) and `vale-run` (invoke Vale, interpret/report findings). A `lint-runner` agent composes these for isolated-context lint sweeps; it is report-only (no `Edit` tool) — it flags findings, it does not rewrite prose. Wiring Vale into the audit pipeline as a prefilter (the original motivation captured in branch `feat/84-vale-audit-prefilter`) is deferred to a follow-up task once this plugin exists standalone. Vale's research docs (`docs/research/docs/vale/`) moved from `plugins/kyberforge/` to `plugins/lint/` to keep the provenance chain same-plugin.
|
A standalone, repo-agnostic plugin (`plugins/lint/`) for configuring and running linters — not scoped to kyberforge's own meta-tooling. First linter is Vale (prose style linting), split into two skills per the git/gitea per-concern pattern: `vale-config` (setup — `.vale.ini`, `StylesPath`, styles) and `vale-run` (invoke Vale, interpret/report findings). A `lint-runner` agent composes these for isolated-context lint sweeps; it is report-only (no `Edit` tool) — it flags findings, it does not rewrite prose. Vale's research docs (`docs/research/docs/vale/`) moved from `plugins/kyberforge/` to `plugins/lint/` to keep the provenance chain same-plugin.
|
||||||
|
|
||||||
|
### Vale audit prefilter (skill-audit / agent-audit)
|
||||||
|
Wiring Vale as a deterministic prefilter for `skill-audit`/`agent-audit`'s Description dimension (ADR motivation: issue #84) is repo-specific, not part of the generic `lint` plugin, so its config lives at the repo root rather than inside `plugins/lint/`: `.vale.ini` plus a custom `Kyberforge` style (`styles/Kyberforge/`) covering description-opener banning ("This skill/agent..."), vague-capability wording ("helps with", "utilize", ...), and generic "see references/ for details" padding — and a `KyberforgeCopilot` style (`styles/KyberforgeCopilot/`) scoped only to `.agent.md` files for the Copilot-only "Use proactively has no effect" check. Both skills' Step 1 run `vale --config .vale.ini <target>` against the specific file(s) being audited (never a repo-wide sweep) — Vale's glob matching crosses directory boundaries (`plugins/*/agents/*.md` matches nested `docs/research/examples/**/agents/*.md` too), so scoping every invocation to a known target file/dir is what keeps research-example files out of the audit's lint pass rather than the glob pattern itself. `error` alerts map to FAIL, `warning`/`suggestion` map to SUGGESTION. Vale only replaces the specific pattern-matchable sub-checks named in issue #84 (imperative opener, vague filler, `Use proactively`, generic reference-pointer padding) — body discipline, near-miss exclusion strength, and control calibration stay LLM judgment per the issue's explicit non-goals.
|
||||||
|
|
||||||
### LESSONS.md
|
### LESSONS.md
|
||||||
Long-loop feedback log for patterns observed across sessions. Three or more entries on the same pattern graduate to the relevant standing file (e.g. a coding convention, a governance rule). Updated by the session-handoff skill or directly by the human. Lives at the repo root.
|
Long-loop feedback log for patterns observed across sessions. Three or more entries on the same pattern graduate to the relevant standing file (e.g. a coding convention, a governance rule). Updated by the session-handoff skill or directly by the human. Lives at the repo root.
|
||||||
|
|||||||
@@ -36,10 +36,13 @@ metadata:
|
|||||||
```bash
|
```bash
|
||||||
bash scripts/validate.sh <path-to-agent-file>
|
bash scripts/validate.sh <path-to-agent-file>
|
||||||
bash scripts/validate-provenance.sh <path-to-agent-file>
|
bash scripts/validate-provenance.sh <path-to-agent-file>
|
||||||
|
vale --config .vale.ini <path-to-cc-file> <path-to-copilot-file>
|
||||||
```
|
```
|
||||||
|
|
||||||
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
The script accepts either the CC file or the Copilot file. It detects provider from extension, derives the counterpart, and runs all structural checks. Note FAILs and SUGGESTIONs for the `### Structure` and `### Provider safety` report dimensions. Findings about missing fields, bad name format, empty body, or missing frontmatter → `### Structure`. Findings about CC-only fields in a Copilot file, Copilot-only fields in a CC file, plugin-silently-ignored fields, body length, or subagent-unavailable tools → `### Provider safety`. A missing counterpart file → `### Pair consistency`.
|
||||||
|
|
||||||
|
Run `vale` from the repo root, against both files of the pair (not just the one passed in), using the repo-root `.vale.ini`. It applies the `Kyberforge` style to both files and the `KyberforgeCopilot` style to the `.agent.md` file only — that split is how the Copilot-only `Use proactively` check stays scoped to the Copilot file without a manual per-file judgment call. Map `error` alerts to `FAIL` and `warning`/`suggestion` alerts to `SUGGESTION` in the `### Description` / `### Body` dimensions below, citing the rule ID (e.g. `KyberforgeCopilot.ProactivePhrase`). If `vale` is not installed or `.vale.ini` is missing, skip this and fall back to the manual judgment calls in Step 2 — do not block the audit on tooling absence.
|
||||||
|
|
||||||
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
`validate-provenance.sh` validates the provenance chain between the agent pair's `source_keys` and the plugin-scoped `sources.md` (plugin root — see ADR-0010). It exits 0 silently for non-plugin-scope agents and when no provenance data exists. Note FAILs from this script for the `### Provenance` dimension — surface them verbatim with Why and Fix.
|
||||||
|
|
||||||
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists).
|
If the scripts cannot run (Bash denied, python3 unavailable), perform checks manually: counterpart file exists, required fields present (`name`, `description`, non-empty body), `name` is kebab-case, Copilot CLI `.agent.md` `name` must match filename stem (CC files are exempt — the CC platform does not require name to match filename), no `FILL IN:` placeholders, no CC-only fields in Copilot file, no Copilot-only fields in CC file (read `references/field-inventory.md` for the authoritative field lists).
|
||||||
@@ -49,15 +52,16 @@ If the scripts cannot run (Bash denied, python3 unavailable), perform checks man
|
|||||||
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
|
Read both agent files. Work through each dimension internally. Collect findings only; report in Step 3.
|
||||||
|
|
||||||
**Description (both files):**
|
**Description (both files):**
|
||||||
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent
|
- Action-verb opening: description starts with a verb ("Reviews...", "Analyzes...", "Generates...") — FAIL if absent. Vale's `Kyberforge.DescriptionOpener` alert flags the specific known-bad "This agent..." opener directly; verifying an arbitrary opening word is genuinely a strong verb still requires judgment.
|
||||||
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague
|
- Specificity: is the trigger condition stated precisely? — SUGGESTION if vague. Vale's `Kyberforge.VagueWording` alert covers known filler ("helps with", "utilize", ...) directly; report those without re-deriving by judgment.
|
||||||
- `Use proactively` in a Copilot description: CC-specific phrasing, has no effect in Copilot — SUGGESTION to remove
|
- `Use proactively` in a Copilot description: Vale's `KyberforgeCopilot.ProactivePhrase` alert (Copilot file only) flags this directly — report it without re-deriving by judgment.
|
||||||
|
|
||||||
If a description finding is borderline, read `references/description-quality.md`.
|
If a description finding is borderline, read `references/description-quality.md`.
|
||||||
|
|
||||||
**Body:**
|
**Body:**
|
||||||
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
|
- Direct role instruction: system prompt opens with `You are a [role]. When invoked, [action].` — SUGGESTION if absent
|
||||||
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
|
- One job per agent: system prompt describes a single bounded task — SUGGESTION if scope appears unbounded
|
||||||
|
- Generic, non-specific reference pointers to the `references/` directory: Vale's `Kyberforge.PaddingPhrase` alert flags this directly — report it without re-deriving by judgment
|
||||||
|
|
||||||
**Body/Frontmatter comments:**
|
**Body/Frontmatter comments:**
|
||||||
- Inspect each comment block in the YAML frontmatter. For each comment, apply: *"Would the agent get this wrong without this comment?"* Flag any that answer "no" as padding.
|
- Inspect each comment block in the YAML frontmatter. For each comment, apply: *"Would the agent get this wrong without this comment?"* Flag any that answer "no" as padding.
|
||||||
|
|||||||
@@ -34,12 +34,15 @@ metadata:
|
|||||||
```bash
|
```bash
|
||||||
bash scripts/validate.sh <skill-dir>
|
bash scripts/validate.sh <skill-dir>
|
||||||
bash scripts/validate-provenance.sh <skill-dir>
|
bash scripts/validate-provenance.sh <skill-dir>
|
||||||
|
vale --config .vale.ini <skill-dir>/SKILL.md
|
||||||
```
|
```
|
||||||
|
|
||||||
Note any structural FAILs — they will appear in the report as a `### Structure` dimension. If the script cannot execute (python3 unavailable, Bash denied, or permission error), perform structural checks manually: name format, name matches directory, description length ≤1024 chars, SKILL.md ≤500 lines, no unfilled `FILL IN:` placeholders, scripts executable and free of interactive prompts.
|
Note any structural FAILs — they will appear in the report as a `### Structure` dimension. If the script cannot execute (python3 unavailable, Bash denied, or permission error), perform structural checks manually: name format, name matches directory, description length ≤1024 chars, SKILL.md ≤500 lines, no unfilled `FILL IN:` placeholders, scripts executable and free of interactive prompts.
|
||||||
|
|
||||||
Note any Provenance FAILs and INFO findings from `validate-provenance.sh` — they surface in the report as a `### Provenance` dimension (separate from `### Structure`). The script embeds full FAIL/INFO format with Why and Fix per finding; surface them verbatim.
|
Note any Provenance FAILs and INFO findings from `validate-provenance.sh` — they surface in the report as a `### Provenance` dimension (separate from `### Structure`). The script embeds full FAIL/INFO format with Why and Fix per finding; surface them verbatim.
|
||||||
|
|
||||||
|
`vale` runs from the repo root against the target `SKILL.md` using the repo-root `.vale.ini` (custom `Kyberforge` style) — it is a deterministic prefilter for a subset of the Description and Patterns dimensions below, not a replacement for Step 3's qualitative pass. Map `error` alerts to `FAIL` and `warning`/`suggestion` alerts to `SUGGESTION` in the `### Description` / `### Patterns` report dimensions, citing the rule ID (e.g. `Kyberforge.DescriptionOpener`) as the finding. If `vale` is not installed or `.vale.ini` is missing, skip this and fall back to the manual judgment calls described in Step 3 — do not block the audit on tooling absence.
|
||||||
|
|
||||||
## Step 2 — Read all skill files
|
## Step 2 — Read all skill files
|
||||||
|
|
||||||
Read every file in the skill directory: `SKILL.md`, `README.md` (if present), all files in `scripts/`, `references/`, `assets/`, and `tests/`. Skip binary files only. Do not skip text files — internal consistency checks require the full picture.
|
Read every file in the skill directory: `SKILL.md`, `README.md` (if present), all files in `scripts/`, `references/`, `assets/`, and `tests/`. Skip binary files only. Do not skip text files — internal consistency checks require the full picture.
|
||||||
@@ -50,8 +53,9 @@ Work through each dimension internally. Collect findings only; report them in St
|
|||||||
|
|
||||||
### Description
|
### Description
|
||||||
|
|
||||||
- **Imperative phrasing**: does it use "Use when..." not "This skill..."?
|
Vale's `Kyberforge.DescriptionOpener` (FAIL — "This skill..." openers) and `Kyberforge.VagueWording` (SUGGESTION — filler like "helps with", "utilize") alerts from Step 1 cover imperative phrasing and known vague-wording filler directly; report them as findings without re-deriving by judgment. The rest is still a judgment call:
|
||||||
- **Specificity**: are capabilities stated precisely ("parses OpenAPI specs") or vaguely ("helps with APIs")?
|
|
||||||
|
- **Specificity beyond the filler blocklist**: are capabilities stated precisely ("parses OpenAPI specs") or genuinely vaguely ("handles files")?
|
||||||
- **Indirect triggers**: does it cover cases where the user doesn't name the domain directly?
|
- **Indirect triggers**: does it cover cases where the user doesn't name the domain directly?
|
||||||
- **Near-miss exclusions**: are "Do not use when..." clauses present if a near-miss skill could steal activations?
|
- **Near-miss exclusions**: are "Do not use when..." clauses present if a near-miss skill could steal activations?
|
||||||
- **Length**: under 1024 characters?
|
- **Length**: under 1024 characters?
|
||||||
@@ -75,7 +79,7 @@ Check each pattern is appropriate and correctly formed:
|
|||||||
- **Gotchas**: placed near the top; each entry is a specific fact that defies a reasonable assumption — not a general tip
|
- **Gotchas**: placed near the top; each entry is a specific fact that defies a reasonable assumption — not a general tip
|
||||||
- **Prescriptive sequence**: inner code fences escaped as `\`\`\`` when nested inside a markdown block
|
- **Prescriptive sequence**: inner code fences escaped as `\`\`\`` when nested inside a markdown block
|
||||||
- **Checklists**: used for multi-step workflows, not single steps
|
- **Checklists**: used for multi-step workflows, not single steps
|
||||||
- **Conditional references**: specific trigger stated ("If X, read `references/file.md`") — not a generic "see references/"
|
- **Conditional references**: specific trigger stated ("If X, read `references/file.md`") — not a generic "see references/". Vale's `Kyberforge.PaddingPhrase` alert from Step 1 flags the generic phrasing directly; other malformed conditional-reference forms still require judgment.
|
||||||
- **Output templates**: present when the agent must produce a specific format; absent otherwise
|
- **Output templates**: present when the agent must produce a specific format; absent otherwise
|
||||||
|
|
||||||
### File structure
|
### File structure
|
||||||
|
|||||||
7
styles/Kyberforge/DescriptionOpener.yml
Normal file
7
styles/Kyberforge/DescriptionOpener.yml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
extends: existence
|
||||||
|
message: "Description opens with '%s' — use an imperative 'Use when...' opener instead"
|
||||||
|
level: error
|
||||||
|
scope: text.frontmatter.description
|
||||||
|
ignorecase: true
|
||||||
|
raw:
|
||||||
|
- '^This (skill|agent)\b'
|
||||||
7
styles/Kyberforge/PaddingPhrase.yml
Normal file
7
styles/Kyberforge/PaddingPhrase.yml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
extends: existence
|
||||||
|
message: "Generic reference pointer: '%s' — use the specific 'If X, read `references/file.md`' form instead"
|
||||||
|
level: warning
|
||||||
|
scope: text
|
||||||
|
ignorecase: true
|
||||||
|
raw:
|
||||||
|
- 'see references?/? for (more )?(info|information|details)\b'
|
||||||
10
styles/Kyberforge/VagueWording.yml
Normal file
10
styles/Kyberforge/VagueWording.yml
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
extends: existence
|
||||||
|
message: "Vague capability wording: '%s' — state the capability precisely instead"
|
||||||
|
level: warning
|
||||||
|
scope: text.frontmatter.description
|
||||||
|
ignorecase: true
|
||||||
|
tokens:
|
||||||
|
- helps with
|
||||||
|
- utilize
|
||||||
|
- assists with
|
||||||
|
- used for
|
||||||
7
styles/KyberforgeCopilot/ProactivePhrase.yml
Normal file
7
styles/KyberforgeCopilot/ProactivePhrase.yml
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
extends: existence
|
||||||
|
message: "'%s' is CC-specific phrasing with no effect in Copilot descriptions — remove it"
|
||||||
|
level: warning
|
||||||
|
scope: text.frontmatter.description
|
||||||
|
ignorecase: true
|
||||||
|
tokens:
|
||||||
|
- Use proactively
|
||||||
Reference in New Issue
Block a user