Add Vale as deterministic pre-filter for skill-audit/agent-audit #84

New Issue

Claude · 2026-07-23T18:26:23Z

Claude commented

2026-07-23 18:26:23 +00:00

Context

skill-audit and agent-audit currently do structural validation via scripts/validate.sh (deterministic) plus a qualitative LLM pass covering: Description, Body discipline, Patterns, File structure, Formatting, Scripts, Internal consistency. Some of the qualitative checks are pattern-matchable prose rules rather than genuine judgment calls, and per this repo's governance principle ("prefer deterministic code for repeatable tasks"), those should move to a scripted/linted check instead of being re-derived by an LLM every audit run.

Vale is a prose linter that supports scoping rules to specific Markdown YAML frontmatter fields (e.g. scope: text.frontmatter.description), confirmed via its docs at vale.sh/docs/formats/front-matter. This makes it viable to lint the description: field of SKILL.md/agent .md files directly, not just body prose.

Proposed split

Move to Vale (deterministic, pattern-based):

Description imperative phrasing — existence rule banning "This skill..." openers / requiring a "Use when" style opener, scoped to text.frontmatter.description
Vague-capability wording — existence/substitution rule against a blocklist ("helps with", "utilize", generic "handles X")
Known padding phrases in the body — e.g. "see references/ for details" (vs the required "If X, read Y" form)
Agent .md/.agent.md description field — same text.frontmatter.description scope Vale generates for any YAML frontmatter key (confirmed via vale.sh/docs/formats/front-matter), not just SKILL.md's. Covers: the Copilot-only "Use proactively has no effect, remove it" check (existence/substitution rule), and banning known-bad openers ("This agent...") via existence rule.
Known literal padding phrases (e.g. "see references/ for details") — same rule proposed for skill bodies, reused verbatim since agent bodies contain the identical phrase pattern.

Explicitly NOT moving to Vale — stays in the LLM qualitative audit:

Body discipline's core test ("would the agent get this wrong without this sentence?") — requires modeling what an agent already knows, not pattern-matchable
Whether near-miss exclusions target a genuinely strong near-miss vs a weak one
Control calibration (prescriptive vs flexible correctly matched to context)
Internal consistency (SKILL.md claims match what scripts actually do) — requires comparing meaning across two artifacts
Formatting nits (heading level consistency, fenced code language tags) — belongs to markdownlint, not Vale; out of scope for this issue
Agent description's "action-verb opening" check — banning bad openers is Vale-able, but verifying an arbitrary word is genuinely a strong verb isn't pattern-matchable; stays LLM judgment.

Proposed implementation (for later)

Add .vale.ini + a custom style scoped to plugins/*/skills/*/SKILL.md and the agent-audit equivalent path
Wire vale --config .vale.ini <target> into skill-audit Step 1 (structural validation) and agent-audit's equivalent step, alongside the existing validate.sh/validate-provenance.sh scripts
Update skill-audit/SKILL.md and agent-audit/SKILL.md "Description" dimension instructions to defer to Vale's output instead of re-deriving those specific checks via LLM judgment, keeping the rest of the qualitative dimensions unchanged

Also investigated, rejected

Agent body's "You are a [role]. When invoked, [action]." opening-line check — Vale has no first-line/paragraph scope selector, so a rule could only confirm the phrase exists somewhere, not that it's positioned as the opening line. Belongs to validate.sh's structural checks, not Vale.
plugin.json/marketplace.json description fields — content is equally pattern-matchable, but Vale has no native JSON format support (no comments, not a markup format); would need a jq-extraction shim rather than native scoping. Dropped as not worth the implementation overhead.
hooks.json — no prose content in any observed instance; not a lint candidate.
README.md/docs/ADRs — plain markdown, would lint fine, but already explicitly out of scope per this issue's original "full repo-wide Vale rollout" exclusion; also, skill-audit's actual README checks are file-vs-directory consistency checks, not prose style, so they wouldn't move to Vale regardless.

Out of scope for this issue

Full repo-wide Vale rollout (docs/, ADRs, README files) — this issue is scoped to skill-audit/agent-audit only
Replacing any part of Body Discipline or Internal Consistency dimensions

## Context `skill-audit` and `agent-audit` currently do structural validation via `scripts/validate.sh` (deterministic) plus a qualitative LLM pass covering: Description, Body discipline, Patterns, File structure, Formatting, Scripts, Internal consistency. Some of the qualitative checks are pattern-matchable prose rules rather than genuine judgment calls, and per this repo's governance principle ("prefer deterministic code for repeatable tasks"), those should move to a scripted/linted check instead of being re-derived by an LLM every audit run. [Vale](https://vale.sh) is a prose linter that supports scoping rules to specific Markdown YAML frontmatter fields (e.g. `scope: text.frontmatter.description`), confirmed via its docs at vale.sh/docs/formats/front-matter. This makes it viable to lint the `description:` field of SKILL.md/agent `.md` files directly, not just body prose. ## Proposed split **Move to Vale (deterministic, pattern-based):** - Description imperative phrasing — existence rule banning "This skill..." openers / requiring a "Use when" style opener, scoped to `text.frontmatter.description` - Vague-capability wording — existence/substitution rule against a blocklist ("helps with", "utilize", generic "handles X") - Known padding phrases in the body — e.g. "see references/ for details" (vs the required "If X, read Y" form) - Agent `.md`/`.agent.md` `description` field — same `text.frontmatter.description` scope Vale generates for any YAML frontmatter key (confirmed via vale.sh/docs/formats/front-matter), not just SKILL.md's. Covers: the Copilot-only "`Use proactively` has no effect, remove it" check (existence/substitution rule), and banning known-bad openers ("This agent...") via existence rule. - Known literal padding phrases (e.g. "see references/ for details") — same rule proposed for skill bodies, reused verbatim since agent bodies contain the identical phrase pattern. **Explicitly NOT moving to Vale — stays in the LLM qualitative audit:** - Body discipline's core test ("would the agent get this wrong without this sentence?") — requires modeling what an agent already knows, not pattern-matchable - Whether near-miss exclusions target a genuinely strong near-miss vs a weak one - Control calibration (prescriptive vs flexible correctly matched to context) - Internal consistency (SKILL.md claims match what scripts actually do) — requires comparing meaning across two artifacts - Formatting nits (heading level consistency, fenced code language tags) — belongs to markdownlint, not Vale; out of scope for this issue - Agent description's "action-verb opening" check — banning bad openers is Vale-able, but verifying an arbitrary word is genuinely a strong verb isn't pattern-matchable; stays LLM judgment. ## Proposed implementation (for later) - Add `.vale.ini` + a custom style scoped to `plugins/*/skills/*/SKILL.md` and the agent-audit equivalent path - Wire `vale --config .vale.ini <target>` into skill-audit Step 1 (structural validation) and agent-audit's equivalent step, alongside the existing `validate.sh`/`validate-provenance.sh` scripts - Update `skill-audit/SKILL.md` and `agent-audit/SKILL.md` "Description" dimension instructions to defer to Vale's output instead of re-deriving those specific checks via LLM judgment, keeping the rest of the qualitative dimensions unchanged ## Also investigated, rejected - Agent body's "You are a [role]. When invoked, [action]." opening-line check — Vale has no first-line/paragraph scope selector, so a rule could only confirm the phrase exists somewhere, not that it's positioned as the opening line. Belongs to `validate.sh`'s structural checks, not Vale. - `plugin.json`/`marketplace.json` `description` fields — content is equally pattern-matchable, but Vale has no native JSON format support (no comments, not a markup format); would need a `jq`-extraction shim rather than native scoping. Dropped as not worth the implementation overhead. - `hooks.json` — no prose content in any observed instance; not a lint candidate. - README.md/docs/ADRs — plain markdown, would lint fine, but already explicitly out of scope per this issue's original "full repo-wide Vale rollout" exclusion; also, skill-audit's actual README checks are file-vs-directory consistency checks, not prose style, so they wouldn't move to Vale regardless. ## Out of scope for this issue - Full repo-wide Vale rollout (docs/, ADRs, README files) — this issue is scoped to skill-audit/agent-audit only - Replacing any part of Body Discipline or Internal Consistency dimensions

Claude added the Kind/Enhancement

Priority

Low

4

labels 2026-07-23 18:26:23 +00:00

Defame1297 commented

2026-07-23 18:28:26 +00:00

Before we implement this, we need to research the vale documentation first. Also, I want to check whether we can validate other plugin elements with vale

Defame1297 referenced this issue from a commit

2026-07-23 18:56:46 +00:00

docs(kyberforge): add vale.sh research docs

Defame1297 referenced this issue from a commit

2026-07-23 20:40:19 +00:00

feat(kyberforge): wire Vale as deterministic prefilter for skill-audit/agent-audit

Defame1297 closed this issue

2026-07-23 20:40:19 +00:00

Claude referenced a pull request that will close this issue

2026-07-23 20:54:04 +00:00

feat(lint): wire Vale as deterministic prefilter for skill-audit/agent-audit #85

Claude referenced this issue