feat(agent-audit): implement ADR-0016's reserved SUGGESTION for unenforceable tool-restriction claims #126

New Issue

Claude · 2026-08-31T08:07:57Z

Claude commented

2026-08-31 08:07:57 +00:00

Found during the #99 ADR-0020 retrofit and the follow-up audit pass on refactor/adr0020-skill-retrofit. Not fixed there — it is new gate behaviour ADR-0016 already reserved and explicitly deferred, not a defect introduced by the retrofit.

What the ADR reserved

docs/adr/0016-apm-agent-primitive-drops-provider-specific-fields.md:50, verbatim:

agent-audit, at plugin scope, is intended to flag — as a SUGGESTION, not a FAIL, since this is an upstream schema limitation rather than an authoring mistake — any agent whose description or body implies a need for tool restriction or a Claude-only behavior the frontmatter can no longer express. This would give visibility into the gap without pretending the schema can do something it can't. Not yet implemented: check_apm_agent_file() in validate.sh currently validates only the field allowlist, name, description, and body-emptiness/length — it has no heuristic for this case. Tracked as follow-up work.

Still accurate. Reading check_apm_agent_file() at plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh:1391 in the working tree, the function checks: unreadable file, unparsable frontmatter, leftover template HTML comments, the field allowlist, name (present / kebab-case / matches filename stem), description (present / no placeholder / ADR-0020 budget / boundary clause), and body (non-empty / no placeholder / Copilot truncation limit). No heuristic for prose-asserted tool restriction.

The SUGGESTION tier itself is already wired — suggest() is used in the same function for the Copilot body-length case — so this is a new check on existing machinery, not new plumbing.

The concrete instance

plugins/kyberforge/.apm/agents/apm-orchestrate.agent.md promises, in its Hard rules:

You are read-only against the working tree. Never create, edit, or delete a file — not an apm.yml, not a .apm/ primitive, not compiled output, not a scratch note.

Its frontmatter carries disallowedTools: Edit, Write, NotebookEdit, which is the maximum ADR-0016 permits: the 2026-08-14 amendment readmits the denylist and keeps the allowlist-shaped tools: excluded, because Claude Code and Copilot read tools: from incompatible vocabularies. An absent tools: means inherit-all on both harnesses.

So Bash remains available — and the agent's own Process step 4 requires it (Confirm apm --version succeeds before dispatching any operation). A heredoc, a > redirect, or apm compile writes files. The guarantee in that Hard rule holds by prose only. Nothing in the frontmatter enforces it and nothing can, which is precisely the visibility gap the ADR reserved a SUGGESTION for.

What implementing it would surface today

Three of the four plugin-scope agents, all carrying the same shape — an explicit never-writes promise plus disallowedTools: Edit, Write, NotebookEdit:

agent	the prose claim
`kyberforge/apm-orchestrate`	"You are read-only against the working tree. Never create, edit, or delete a file…"
`gitea/gitea-orchestrate`	"You are read-only against the local working tree. Never create, edit, or delete a local file…"
`lint/lint-runner`	"You never edit files." plus "wants findings back, not fixes applied" in the description

git/git-orchestrate would not be surfaced, and correctly so: it carries no disallowedTools and no never-writes claim. Its "you do not execute git logic yourself, you delegate to domain skills" is a routing statement, not a tool-restriction claim.

That distribution is a useful property for calibration — a heuristic that fires on all four is too loose.

Design note

The cheapest defensible heuristic is the conjunction, not either half alone: an agent that both (a) makes a never-writes / read-only claim in description or body, and (b) has no way to enforce it — either disallowedTools is absent, or it is present but Bash (unrestricted) remains, which under ADR-0016 it always does. Firing on (a) alone would flag correct authoring; firing on (b) alone would flag every agent in the repo.

The ADR's wording is broader than tool restriction ("or a Claude-only behavior the frontmatter can no longer express"). That second half has no live instance in the corpus right now and can be deferred; the tool-restriction half has three.

Files

plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh:1391 — check_apm_agent_file()
docs/adr/0016-apm-agent-primitive-drops-provider-specific-fields.md:50 — the reservation, and the 2026-08-14 amendment that fixes disallowedTools as the ceiling
plugins/{kyberforge,gitea,lint}/.apm/agents/*.agent.md — the three instances

#120 — the other apm-orchestrate findings from this pass
#99 — the retrofit during which this surfaced

Found during the #99 ADR-0020 retrofit and the follow-up audit pass on `refactor/adr0020-skill-retrofit`. Not fixed there — it is new gate behaviour ADR-0016 already reserved and explicitly deferred, not a defect introduced by the retrofit. ## What the ADR reserved `docs/adr/0016-apm-agent-primitive-drops-provider-specific-fields.md:50`, verbatim: > `agent-audit`, at plugin scope, is intended to flag — as a **SUGGESTION**, not a FAIL, since this is an upstream schema limitation rather than an authoring mistake — any agent whose description or body implies a need for tool restriction or a Claude-only behavior the frontmatter can no longer express. This would give visibility into the gap without pretending the schema can do something it can't. **Not yet implemented**: `check_apm_agent_file()` in `validate.sh` currently validates only the field allowlist, `name`, `description`, and body-emptiness/length — it has no heuristic for this case. Tracked as follow-up work. Still accurate. Reading `check_apm_agent_file()` at `plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh:1391` in the working tree, the function checks: unreadable file, unparsable frontmatter, leftover template HTML comments, the field allowlist, `name` (present / kebab-case / matches filename stem), `description` (present / no placeholder / ADR-0020 budget / boundary clause), and body (non-empty / no placeholder / Copilot truncation limit). No heuristic for prose-asserted tool restriction. The SUGGESTION tier itself is already wired — `suggest()` is used in the same function for the Copilot body-length case — so this is a new check on existing machinery, not new plumbing. ## The concrete instance `plugins/kyberforge/.apm/agents/apm-orchestrate.agent.md` promises, in its Hard rules: > - You are read-only against the working tree. Never create, edit, or delete a file — not an `apm.yml`, not a `.apm/` primitive, not compiled output, not a scratch note. Its frontmatter carries `disallowedTools: Edit, Write, NotebookEdit`, which is the **maximum** ADR-0016 permits: the 2026-08-14 amendment readmits the denylist and keeps the allowlist-shaped `tools:` excluded, because Claude Code and Copilot read `tools:` from incompatible vocabularies. An absent `tools:` means inherit-all on both harnesses. So `Bash` remains available — and the agent's own Process step 4 *requires* it (`Confirm `apm --version` succeeds before dispatching any operation`). A heredoc, a `>` redirect, or `apm compile` writes files. The guarantee in that Hard rule holds by prose only. Nothing in the frontmatter enforces it and nothing can, which is precisely the visibility gap the ADR reserved a SUGGESTION for. ## What implementing it would surface today Three of the four plugin-scope agents, all carrying the same shape — an explicit never-writes promise plus `disallowedTools: Edit, Write, NotebookEdit`: | agent | the prose claim | |---|---| | `kyberforge/apm-orchestrate` | "You are read-only against the working tree. Never create, edit, or delete a file…" | | `gitea/gitea-orchestrate` | "You are read-only against the local working tree. Never create, edit, or delete a local file…" | | `lint/lint-runner` | "You never edit files." plus "wants findings back, not fixes applied" in the description | `git/git-orchestrate` would **not** be surfaced, and correctly so: it carries no `disallowedTools` and no never-writes claim. Its "you do not execute git logic yourself, you delegate to domain skills" is a routing statement, not a tool-restriction claim. That distribution is a useful property for calibration — a heuristic that fires on all four is too loose. ## Design note The cheapest defensible heuristic is the conjunction, not either half alone: an agent that both (a) makes a never-writes / read-only claim in description or body, and (b) has no way to enforce it — either `disallowedTools` is absent, or it is present but `Bash` (unrestricted) remains, which under ADR-0016 it always does. Firing on (a) alone would flag correct authoring; firing on (b) alone would flag every agent in the repo. The ADR's wording is broader than tool restriction ("or a Claude-only behavior the frontmatter can no longer express"). That second half has no live instance in the corpus right now and can be deferred; the tool-restriction half has three. ## Files - `plugins/kyberforge/.apm/skills/agent-audit/scripts/validate.sh:1391` — `check_apm_agent_file()` - `docs/adr/0016-apm-agent-primitive-drops-provider-specific-fields.md:50` — the reservation, and the 2026-08-14 amendment that fixes `disallowedTools` as the ceiling - `plugins/{kyberforge,gitea,lint}/.apm/agents/*.agent.md` — the three instances ## Related - #120 — the other `apm-orchestrate` findings from this pass - #99 — the retrofit during which this surfaced

Claude added this to the Skills & Agents milestone 2026-08-31 08:07:57 +00:00

Claude added the Kind/Enhancement

labels 2026-08-31 08:07:57 +00:00

Defame1297 commented

2026-09-01 20:15:25 +00:00

Wont fix, not a significant enough issue