Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix(kyberforge): announce every provenance skip and scope checks 7-8 to source indexes

Browse Source 
The clean provenance bill was an artifact. Checks 7 and 8 assume `Research doc:`
names a source index whose H2s are slugs, but 30 of 121 corpus entries point at
topic content documents whose H2s are topics. Those 30 produced every new check-7
INFO — all false positives. Check 8 aimed at the same documents, which carry no
`Status:` line at all, would have emitted a large false-FAIL flood; the only thing
preventing it was an unannounced `rd_status != extracted` skip. So "0 new FAILs"
rested on exactly the fail-open class this branch exists to remove, and naively
fixing the skip would have turned the branch red.

Checks 7/8 now run only when the research doc's basename is `sources.md`, and every
other case emits a visible INFO naming the slug. The dangling-path INFO stays ahead
of the basename gate, because a path that does not resolve is rot whatever it is
named. `parse_status` accepts the bullet form and a trailing note after the
backticked value, so a status it cannot read no longer reads as "nothing to check".

Corpus: 36 INFOs of which 30 were false, to 56 of which none are. FAIL stays 0, and
no Status line flipped to `extracted` under the new parser, so no FAIL was
suppressed by luck.

Also closed, each a silent pass: a nonexistent directory, a directory with no
SKILL.md, and extra arguments now exit 2; a UTF-8 BOM no longer defeats frontmatter
parsing; the bare `except Exception: return False` that turned an unreadable file
into a clean pass is gone, with all reads pinned to UTF-8; check 3 walks nested
`references/` subdirectories; `FILL IN:` at end of line no longer escapes checks 1
and 6; duplicate `## slug` blocks and repeated `Research doc:` lines are announced
rather than half-read.

The agent-audit copy carried all of the above unfixed and is now ported, minus the
four fixes that are genuinely N/A at agent scope — it reads a plugin-root
`sources.md` and has no checks 7/8 and no `references/` tree. Its silent exit 0 for
a file outside plugin scope is preserved deliberately: that is a verdict about a
valid file, not a skip, and `check-scope-walkup-sync.sh` pins it. Every exit-2 gate
therefore decides from the argument alone, before the walk-up runs.

`validation-scripts.md` said flatly that silence from the validator is a pass, not
a skip. That sentence is what made a typo'd path dangerous, and both copies are
corrected here. The matching SKILL.md exit-code guidance lands with the audit
rubric change, which touches the same files.

Tests: skill-audit 45 to 65, agent-audit 24 to 43, every new case proven by mutation.

Refs: #111, #118, #121
This commit is contained in:
Defame1297
2026-09-01 12:37:32 +00:00
parent 9fe734573d
commit 88866b81a3
10 changed files with 1959 additions and 204 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
plugins/kyberforge/.apm/skills/agent-audit/references/validation-scripts.md
View File

@@ -34,8 +34,14 @@ first of these:
`plugin.json` and no `apm.yml` falls through to project or user scope.
`validate-provenance.sh` exits 0 silently when that walk does not land on a package root, and again
when the package has no provenance data. Silence from it is a pass, not a skip you need to
investigate.
when the package has no provenance data. Check the exit code before you believe the silence:
- **0** — a pass, not a skip you need to investigate. Both silent cases above land here.
- **1** — real findings, on stdout with Why and Fix.
- **2** — the check never ran. A missing, doubled, non-file or wrongly-named argument, an
undecodable `apm.yml`, or an absent `python3`, each with a diagnostic on stderr and no findings
at all. Report the `### Provenance` dimension as unverified and quote the reason. An exit 2 is
never a clean pass: empty stdout there means nothing was checked, not that nothing was wrong.
## Manual fallback

294
plugins/kyberforge/.apm/skills/agent-audit/scripts/validate-provenance.sh
View File

@@ -16,7 +16,30 @@ Arguments:
Exit codes:
0 All checks passed (or nothing to validate, or not plugin scope)
1 One or more checks failed
2 Script error (unrecognized file extension — expected .md or .agent.md)
2 Usage error, or the argument is not an agent file this script can read
An exit code of 2 is NOT a finding. SKILL.md tells the auditor to surface a
non-zero exit as findings, so a usage error leaving exit 1 with nothing on
stdout was indistinguishable from a clean-but-failing run. Environment and
argument problems exit 2; only real findings exit 1.
Exit 2 and the silent exit 0 answer two DIFFERENT questions, and neither may
be spelled with the other's code:
exit 2 the argument is not something this script can audit at all — it is
missing, doubled, not a file, or not named .md / .agent.md. Decided
before the scope walk-up runs, from the argument alone.
exit 0 the argument IS a readable agent file, and the scope walk-up found
no type:-bearing apm.yml above it before hitting the \$HOME, .git or
filesystem-root boundary. That is a real verdict about a real file —
"this agent is user or project scope, so plugin-scope provenance
does not apply to it" — not a rejected input.
scripts/check-scope-walkup-sync.sh's fixture 6 pins the second: a real agent
file under a \$HOME with a type-bearing apm.yml ABOVE it must exit 0 with empty
output. Widening exit 2 to cover "the walk-up found no package" would break
that fixture AND would be wrong on its own terms, because new-agent.sh happily
scaffolds exactly that layout.
Checks performed:
0 source_keys present in agent pair but sources.md absent
@@ -28,6 +51,13 @@ Checks performed:
not run, never skipped silently.
4 Contributing files back-reference the parent slug in their source_keys
5 Research doc field present and not placeholder
This script has no counterpart to skill-audit's checks 6, 7 and 8 (Research
doc field / upstream forward / upstream reverse are numbered 6, 7, 8 there and
5 here): an agent at plugin scope is a single file with a plugin-root
sources.md, so there is no references/ tree to walk and no upstream research
source index to cross-check. parse_status() and the sources.md-basename gate
that those checks need exist only in the skill-audit copy.
EOF
}
@@ -36,26 +66,131 @@ if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then
exit 0
fi
# Usage and environment problems exit 2, findings exit 1. See the usage text
# above for why the two must not share a code, and for why "not plugin scope"
# is neither of them. This is a deliberate divergence from validate.sh, which
# has no 2 tier for content: validate.sh always prints PASS lines, so a usage
# error there is visibly not a findings report. This script prints NOTHING on a
# clean run, so exit 1 plus empty stdout was the only signal a caller got
# either way.
if [[ $# -lt 1 ]]; then
echo "Error: agent-file is required." >&2
echo "" >&2
usage >&2
exit 1
exit 2
fi
# Extra positional arguments were silently dropped, so a typo'd flag or a second
# path looked like it had been honoured.
if [[ $# -gt 1 ]]; then
echo "Error: expected exactly one argument, got $#: $*" >&2
echo "" >&2
usage >&2
exit 2
fi
# python3 is a HARD dependency. Without this preflight a missing interpreter
# produced 'line NN: python3: command not found' and exit 127 — an exit code no
# caller maps to anything, from a message that names this script's line number
# rather than the missing dependency.
if ! command -v python3 > /dev/null 2>&1; then
echo "Error: python3 is required but was not found on PATH." >&2
echo " Why: skipping the provenance checks entirely would be a vacuous pass." >&2
echo " Fix: install python3 (pre-commit itself is a Python application, so it is almost certainly already present)." >&2
exit 2
fi
# A path that does not exist, or exists but is not a regular file, used to reach
# the Python body, get os.path.dirname()'d into some ancestor directory and then
# either report a silent exit 0 (no package above it) or — worse — audit a
# DIFFERENT agent's package while naming the typo'd path. A typo'd target was
# indistinguishable from a clean agent. vale-wrap.sh hard-errors on a
# nonexistent path for exactly this reason.
#
# This is decided from the argument alone, before any walk-up runs, so it cannot
# collide with the not-plugin-scope exit 0: that verdict is only ever reached by
# a file that got past here.
if [[ ! -e "$1" ]]; then
echo "Error: no such file: $1" >&2
echo " Why: a nonexistent target would otherwise report a silent pass." >&2
echo " Fix: pass the path of the agent file to validate." >&2
exit 2
fi
if [[ ! -f "$1" ]]; then
echo "Error: not a regular file: $1" >&2
echo " Why: this script audits one agent file, not a directory of them, and reporting a directory as a pass hides the wrong-target mistake." >&2
echo " Fix: pass the agent file itself — .apm/agents/<name>.agent.md — not its parent directory." >&2
exit 2
fi
# The extension check used to live inside the Python body. It stays exit 2 and
# keeps its wording; it moves up here so that every "this argument is not
# auditable" verdict is reached in one place, before the interpreter starts and
# before the scope walk-up can turn a bad argument into a silent exit 0.
case "$1" in
*.agent.md | *.md) ;;
*)
echo "Error: unrecognized extension '$(basename "$1")' — expected .md or .agent.md" >&2
exit 2
;;
esac
python3 -u - "$1" <<'PYTHON'
import sys
import os
import re
# Output is UTF-8 for the same reason input is: under LC_ALL=C the streams
# default to ASCII, and every finding this script prints contains an em dash.
# Pinning only the reads moved the crash from the read to the write — a
# UnicodeEncodeError inside print_findings(), which loses the whole report
# after all the checks have already run.
for _stream in (sys.stdout, sys.stderr):
try:
_stream.reconfigure(encoding='utf-8')
except AttributeError: # pragma: no cover — Python < 3.7
pass
agent_file = os.path.abspath(sys.argv[1])
fname = os.path.basename(agent_file)
agent_dir = os.path.dirname(agent_file)
# --- Sanity-check extension (single vendor-neutral .agent.md file at plugin/APM scope) ---
if not (fname.endswith('.agent.md') or fname.endswith('.md')):
print(f"Error: unrecognized extension '{fname}' — expected .md or .agent.md", file=sys.stderr)
sys.exit(2)
# --- Input ----------------------------------------------------------------
# Ported from the skill-audit copy, where the same two problems were already
# fixed.
#
# read_text() pins UTF-8 explicitly instead of inheriting
# locale.getpreferredencoding(), which is ASCII under LC_ALL=C — an ordinary em
# dash in an agent file or in sources.md then aborted the run with a bare
# UnicodeDecodeError traceback, or, at the one call site that wrapped its read
# in `except Exception: return []`, reported the unreadable file as having no
# source_keys and therefore as clean. A file that genuinely is not UTF-8 still
# fails; it just says which file and why.
#
# strip_bom() runs on every read because a leading BOM defeats
# parse_frontmatter()'s `^---` anchor, which silently disabled check 2 on a
# BOM-prefixed agent file: no frontmatter parsed means no source_keys parsed
# means nothing to validate.
class EncodingError(Exception):
pass
def strip_bom(text):
return text[1:] if text.startswith(u'\ufeff') else text
def read_text(path):
"""File contents as text, UTF-8 and BOM-free, with a diagnostic instead of a traceback."""
try:
with open(path, encoding='utf-8') as fh:
return strip_bom(fh.read())
except UnicodeDecodeError as exc:
raise EncodingError(
"not valid UTF-8 (%s at byte %d) — re-save the file as UTF-8; "
"this gate does not guess at other encodings"
% (exc.reason, exc.start))
# Matches a top-level `type:` line whose value is exactly one of the four
# package content types — identical to validate.sh's APM_TYPE_RE. Group 1's
@@ -70,15 +205,31 @@ TYPE_RE = re.compile(r"^type:\s*(['\"]?)(instructions|skill|hybrid|prompts)\1(?:
# keep walking. Stop at a $HOME boundary, a .git boundary, or the filesystem
# root: none of these is plugin/APM scope, so this script has nothing to
# check there.
#
# Returning None here means NOT PLUGIN SCOPE, which is a verdict, not an error:
# the caller exits 0 silently, and scripts/check-scope-walkup-sync.sh fixture 6
# pins that. It is deliberately NOT folded into the exit-2 tier above.
def find_plugin_root(start_dir):
home = os.path.expanduser('~')
current = os.path.abspath(start_dir)
while True:
apm_yml = os.path.join(current, 'apm.yml')
if os.path.isfile(apm_yml):
with open(apm_yml) as f:
if any(TYPE_RE.match(line) for line in f):
return current
# An apm.yml is a manifest this script must be able to READ to
# classify scope at all. Under LC_ALL=C the old bare open() decoded
# as ASCII, so a manifest with an accented author name raised
# UnicodeDecodeError mid-walk and killed the run with a traceback.
# It is an environment problem, not a finding, so it exits 2 rather
# than being swallowed into a silent "no package here".
try:
content = read_text(apm_yml)
except EncodingError as exc:
print(
"Error: %s is %s" % (apm_yml, exc),
file=sys.stderr)
sys.exit(2)
if any(TYPE_RE.match(line) for line in content.splitlines()):
return current
# $HOME is a non-plugin-scope boundary — checked before the .git test
# below (mirrors validate.sh's detect_scope ordering), so a
# dotfiles-managed $HOME (yadm, chezmoi bare-repo, etc.) can't shadow
@@ -104,7 +255,14 @@ if plugin_root is None:
sources_md_path = os.path.join(plugin_root, 'sources.md')
# --- Helpers ---
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:[^`\n]')
# The trailing character class used to be CONSUMING — `[^`\n]` — so a
# `FILL IN:` at end of line matched nothing and escaped checks 1 and 5
# entirely. `- **Description:** FILL IN:` is the most likely spelling of a
# half-written entry, and it was the one spelling the placeholder gate could
# not see. The exclusion it was really expressing is "not inside backticks",
# which a lookahead states without eating a character.
PLACEHOLDER_RE = re.compile(r'(?<!`)FILL IN:(?!`)')
def parse_frontmatter(content):
m = re.match(r'^---\n(.*?)\n---', content, re.DOTALL)
@@ -227,33 +385,48 @@ def parse_contributing_files(content, slug):
return files or None
# ===== END SHARED CONTRIBUTING-FILES PARSER =====
def parse_research_doc(content, slug):
def parse_research_docs(content, slug):
"""Every Research doc value under a given slug H2, in document order.
The caller uses the first and reports the rest. Returning only the first —
what this did before — meant a second '- **Research doc:**' line in one
entry was silently ignored, so an author who added a doc rather than
replacing one got check 5 run against the old value and no hint that the
new one was never looked at.
"""
pattern = re.compile(
r'^## ' + re.escape(slug) + r'\s*\n(.*?)(?=^## |\Z)',
re.MULTILINE | re.DOTALL
)
m = pattern.search(content)
if not m:
return None
return []
block = m.group(1)
rd_m = re.search(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)
if not rd_m:
return None
return rd_m.group(1).strip()
return [v.strip() for v in
re.findall(r'^\- \*\*Research doc:\*\* (.+)$', block, re.MULTILINE)]
findings = []
has_fail = False
# A finding identical in every field is the same finding, and the same file is
# now reached by more than one check — the agent file is read once for its own
# source_keys and again as a contributing file, so an unreadable one would
# otherwise be reported twice with the same words. Distinct findings about the
# same file still both appear.
def _record(entry):
if entry not in findings:
findings.append(entry)
def emit_fail(desc, fpath, why, fix):
global has_fail
has_fail = True
findings.append(("FAIL", desc, fpath, why, fix, None))
_record(("FAIL", desc, fpath, why, fix, None))
# INFO does not set has_fail and does not change the exit code. It is for a
# check that could not RUN — an unverified entry, not a broken one — and it
# exists so that "did not run" is never spelled the same way as "passed".
def emit_info(desc, fpath, note):
findings.append(("INFO", desc, fpath, None, None, note))
_record(("INFO", desc, fpath, None, None, note))
def print_findings():
for entry in findings:
@@ -273,38 +446,56 @@ def print_findings():
print(f" Note: {note}")
print()
def emit_unreadable(rel, exc):
"""Report a file this script cannot decode. Never a silent skip."""
emit_fail(
f"File is {exc}",
rel,
f"'{rel}' cannot be decoded, so its frontmatter — and any source_keys in it — "
f"cannot be read. This used to be swallowed by a bare 'except Exception: return []', "
f"which reported the unreadable file as having no source_keys and therefore as clean.",
f"Re-save '{rel}' as UTF-8."
)
# --- Collect source_keys from agent pair ---
def get_source_keys_from_file(fpath):
def get_source_keys_from_file(fpath, rel):
if not os.path.isfile(fpath):
return []
try:
with open(fpath) as f:
content = f.read()
except Exception:
content = read_text(fpath)
except EncodingError as exc:
emit_unreadable(rel, exc)
return []
fm, _ = parse_frontmatter(content)
return parse_source_keys(fm)
# Plugin/APM scope is a single vendor-neutral file — no counterpart to merge.
given_keys = get_source_keys_from_file(agent_file)
rel_given = os.path.relpath(agent_file, plugin_root)
given_keys = get_source_keys_from_file(agent_file, rel_given)
all_source_keys = given_keys
sources_md_exists = os.path.isfile(sources_md_path)
# Early exit: nothing to validate
# Early exit: nothing to validate. The read above can itself raise a finding —
# an unreadable agent file — so print before leaving; the clean case still
# prints nothing and exits 0.
if not all_source_keys and not sources_md_exists:
sys.exit(0)
print_findings()
sys.exit(1 if has_fail else 0)
sources_content = None
sources_slugs = set()
if sources_md_exists:
with open(sources_md_path) as f:
sources_content = f.read()
try:
sources_content = read_text(sources_md_path)
except EncodingError as exc:
emit_unreadable("sources.md", exc)
print_findings()
sys.exit(1)
sources_slugs = set(parse_h2_slugs(sources_content))
# --- Check 0: source_keys present but sources.md absent ---
if not sources_md_exists and all_source_keys:
rel_given = os.path.relpath(agent_file, plugin_root)
emit_fail(
"source_keys declared but sources.md is absent",
rel_given,
@@ -340,7 +531,31 @@ for fpath, keys in [(agent_file, given_keys)]:
)
# --- Checks 3, 4, 5: Per-slug checks in sources.md ---
for slug in parse_h2_slugs(sources_content):
# Every per-slug parser below — parse_contributing_files, parse_research_docs —
# locates its block with pattern.search(), so a slug written twice resolves to
# the FIRST block every time. Iterating the raw heading list therefore checked
# the first block's fields twice and the second block's never: a duplicated slug
# is half-validated, and looked fully validated. The duplicate is announced and
# the repeat visit dropped.
all_slugs = parse_h2_slugs(sources_content)
unique_slugs = []
for _slug in all_slugs:
if _slug in unique_slugs:
continue
unique_slugs.append(_slug)
_count = all_slugs.count(_slug)
if _count > 1:
emit_info(
f"Duplicate '## {_slug}' entry in sources.md — only the first block is checked",
f"sources.md (## {_slug})",
f"'## {_slug}' appears {_count} times. Every field parser here takes the first match, so the "
f"second and later blocks' Contributing files and Research doc are never validated — "
f"checks 3, 4 and 5 did not run for them. "
f"Merge the blocks into one entry, or give each a distinct slug and reference it from source_keys."
)
for slug in unique_slugs:
# Checks 3 and 4: Contributing files exist (paths relative to plugin root),
# and back-reference the slug. `[]` and None are NOT the same answer here.
# `[]` is the author writing "(none)" — there is nothing to check and the
@@ -370,8 +585,11 @@ for slug in parse_h2_slugs(sources_content):
)
else:
# Check 4: Bidirectional — file should list slug in its source_keys
with open(cf_abs) as f:
cf_content = f.read()
try:
cf_content = read_text(cf_abs)
except EncodingError as exc:
emit_unreadable(cf_rel, exc)
continue
cf_fm, _ = parse_frontmatter(cf_content)
cf_keys = parse_source_keys(cf_fm)
if slug not in cf_keys:
@@ -383,7 +601,17 @@ for slug in parse_h2_slugs(sources_content):
)
# Check 5: Research doc field required
rd_value = parse_research_doc(sources_content, slug)
rd_values = parse_research_docs(sources_content, slug)
if len(rd_values) > 1:
emit_info(
f"Multiple '- **Research doc:**' lines for '{slug}' — only the first is used",
f"sources.md (## {slug})",
f"The '## {slug}' entry has {len(rd_values)} Research doc lines; check 5 ran against the first "
f"('{rd_values[0]}') and never looked at the rest. "
f"Keep one Research doc line per entry — if a slug genuinely came from two documents, split it into two slugs, "
f"or name the extra document inside the first value's annotation where it is at least visible."
)
rd_value = rd_values[0] if rd_values else None
if rd_value is None:
emit_fail(
"Research doc field missing",

337
plugins/kyberforge/.apm/skills/agent-audit/tests/validate-provenance.bats
View File

@@ -570,3 +570,340 @@ EOF
assert_output --partial "INFO Contributing-file checks skipped for 'ghost-source'"
assert_output --partial "Note:"
}
# ---------------------------------------------------------------------------
# The exit-2 tier, and its boundary with the silent exit 0
#
# Ported from the skill-audit sibling, which had already split usage and
# environment errors (exit 2) away from findings (exit 1). SKILL.md tells the
# auditor to surface a non-zero exit, so a usage error leaving exit 1 with
# nothing on stdout was indistinguishable from a clean-but-failing run.
#
# The reconciliation this script needs and the sibling does not: "the walk-up
# found no type:-bearing apm.yml" is NOT bad input. It is a verdict about a
# real, readable agent file — user or project scope, where plugin-scope
# provenance does not apply — and scripts/check-scope-walkup-sync.sh fixture 6
# pins it as exit 0 with empty output. Every exit-2 gate is therefore decided
# from the ARGUMENT ALONE, before the walk-up runs, so the two can never
# collide. The two tests at the end of this block assert both halves.
# ---------------------------------------------------------------------------
@test "exit 2: no arguments is a usage error, not a finding" {
run bash "$SCRIPT"
[ "$status" -eq 2 ]
assert_output --partial "agent-file is required"
}
@test "exit 2: a second positional argument is rejected instead of silently dropped" {
local root="$TMPDIR/package"
make_package "$root"
make_clean_agent "$root"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" --some-typo
[ "$status" -eq 2 ]
assert_output --partial "expected exactly one argument"
}
@test "exit 2: a nonexistent path is an error, not a silent pass" {
run bash "$SCRIPT" "$TMPDIR/no-such-agent.agent.md"
[ "$status" -eq 2 ]
assert_output --partial "no such file"
}
@test "exit 2: a directory is not an agent file" {
local root="$TMPDIR/package"
make_package "$root"
run bash "$SCRIPT" "$root/.apm/agents"
[ "$status" -eq 2 ]
assert_output --partial "not a regular file"
}
@test "exit 2: an unrecognized extension is rejected before the walk-up runs" {
local root="$TMPDIR/package"
make_package "$root"
echo "not an agent" > "$root/.apm/agents/my-agent.txt"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.txt"
[ "$status" -eq 2 ]
assert_output --partial "unrecognized extension"
}
@test "exit 2: a PATH with no python3 names the missing dependency instead of exiting 127" {
local root="$TMPDIR/package"
make_package "$root"
make_clean_agent "$root"
local emptybin="$TMPDIR/emptybin"
mkdir -p "$emptybin"
local bash_bin
bash_bin="$(command -v bash)"
run env -i PATH="$emptybin" HOME="$HOME" "$bash_bin" "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
[ "$status" -eq 2 ]
# The needle is the DIAGNOSTIC, not the bare word: with no preflight, bash's
# own "python3: command not found" would satisfy a bare-word match.
assert_output --partial "python3 is required"
}
@test "reconciliation: a REAL agent file at non-plugin scope still exits 0 silently, never 2" {
# scripts/check-scope-walkup-sync.sh fixture 6 in miniature. The exit-2 tier
# must not widen to cover "find_plugin_root returned None": the file exists,
# is readable and is correctly named — it is simply user/project scope.
local dir="$TMPDIR/anc"
mkdir -p "$dir"
cat > "$dir/apm.yml" <<EOF
name: outer-package
version: 0.1.0
type: skill
EOF
local fake_home="$dir/fakehome"
mkdir -p "$fake_home/.apm/agents"
cat > "$fake_home/.apm/agents/my-agent.agent.md" <<EOF
---
name: my-agent
description: A valid agent description.
source_keys:
- my-source
---
You are a test agent.
EOF
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.apm/agents/my-agent.agent.md"
[ "$status" -eq 0 ]
assert_output ""
}
@test "reconciliation: a nonexistent path inside a non-plugin-scope tree exits 2, not the old silent 0" {
# The other half. Before the exit-2 tier, a typo'd path anywhere outside a
# package took the not-plugin-scope exit and reported a silent pass, so the
# typo and a clean agent produced identical output and identical status.
local fake_home="$TMPDIR/plainhome"
mkdir -p "$fake_home/.apm/agents"
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.apm/agents/typo.agent.md"
[ "$status" -eq 2 ]
assert_output --partial "no such file"
}
# ---------------------------------------------------------------------------
# PLACEHOLDER_RE: the trailing character class was CONSUMING
#
# `(?<!\`)FILL IN:[^\`\n]` required a character after the colon, so a `FILL IN:`
# at end of line matched nothing and escaped checks 1 and 5 entirely — and
# `- **Description:** FILL IN:` is the most likely spelling of a half-written
# entry. The lookahead states the same exclusion without eating a character.
# ---------------------------------------------------------------------------
@test "FAIL: a FILL IN: placeholder at end of line is caught, not skipped" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
cat > "$root/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** FILL IN:
- **Contributing files:** .apm/agents/my-agent.agent.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "Unfilled FILL IN: placeholder"
}
@test "FAIL: a Research doc value that is a bare end-of-line FILL IN: is caught" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
cat > "$root/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** .apm/agents/my-agent.agent.md
- **Research doc:** FILL IN:
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "Research doc field is empty or placeholder"
}
# ---------------------------------------------------------------------------
# Encoding, read side: read_text() pins UTF-8 and strips a BOM
#
# The old code used bare open() calls inheriting locale.getpreferredencoding(),
# which is ASCII under LC_ALL=C, and wrapped exactly one of them in
# `except Exception: return []` — so an unreadable agent file was reported as
# having no source_keys and therefore as CLEAN. The other call sites had no
# handler at all and died with a traceback.
# ---------------------------------------------------------------------------
@test "FAIL: an undecodable agent file is reported, not swallowed into a clean pass" {
local root="$TMPDIR/package"
make_package "$root"
printf '\xff\xfe---\nname: my-agent\n---\n' > "$root/.apm/agents/my-agent.agent.md"
make_sources_md "$root" "my-source" "(none)"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "not valid UTF-8"
refute_output --partial "Traceback"
}
@test "FAIL: an undecodable contributing file is reported, not a traceback" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
printf '\xff\xfe---\nname: other\n---\n' > "$root/.apm/agents/other.agent.md"
make_sources_md "$root" "my-source" ".apm/agents/other.agent.md"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "not valid UTF-8"
refute_output --partial "Traceback"
}
@test "exit 2: an undecodable apm.yml names the file instead of dying mid walk-up" {
local root="$TMPDIR/package"
make_package "$root"
make_clean_agent "$root"
printf 'name: t\nversion: 0.1.0\ntype: skill\n# \xff\xfe\n' > "$root/apm.yml"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
[ "$status" -eq 2 ]
assert_output --partial "not valid UTF-8"
refute_output --partial "Traceback"
}
@test "a BOM-prefixed agent file still has its source_keys read (check 2 runs)" {
# A leading BOM defeats parse_frontmatter()'s ^--- anchor, so no frontmatter
# parsed means no source_keys parsed means nothing to validate — check 2
# went silently missing on exactly the file it was pointed at.
local root="$TMPDIR/package"
make_package "$root"
printf '\xef\xbb\xbf---\nname: my-agent\ndescription: A valid agent description.\nsource_keys:\n - ghost-source\n---\n\nYou are a test agent.\n' \
> "$root/.apm/agents/my-agent.agent.md"
make_sources_md "$root" "my-source" "(none)"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "source_keys slug 'ghost-source' not found in sources.md"
}
@test "under LC_ALL=C a sources.md carrying an em dash is read, not a UnicodeDecodeError" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
cat > "$root/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source — with an em dash.
- **Contributing files:** .apm/agents/ghost.agent.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "Contributing file '.apm/agents/ghost.agent.md' does not exist"
refute_output --partial "Traceback"
}
# ---------------------------------------------------------------------------
# Encoding, write side: sys.stdout/stderr.reconfigure(encoding='utf-8')
#
# Pinning only the reads moved the crash from the read to the WRITE. Every
# finding this script prints contains an em dash, so under LC_ALL=C
# print_findings() died with UnicodeEncodeError after every check had already
# run — losing the whole report at the last step.
# ---------------------------------------------------------------------------
@test "under LC_ALL=C the findings report is printed, not lost to a UnicodeEncodeError" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
make_sources_md "$root" "my-source" ".apm/agents/ghost.agent.md"
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
assert_output --partial "FAIL Contributing file '.apm/agents/ghost.agent.md' does not exist"
assert_output --partial "Why:"
refute_output --partial "UnicodeEncodeError"
}
# ---------------------------------------------------------------------------
# Half-validated entries announced instead of passing silently
# ---------------------------------------------------------------------------
@test "INFO: a duplicated '## slug' says only the first block was checked" {
# Every per-slug parser locates its block with pattern.search(), so a slug
# written twice resolves to the FIRST block every time: the second block's
# fields are never validated, and the entry looked fully checked.
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
cat > "$root/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** (none)
- **Research doc:** (none)
- **Status:** \`extracted\`
## my-source
- **URL:** https://example.com/dup
- **Description:** A duplicate entry.
- **Contributing files:** .apm/agents/ghost.agent.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_success
assert_output --partial "Duplicate '## my-source' entry in sources.md"
# The second block's ghost contributing file is genuinely never checked —
# the INFO is what makes that visible rather than a silent half-pass.
refute_output --partial "does not exist"
}
@test "INFO: a second '- **Research doc:**' line in one entry is announced, not ignored" {
local root="$TMPDIR/package"
make_package "$root"
make_agent_with_source_keys "$root"
cat > "$root/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** (none)
- **Research doc:** (none)
- **Research doc:** docs/research/added-later.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_success
assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'"
}
# ---------------------------------------------------------------------------
# Finding dedup
#
# The agent file is read once for its own source_keys and again as a
# contributing file, so an unreadable one produced the identical finding twice.
# Distinct findings about the same file still both appear.
# ---------------------------------------------------------------------------
@test "the same unreadable file reached by two checks is reported once, not twice" {
local root="$TMPDIR/package"
make_package "$root"
printf '\xff\xfe---\nname: my-agent\n---\n' > "$root/.apm/agents/my-agent.agent.md"
make_sources_md "$root" "my-source" ".apm/agents/my-agent.agent.md"
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
assert_failure
local count
count="$(printf '%s\n' "$output" | grep -c "^FAIL File is not valid UTF-8" || true)"
[ "$count" -eq 1 ]
}