fix(kyberforge): announce every provenance skip and scope checks 7-8 to source indexes
The clean provenance bill was an artifact. Checks 7 and 8 assume `Research doc:` names a source index whose H2s are slugs, but 30 of 121 corpus entries point at topic content documents whose H2s are topics. Those 30 produced every new check-7 INFO — all false positives. Check 8 aimed at the same documents, which carry no `Status:` line at all, would have emitted a large false-FAIL flood; the only thing preventing it was an unannounced `rd_status != extracted` skip. So "0 new FAILs" rested on exactly the fail-open class this branch exists to remove, and naively fixing the skip would have turned the branch red. Checks 7/8 now run only when the research doc's basename is `sources.md`, and every other case emits a visible INFO naming the slug. The dangling-path INFO stays ahead of the basename gate, because a path that does not resolve is rot whatever it is named. `parse_status` accepts the bullet form and a trailing note after the backticked value, so a status it cannot read no longer reads as "nothing to check". Corpus: 36 INFOs of which 30 were false, to 56 of which none are. FAIL stays 0, and no Status line flipped to `extracted` under the new parser, so no FAIL was suppressed by luck. Also closed, each a silent pass: a nonexistent directory, a directory with no SKILL.md, and extra arguments now exit 2; a UTF-8 BOM no longer defeats frontmatter parsing; the bare `except Exception: return False` that turned an unreadable file into a clean pass is gone, with all reads pinned to UTF-8; check 3 walks nested `references/` subdirectories; `FILL IN:` at end of line no longer escapes checks 1 and 6; duplicate `## slug` blocks and repeated `Research doc:` lines are announced rather than half-read. The agent-audit copy carried all of the above unfixed and is now ported, minus the four fixes that are genuinely N/A at agent scope — it reads a plugin-root `sources.md` and has no checks 7/8 and no `references/` tree. Its silent exit 0 for a file outside plugin scope is preserved deliberately: that is a verdict about a valid file, not a skip, and `check-scope-walkup-sync.sh` pins it. Every exit-2 gate therefore decides from the argument alone, before the walk-up runs. `validation-scripts.md` said flatly that silence from the validator is a pass, not a skip. That sentence is what made a typo'd path dangerous, and both copies are corrected here. The matching SKILL.md exit-code guidance lands with the audit rubric change, which touches the same files. Tests: skill-audit 45 to 65, agent-audit 24 to 43, every new case proven by mutation. Refs: #111, #118, #121
This commit is contained in:
@@ -570,3 +570,340 @@ EOF
|
||||
assert_output --partial "INFO Contributing-file checks skipped for 'ghost-source'"
|
||||
assert_output --partial "Note:"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The exit-2 tier, and its boundary with the silent exit 0
|
||||
#
|
||||
# Ported from the skill-audit sibling, which had already split usage and
|
||||
# environment errors (exit 2) away from findings (exit 1). SKILL.md tells the
|
||||
# auditor to surface a non-zero exit, so a usage error leaving exit 1 with
|
||||
# nothing on stdout was indistinguishable from a clean-but-failing run.
|
||||
#
|
||||
# The reconciliation this script needs and the sibling does not: "the walk-up
|
||||
# found no type:-bearing apm.yml" is NOT bad input. It is a verdict about a
|
||||
# real, readable agent file — user or project scope, where plugin-scope
|
||||
# provenance does not apply — and scripts/check-scope-walkup-sync.sh fixture 6
|
||||
# pins it as exit 0 with empty output. Every exit-2 gate is therefore decided
|
||||
# from the ARGUMENT ALONE, before the walk-up runs, so the two can never
|
||||
# collide. The two tests at the end of this block assert both halves.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "exit 2: no arguments is a usage error, not a finding" {
|
||||
run bash "$SCRIPT"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "agent-file is required"
|
||||
}
|
||||
|
||||
@test "exit 2: a second positional argument is rejected instead of silently dropped" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_clean_agent "$root"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md" --some-typo
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "expected exactly one argument"
|
||||
}
|
||||
|
||||
@test "exit 2: a nonexistent path is an error, not a silent pass" {
|
||||
run bash "$SCRIPT" "$TMPDIR/no-such-agent.agent.md"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "no such file"
|
||||
}
|
||||
|
||||
@test "exit 2: a directory is not an agent file" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
run bash "$SCRIPT" "$root/.apm/agents"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "not a regular file"
|
||||
}
|
||||
|
||||
@test "exit 2: an unrecognized extension is rejected before the walk-up runs" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
echo "not an agent" > "$root/.apm/agents/my-agent.txt"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.txt"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "unrecognized extension"
|
||||
}
|
||||
|
||||
@test "exit 2: a PATH with no python3 names the missing dependency instead of exiting 127" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_clean_agent "$root"
|
||||
local emptybin="$TMPDIR/emptybin"
|
||||
mkdir -p "$emptybin"
|
||||
local bash_bin
|
||||
bash_bin="$(command -v bash)"
|
||||
run env -i PATH="$emptybin" HOME="$HOME" "$bash_bin" "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
[ "$status" -eq 2 ]
|
||||
# The needle is the DIAGNOSTIC, not the bare word: with no preflight, bash's
|
||||
# own "python3: command not found" would satisfy a bare-word match.
|
||||
assert_output --partial "python3 is required"
|
||||
}
|
||||
|
||||
@test "reconciliation: a REAL agent file at non-plugin scope still exits 0 silently, never 2" {
|
||||
# scripts/check-scope-walkup-sync.sh fixture 6 in miniature. The exit-2 tier
|
||||
# must not widen to cover "find_plugin_root returned None": the file exists,
|
||||
# is readable and is correctly named — it is simply user/project scope.
|
||||
local dir="$TMPDIR/anc"
|
||||
mkdir -p "$dir"
|
||||
cat > "$dir/apm.yml" <<EOF
|
||||
name: outer-package
|
||||
version: 0.1.0
|
||||
type: skill
|
||||
EOF
|
||||
local fake_home="$dir/fakehome"
|
||||
mkdir -p "$fake_home/.apm/agents"
|
||||
cat > "$fake_home/.apm/agents/my-agent.agent.md" <<EOF
|
||||
---
|
||||
name: my-agent
|
||||
description: A valid agent description.
|
||||
source_keys:
|
||||
- my-source
|
||||
---
|
||||
|
||||
You are a test agent.
|
||||
EOF
|
||||
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.apm/agents/my-agent.agent.md"
|
||||
[ "$status" -eq 0 ]
|
||||
assert_output ""
|
||||
}
|
||||
|
||||
@test "reconciliation: a nonexistent path inside a non-plugin-scope tree exits 2, not the old silent 0" {
|
||||
# The other half. Before the exit-2 tier, a typo'd path anywhere outside a
|
||||
# package took the not-plugin-scope exit and reported a silent pass, so the
|
||||
# typo and a clean agent produced identical output and identical status.
|
||||
local fake_home="$TMPDIR/plainhome"
|
||||
mkdir -p "$fake_home/.apm/agents"
|
||||
run env HOME="$fake_home" bash "$SCRIPT" "$fake_home/.apm/agents/typo.agent.md"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "no such file"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# PLACEHOLDER_RE: the trailing character class was CONSUMING
|
||||
#
|
||||
# `(?<!\`)FILL IN:[^\`\n]` required a character after the colon, so a `FILL IN:`
|
||||
# at end of line matched nothing and escaped checks 1 and 5 entirely — and
|
||||
# `- **Description:** FILL IN:` is the most likely spelling of a half-written
|
||||
# entry. The lookahead states the same exclusion without eating a character.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: a FILL IN: placeholder at end of line is caught, not skipped" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** FILL IN:
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "Unfilled FILL IN: placeholder"
|
||||
}
|
||||
|
||||
@test "FAIL: a Research doc value that is a bare end-of-line FILL IN: is caught" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** .apm/agents/my-agent.agent.md
|
||||
- **Research doc:** FILL IN:
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "Research doc field is empty or placeholder"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Encoding, read side: read_text() pins UTF-8 and strips a BOM
|
||||
#
|
||||
# The old code used bare open() calls inheriting locale.getpreferredencoding(),
|
||||
# which is ASCII under LC_ALL=C, and wrapped exactly one of them in
|
||||
# `except Exception: return []` — so an unreadable agent file was reported as
|
||||
# having no source_keys and therefore as CLEAN. The other call sites had no
|
||||
# handler at all and died with a traceback.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "FAIL: an undecodable agent file is reported, not swallowed into a clean pass" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
printf '\xff\xfe---\nname: my-agent\n---\n' > "$root/.apm/agents/my-agent.agent.md"
|
||||
make_sources_md "$root" "my-source" "(none)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "not valid UTF-8"
|
||||
refute_output --partial "Traceback"
|
||||
}
|
||||
|
||||
@test "FAIL: an undecodable contributing file is reported, not a traceback" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
printf '\xff\xfe---\nname: other\n---\n' > "$root/.apm/agents/other.agent.md"
|
||||
make_sources_md "$root" "my-source" ".apm/agents/other.agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "not valid UTF-8"
|
||||
refute_output --partial "Traceback"
|
||||
}
|
||||
|
||||
@test "exit 2: an undecodable apm.yml names the file instead of dying mid walk-up" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_clean_agent "$root"
|
||||
printf 'name: t\nversion: 0.1.0\ntype: skill\n# \xff\xfe\n' > "$root/apm.yml"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
[ "$status" -eq 2 ]
|
||||
assert_output --partial "not valid UTF-8"
|
||||
refute_output --partial "Traceback"
|
||||
}
|
||||
|
||||
@test "a BOM-prefixed agent file still has its source_keys read (check 2 runs)" {
|
||||
# A leading BOM defeats parse_frontmatter()'s ^--- anchor, so no frontmatter
|
||||
# parsed means no source_keys parsed means nothing to validate — check 2
|
||||
# went silently missing on exactly the file it was pointed at.
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
printf '\xef\xbb\xbf---\nname: my-agent\ndescription: A valid agent description.\nsource_keys:\n - ghost-source\n---\n\nYou are a test agent.\n' \
|
||||
> "$root/.apm/agents/my-agent.agent.md"
|
||||
make_sources_md "$root" "my-source" "(none)"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "source_keys slug 'ghost-source' not found in sources.md"
|
||||
}
|
||||
|
||||
@test "under LC_ALL=C a sources.md carrying an em dash is read, not a UnicodeDecodeError" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source — with an em dash.
|
||||
- **Contributing files:** .apm/agents/ghost.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "Contributing file '.apm/agents/ghost.agent.md' does not exist"
|
||||
refute_output --partial "Traceback"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Encoding, write side: sys.stdout/stderr.reconfigure(encoding='utf-8')
|
||||
#
|
||||
# Pinning only the reads moved the crash from the read to the WRITE. Every
|
||||
# finding this script prints contains an em dash, so under LC_ALL=C
|
||||
# print_findings() died with UnicodeEncodeError after every check had already
|
||||
# run — losing the whole report at the last step.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "under LC_ALL=C the findings report is printed, not lost to a UnicodeEncodeError" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
make_sources_md "$root" "my-source" ".apm/agents/ghost.agent.md"
|
||||
run env LC_ALL=C PYTHONUTF8=0 bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
assert_output --partial "FAIL Contributing file '.apm/agents/ghost.agent.md' does not exist"
|
||||
assert_output --partial "Why:"
|
||||
refute_output --partial "UnicodeEncodeError"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Half-validated entries announced instead of passing silently
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "INFO: a duplicated '## slug' says only the first block was checked" {
|
||||
# Every per-slug parser locates its block with pattern.search(), so a slug
|
||||
# written twice resolves to the FIRST block every time: the second block's
|
||||
# fields are never validated, and the entry looked fully checked.
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** (none)
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/dup
|
||||
- **Description:** A duplicate entry.
|
||||
- **Contributing files:** .apm/agents/ghost.agent.md
|
||||
- **Research doc:** (none)
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output --partial "Duplicate '## my-source' entry in sources.md"
|
||||
# The second block's ghost contributing file is genuinely never checked —
|
||||
# the INFO is what makes that visible rather than a silent half-pass.
|
||||
refute_output --partial "does not exist"
|
||||
}
|
||||
|
||||
@test "INFO: a second '- **Research doc:**' line in one entry is announced, not ignored" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
make_agent_with_source_keys "$root"
|
||||
cat > "$root/sources.md" <<EOF
|
||||
# Sources
|
||||
|
||||
## my-source
|
||||
|
||||
- **URL:** https://example.com/my-source
|
||||
- **Description:** A test source.
|
||||
- **Contributing files:** (none)
|
||||
- **Research doc:** (none)
|
||||
- **Research doc:** docs/research/added-later.md
|
||||
- **Status:** \`extracted\`
|
||||
EOF
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_success
|
||||
assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Finding dedup
|
||||
#
|
||||
# The agent file is read once for its own source_keys and again as a
|
||||
# contributing file, so an unreadable one produced the identical finding twice.
|
||||
# Distinct findings about the same file still both appear.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@test "the same unreadable file reached by two checks is reported once, not twice" {
|
||||
local root="$TMPDIR/package"
|
||||
make_package "$root"
|
||||
printf '\xff\xfe---\nname: my-agent\n---\n' > "$root/.apm/agents/my-agent.agent.md"
|
||||
make_sources_md "$root" "my-source" ".apm/agents/my-agent.agent.md"
|
||||
run bash "$SCRIPT" "$root/.apm/agents/my-agent.agent.md"
|
||||
assert_failure
|
||||
local count
|
||||
count="$(printf '%s\n' "$output" | grep -c "^FAIL File is not valid UTF-8" || true)"
|
||||
[ "$count" -eq 1 ]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user