Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity
Files
88866b81a39b0c679a07be5e1096f980dda2ba45
holocron/plugins/kyberforge/.apm/skills/skill-audit/tests/validate-provenance.bats
Defame1297 88866b81a3 fix(kyberforge): announce every provenance skip and scope checks 7-8 to source indexes 
The clean provenance bill was an artifact. Checks 7 and 8 assume `Research doc:`
names a source index whose H2s are slugs, but 30 of 121 corpus entries point at
topic content documents whose H2s are topics. Those 30 produced every new check-7
INFO — all false positives. Check 8 aimed at the same documents, which carry no
`Status:` line at all, would have emitted a large false-FAIL flood; the only thing
preventing it was an unannounced `rd_status != extracted` skip. So "0 new FAILs"
rested on exactly the fail-open class this branch exists to remove, and naively
fixing the skip would have turned the branch red.

Checks 7/8 now run only when the research doc's basename is `sources.md`, and every
other case emits a visible INFO naming the slug. The dangling-path INFO stays ahead
of the basename gate, because a path that does not resolve is rot whatever it is
named. `parse_status` accepts the bullet form and a trailing note after the
backticked value, so a status it cannot read no longer reads as "nothing to check".

Corpus: 36 INFOs of which 30 were false, to 56 of which none are. FAIL stays 0, and
no Status line flipped to `extracted` under the new parser, so no FAIL was
suppressed by luck.

Also closed, each a silent pass: a nonexistent directory, a directory with no
SKILL.md, and extra arguments now exit 2; a UTF-8 BOM no longer defeats frontmatter
parsing; the bare `except Exception: return False` that turned an unreadable file
into a clean pass is gone, with all reads pinned to UTF-8; check 3 walks nested
`references/` subdirectories; `FILL IN:` at end of line no longer escapes checks 1
and 6; duplicate `## slug` blocks and repeated `Research doc:` lines are announced
rather than half-read.

The agent-audit copy carried all of the above unfixed and is now ported, minus the
four fixes that are genuinely N/A at agent scope — it reads a plugin-root
`sources.md` and has no checks 7/8 and no `references/` tree. Its silent exit 0 for
a file outside plugin scope is preserved deliberately: that is a verdict about a
valid file, not a skip, and `check-scope-walkup-sync.sh` pins it. Every exit-2 gate
therefore decides from the argument alone, before the walk-up runs.

`validation-scripts.md` said flatly that silence from the validator is a pass, not
a skip. That sentence is what made a typo'd path dangerous, and both copies are
corrected here. The matching SKILL.md exit-code guidance lands with the audit
rubric change, which touches the same files.

Tests: skill-audit 45 to 65, agent-audit 24 to 43, every new case proven by mutation.

Refs: #111, #118, #121
2026-09-01 12:37:50 +00:00

1497 lines
43 KiB
Bash
Raw Blame History

#!/usr/bin/env bats
setup() {
REPO_ROOT="$(cd "$BATS_TEST_DIRNAME/../../../../../../" && pwd)"
load "$REPO_ROOT/tests/test_helper/bats-support/load"
load "$REPO_ROOT/tests/test_helper/bats-assert/load"
SCRIPT="$(cd "$BATS_TEST_DIRNAME/../scripts" && pwd)/validate-provenance.sh"
TMPDIR="$(mktemp -d)"
# Helper: create a minimal skill directory with no sources.md and no source_keys
make_clean_skill() {
local dir="$1"
local name
name="$(basename "$dir")"
mkdir -p "$dir"
cat > "$dir/SKILL.md" <<EOF
---
name: $name
description: A valid skill description.
---
## Step 1
Do the thing.
EOF
}
# Helper: create a skill with source_keys in SKILL.md
make_skill_with_source_keys() {
local dir="$1"
local name
name="$(basename "$dir")"
mkdir -p "$dir"
cat > "$dir/SKILL.md" <<EOF
---
name: $name
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
}
# Helper: create a valid sources.md with one entry
make_sources_md() {
local dir="$1"
local slug="${2:-my-source}"
local contrib="${3:-SKILL.md}"
local research="${4:-(none)}"
mkdir -p "$dir/references"
cat > "$dir/references/sources.md" <<EOF
# Sources
## ${slug}
- **URL:** https://example.com/${slug}
- **Description:** A test source.
- **Contributing files:** ${contrib}
- **Research doc:** ${research}
- **Status:** \`extracted\`
EOF
}
# Helper: create a fake repo (a .git marker makes find_repo_root stop there)
# holding one skill whose single sources.md slug points at the given
# Research doc value. Checks 7 and 8 only run for a skill inside a checkout,
# so every upstream case needs this shape; the research doc itself is
# written per test into "$repo/docs/research/sources.md".
make_upstream_skill() {
local repo="$1"
local research="${2:-docs/research/sources.md}"
local skill="$repo/my-skill"
mkdir -p "$skill/references" "$repo/docs/research"
touch "$repo/.git"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** ${research}
- **Status:** \`extracted\`
EOF
}
}
teardown() {
rm -rf "$TMPDIR"
}
# ---------------------------------------------------------------------------
# Cycle 1 — --help
# ---------------------------------------------------------------------------
@test "--help exits 0" {
run bash "$SCRIPT" --help
assert_success
assert_output --partial "Usage:"
}
# ---------------------------------------------------------------------------
# Cycle 2 — Early exit: no sources.md, no source_keys → exit 0, no output
# ---------------------------------------------------------------------------
@test "clean pass: no sources.md and no source_keys anywhere → exit 0, no output" {
local skill="$TMPDIR/my-skill"
make_clean_skill "$skill"
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
}
# ---------------------------------------------------------------------------
# Cycle 3 — Check 0: source_keys present but no sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: source_keys in SKILL.md but sources.md absent" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Cycle 4 — Check 1: FILL IN: placeholder in sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: FILL IN: placeholder in sources.md" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** FILL IN: add url
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
@test "FILL IN: inside backticks in sources.md does not fail" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
echo "Use \`FILL IN: value\` as example." >> "$skill/references/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 5 — Check 2: source_keys slug missing from sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: source_keys slug in SKILL.md not present as H2 in sources.md" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## different-source
- **URL:** https://example.com/different-source
- **Description:** A different source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Cycle 6 — Check 4: Contributing file path doesn't exist → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: Contributing file listed in sources.md does not exist" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "references/nonexistent.md"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
@test "pass: (none) in Contributing files is skipped" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "(none — not used directly)"
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 7 — Check 6: Research doc field missing → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: Research doc field missing from sources.md entry" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
@test "FAIL: Research doc field is FILL IN: placeholder" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** FILL IN: path to research doc
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Cycle 8 — Check 5: Bidirectional mismatch → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: Contributing file exists but does not list parent slug in source_keys" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "SKILL.md"
# SKILL.md has source_keys: my-source, but let's change it to NOT have my-source
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- other-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## other-source
- **URL:** https://example.com/other-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
# Now add my-source that references SKILL.md but SKILL.md doesn't back-reference it
cat >> "$skill/references/sources.md" <<EOF
## my-source
- **URL:** https://example.com/my-source
- **Description:** Another source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
# ---------------------------------------------------------------------------
# Cycle 9 — Check 3: references/*.md with no source_keys → INFO (exit 0)
# ---------------------------------------------------------------------------
@test "INFO: references doc with no source_keys frontmatter emits INFO but exits 0" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/extra.md" <<EOF
# Extra Reference
No frontmatter here.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
}
@test "pass: references doc with source_keys all matching sources.md exits 0" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/extra.md" <<EOF
---
source_keys:
- my-source
---
# Extra Reference
Content here.
EOF
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 10 — Clean full pass: valid sources.md, all source_keys match, files exist
# ---------------------------------------------------------------------------
@test "clean full pass: all checks satisfied" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 11 — Check 7: Upstream forward: slug in sources.md not in research doc → INFO
# ---------------------------------------------------------------------------
@test "INFO: slug in sources.md not found in research doc → INFO, exits 0" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
# Create a research doc that does NOT have the slug
local research_dir="$TMPDIR/research"
mkdir -p "$research_dir"
cat > "$research_dir/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
# Use a path relative to repo root — we'll place research doc inside TMPDIR
# and reference it as absolute for test purposes.
# The script finds repo root by walking up from skill-dir until .git is found.
# Since TMPDIR won't have .git, we simulate a repo structure.
local fake_repo="$TMPDIR/fakerepo"
mkdir -p "$fake_repo"
touch "$fake_repo/.git" # fake .git marker
local skill2="$fake_repo/my-skill"
mkdir -p "$skill2"
cat > "$skill2/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill2/references"
mkdir -p "$fake_repo/docs/research"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
cat > "$skill2/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill2"
assert_success
assert_output --partial "INFO"
}
# ---------------------------------------------------------------------------
# Cycle 12 — Check 8: Upstream reverse: extracted slug in research doc not in sources.md → FAIL
# ---------------------------------------------------------------------------
@test "FAIL: extracted non-(none) slug in research doc missing from skill sources.md" {
local fake_repo="$TMPDIR/fakerepo"
mkdir -p "$fake_repo"
touch "$fake_repo/.git"
local skill="$fake_repo/my-skill"
mkdir -p "$skill"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill/references"
mkdir -p "$fake_repo/docs/research"
# Research doc has my-source (extracted, with a contributing file) AND extra-source (also extracted)
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
}
@test "pass: extracted slug in research doc with (none) contributing files is not required in sources.md" {
local fake_repo="$TMPDIR/fakerepo"
mkdir -p "$fake_repo"
touch "$fake_repo/.git"
local skill="$fake_repo/my-skill"
mkdir -p "$skill"
cat > "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
mkdir -p "$skill/references"
mkdir -p "$fake_repo/docs/research"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (none — not relevant)
- **Status:** \`extracted\`
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 13 — parse_contributing_files: None ("could not parse") is NOT []
# ("explicitly (none)")
#
# Check 8 reads [] as "the research doc deliberately records no contributing
# files" and SKIPS the slug on that basis. A block the parser cannot read must
# therefore return None, so the slug stays exposed to check 8. Each case below
# asserts that CONSEQUENCE — check 8 firing on the unreadable entry — not the
# parser's return value, because returning [] is exactly the shape that makes
# the check silently do nothing while still exiting 0.
# ---------------------------------------------------------------------------
@test "check 8 runs: bullet form with '*' asterisk bullets is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
* some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form with a numbered list is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
1. some-skill/references/extra.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form followed by prose is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
See the table below for the file list.
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: bullet form heading with a blank line and nothing after is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Status:** \`extracted\`
**Contributing files:**
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 8 runs: inline form whose whole value is a parenthetical is unparsable, not '(none)'" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (see notes below)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "extra-source"
}
@test "check 4 runs: bullet form with '-' hyphen bullets still parses each path" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
**Contributing files:**
- SKILL.md (the dispatch table, with a comma in the note)
- references/nonexistent.md (why this one matters)
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "Contributing file 'references/nonexistent.md' does not exist"
}
@test "check 4 runs: inline comma-separated form still parses each path" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "SKILL.md, references/nonexistent.md"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "Contributing file 'references/nonexistent.md' does not exist"
}
@test "check 8 skips: bullet form '- (none)' is an explicit declaration" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
**Contributing files:**
- (none — nothing was extracted from this section)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
}
@test "check 8 skips: inline bare '(none)' is an explicit declaration" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
}
# ---------------------------------------------------------------------------
# Cycle 14 — Check 3 (#111): an explicit 'source_keys: []' is a house-authored
# declaration, a bare 'source_keys:' is truncation
#
# Three states, three outcomes. Only the middle one — declared empty — is
# silent; collapsing any pair of them is the defect #111 filed.
# ---------------------------------------------------------------------------
@test "check 3 silent: references doc with top-level 'source_keys: []' emits nothing" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
source_keys: []
---
# Extra Reference
House-authored, no external source.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
}
@test "check 3 silent: references doc with 'source_keys: []' nested under metadata: emits nothing" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
metadata:
source_keys: []
---
# Extra Reference
House-authored, no external source.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
}
@test "INFO: references doc with a bare 'source_keys:' and no value still emits INFO" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
source_keys:
---
# Extra Reference
Truncated frontmatter — this is not a decision.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "No source_keys frontmatter"
}
@test "INFO: references doc with frontmatter but no source_keys key still emits INFO" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
title: Extra Reference
---
# Extra Reference
Never said either way.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "No source_keys frontmatter"
}
# ---------------------------------------------------------------------------
# Cycle 15 — Checks 7 and 8: Research doc annotation stripping, and the INFO
# that replaced the silent skip
#
# A Research doc value is very often a path PLUS a section annotation, and
# os.path.isfile() is false for every such string. Before the strip, checks 7
# and 8 skipped SILENTLY on those — so the negative assertions here (an INFO
# that must appear) are what distinguishes a running check from a dead one.
# ---------------------------------------------------------------------------
@test "check 7 runs: '§' section annotation is stripped before the path is resolved" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" 'docs/research/sources.md § "Some Section"'
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "§"
}
@test "check 7 runs: '→' section annotation is stripped before the path is resolved" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" 'docs/research/sources.md → `## Pushing`'
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "→"
}
@test "check 7 runs: parenthetical annotation is stripped before the path is resolved" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "docs/research/sources.md (whole-document reference)"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "whole-document reference"
}
@test "check 7 runs: a bare path with no annotation survives the strip intact" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "docs/research/sources.md"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Research
## my-source
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output ""
}
@test "checks 7 and 8 skipped silently: Research doc '(none)' is recognised before the strip" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "(none)"
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output ""
}
@test "checks 7 and 8 skipped silently: bare 'none — reason' is recognised as a declaration" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "none — org convention, no upstream research doc"
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output ""
}
@test "INFO: a stripped path that does not resolve names the slug instead of skipping silently" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" 'docs/research/missing.md § "Some Section"'
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "Upstream checks skipped for 'my-source' — research doc 'docs/research/missing.md' does not exist"
}
@test "INFO: a Research doc value that is entirely annotation names the slug instead of skipping silently" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" '§ "Some Section"'
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "Upstream checks skipped for 'my-source' — Research doc value names no path"
}
@test "INFO: no repo root above the skill directory names the slug instead of skipping silently" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "SKILL.md" "docs/research/sources.md"
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "Upstream checks skipped for 'my-source' — no repo root above the skill directory"
}
# ---------------------------------------------------------------------------
# Cycle 16 — Checks 4 and 5: None ("could not parse") is NOT [] ("explicitly
# (none)"), on the sources.md side this time
#
# Cycle 13 pinned the distinction for check 8, which reads the parser's output
# against a RESEARCH doc. Checks 4 and 5 read it against the skill's own
# sources.md and honoured neither half: a truthiness test collapsed None into
# [], so an unreadable Contributing files block disabled both checks and the
# script still exited 0 with no output — the failure mode
# parse_contributing_files' docstring names in as many words. The assertions
# below are therefore about the INFO appearing; a silent exit 0 is exactly the
# bug.
# ---------------------------------------------------------------------------
@test "INFO: an unparsable Contributing files block names the slug instead of skipping checks 4 and 5 silently" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Research doc:** (none)
**Contributing files:**
* references/ghost.md (asterisk bullets are not the bullet form)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "Contributing-file checks skipped for 'my-source' — the Contributing files block could not be parsed"
}
@test "INFO: an entry with no Contributing files field at all is reported, not skipped silently" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "Contributing-file checks skipped for 'my-source' — the Contributing files block could not be parsed"
}
@test "checks 4 and 5 skipped silently: an explicit '(none)' emits no INFO" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "(none — not used directly)"
run bash "$SCRIPT" "$skill"
assert_success
assert_output ""
}
@test "checks 4 and 5 still run: a parseable Contributing files list is not diverted to the INFO" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "references/nonexistent.md"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "FAIL"
assert_output --partial "Contributing file 'references/nonexistent.md' does not exist"
refute_output --partial "could not be parsed"
}
# ---------------------------------------------------------------------------
# Cycle 17 — Check 3 (#111): 'source_keys: []' only declares anything in the
# two positions parse_source_keys() actually reads
#
# The declaration and the parse have to agree on WHERE the key lives. They did
# not: the declaration regex accepted any indent, so a `source_keys: []` buried
# under an unrelated key — a position parse_source_keys() never reads — passed
# as a house-authored declaration and silenced the INFO for a file that had
# declared nothing.
# ---------------------------------------------------------------------------
@test "INFO: 'source_keys: []' nested under an unrelated key is not a declaration" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
title: Extra Reference
unrelated:
nested:
source_keys: []
---
# Extra Reference
The empty list is nested where parse_source_keys never looks.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "No source_keys frontmatter"
}
@test "INFO: a four-space-indented 'source_keys: []' is not a declaration either" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/extra.md" <<EOF
---
metadata:
source_keys: []
---
# Extra Reference
Two spaces is the position parse_source_keys reads under metadata:, not four.
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "INFO"
assert_output --partial "No source_keys frontmatter"
}
# ---------------------------------------------------------------------------
# Cycle 20 — F: checks 7 and 8 apply only to a research SOURCE INDEX, and
# every skip announces itself
# ---------------------------------------------------------------------------
@test "F: a topic-doc Research doc is reported as not applicable, not as a missing slug" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo" "docs/research/remotes.md"
cat > "$fake_repo/docs/research/remotes.md" <<EOF
# Remotes
## Core Philosophy
Prose about remotes.
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Upstream checks not applicable for 'my-source'"
assert_output --partial "is a topic document, not a source index"
refute_output --partial "not found as H2 in research doc"
}
@test "F: check 7 still runs when the Research doc IS a source index" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## different-slug
- **Contributing files:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Slug 'my-source' not found as H2 in research doc 'docs/research/sources.md'"
refute_output --partial "not applicable"
}
@test "F: check 8 announces the slug it skipped for a non-extracted Status" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`referenced\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Check 8 skipped for research-doc slug 'extra-source'"
assert_output --partial "its Status is \`referenced\`, not \`extracted\`"
}
@test "F: a Status with a trailing note after the backticked value still reads as extracted" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
- **Status:** \`extracted\` — partial fetch, section 3 only
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "Research doc slug 'extra-source' missing from skill sources.md"
}
@test "F: a bullet-form Status still reads as extracted" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
**Status:**
- \`extracted\`
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_failure
assert_output --partial "Research doc slug 'extra-source' missing from skill sources.md"
}
@test "F: a research-doc slug with no Status line at all is announced, not skipped silently" {
local fake_repo="$TMPDIR/fakerepo"
make_upstream_skill "$fake_repo"
cat > "$fake_repo/docs/research/sources.md" <<EOF
# Sources
## my-source
- **Contributing files:** some-skill/SKILL.md
- **Status:** \`extracted\`
## extra-source
- **Contributing files:** some-skill/references/extra.md
EOF
run bash "$SCRIPT" "$fake_repo/my-skill"
assert_success
assert_output --partial "Check 8 skipped for research-doc slug 'extra-source'"
assert_output --partial "its Status is absent, not \`extracted\`"
}
# ---------------------------------------------------------------------------
# Cycle 21 — G1: a bad target is a hard error, not a silent pass
# ---------------------------------------------------------------------------
@test "G1: a nonexistent directory is a hard error (exit 2), not a silent exit 0" {
run bash "$SCRIPT" "$TMPDIR/does-not-exist"
[ "$status" -eq 2 ]
assert_output --partial "not a directory"
}
@test "G1: a directory with no SKILL.md is a hard error (exit 2), not a silent exit 0" {
mkdir -p "$TMPDIR/not-a-skill/references"
run bash "$SCRIPT" "$TMPDIR/not-a-skill"
[ "$status" -eq 2 ]
assert_output --partial "not a skill directory"
}
# ---------------------------------------------------------------------------
# Cycle 22 — G2: a UTF-8 BOM does not disable check 2
# ---------------------------------------------------------------------------
@test "G2: a BOM at the head of SKILL.md does not silently disable check 2" {
local skill="$TMPDIR/my-skill"
mkdir -p "$skill/references"
printf '\xef\xbb\xbf' > "$skill/SKILL.md"
cat >> "$skill/SKILL.md" <<EOF
---
name: my-skill
description: A valid skill description.
metadata:
source_keys:
- my-source
---
## Step 1
Do the thing.
EOF
cat > "$skill/references/sources.md" <<EOF
# Sources
## different-source
- **URL:** https://example.com/different-source
- **Description:** A test source.
- **Contributing files:** references/sources.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "source_keys slug 'my-source' not found in sources.md"
}
# ---------------------------------------------------------------------------
# Cycle 23 — G3: reads are UTF-8 and an unreadable file is a finding
# ---------------------------------------------------------------------------
@test "G3: an em dash under LC_ALL=C is read, not turned into a traceback or a silent pass" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
cat > "$skill/references/topic.md" <<EOF
---
source_keys:
- ghost-source
---
Prose — with an em dash.
EOF
LC_ALL=C PYTHONUTF8=0 PYTHONCOERCECLOCALE=0 run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "source_keys slug 'ghost-source' not found in sources.md"
refute_output --partial "Traceback"
}
@test "G3: a file that is genuinely not UTF-8 is a FAIL, not a clean pass" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
printf -- '---\nsource_keys:\n - my-source\n---\n\nLatin-1 byte: \xe9\n' \
> "$skill/references/topic.md"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "not valid UTF-8"
assert_output --partial "references/topic.md"
}
# ---------------------------------------------------------------------------
# Cycle 24 — G4: check 3 walks references/ recursively
# ---------------------------------------------------------------------------
@test "G4: a source_keys file in references/<subdir>/ is validated, not skipped" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill"
mkdir -p "$skill/references/nested"
cat > "$skill/references/nested/topic.md" <<EOF
---
source_keys:
- ghost-source
---
Nested prose.
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "source_keys slug 'ghost-source' not found in sources.md"
assert_output --partial "references/nested/topic.md"
}
# ---------------------------------------------------------------------------
# Cycle 25 — G5: a placeholder at end of line is still a placeholder
# ---------------------------------------------------------------------------
@test "G5: 'FILL IN:' at end of line is caught by check 1" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** FILL IN:
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "Unfilled FILL IN: placeholder"
}
@test "G5: a Research doc value that is a bare 'FILL IN:' is caught by check 6" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
make_sources_md "$skill" "my-source" "SKILL.md" "FILL IN:"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "Research doc field is empty or placeholder"
}
# ---------------------------------------------------------------------------
# Cycle 26 — G6/G7: duplicated entries and duplicated fields are announced
# ---------------------------------------------------------------------------
@test "G6: a duplicate '## slug' block is announced, not half-checked in silence" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Status:** \`extracted\`
## my-source
- **URL:** https://example.com/my-source-again
- **Description:** The same slug a second time.
- **Contributing files:** references/nonexistent.md
- **Research doc:** (none)
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "Duplicate '## my-source' entry in sources.md"
assert_output --partial "only the first block is checked"
}
@test "G7: a second '- **Research doc:**' line in one entry is announced" {
local skill="$TMPDIR/my-skill"
make_skill_with_source_keys "$skill"
mkdir -p "$skill/references"
cat > "$skill/references/sources.md" <<EOF
# Sources
## my-source
- **URL:** https://example.com/my-source
- **Description:** A test source.
- **Contributing files:** SKILL.md
- **Research doc:** (none)
- **Research doc:** docs/research/sources.md
- **Status:** \`extracted\`
EOF
run bash "$SCRIPT" "$skill"
assert_success
assert_output --partial "Multiple '- **Research doc:**' lines for 'my-source'"
}
# ---------------------------------------------------------------------------
# Cycle 27 — G8: usage and environment errors exit 2, never 1
# ---------------------------------------------------------------------------
@test "G8: a missing argument exits 2, not 1" {
run bash "$SCRIPT"
[ "$status" -eq 2 ]
assert_output --partial "skill-dir is required"
}
@test "G8: an extra positional argument is rejected, not silently ignored" {
local skill="$TMPDIR/my-skill"
make_clean_skill "$skill"
run bash "$SCRIPT" "$skill" extra
[ "$status" -eq 2 ]
assert_output --partial "expected exactly one argument"
}
@test "G8: a missing python3 is reported by name and exits 2, not 127" {
local skill="$TMPDIR/my-skill"
make_clean_skill "$skill"
local stub="$TMPDIR/emptybin"
mkdir -p "$stub"
for cmd in bash cat sed; do
ln -s "$(command -v "$cmd")" "$stub/$cmd"
done
PATH="$stub" run bash "$SCRIPT" "$skill"
[ "$status" -eq 2 ]
assert_output --partial "python3 is required"
}
@test "G3: an unreadable file is reported even when there is nothing else to validate" {
local skill="$TMPDIR/my-skill"
make_clean_skill "$skill"
mkdir -p "$skill/references"
printf -- '---\nsource_keys:\n - my-source\n---\n\nLatin-1 byte: \xe9\n' \
> "$skill/references/topic.md"
run bash "$SCRIPT" "$skill"
assert_failure
assert_output --partial "not valid UTF-8"
}
Reference in New Issue View Git Blame Copy Permalink