audit: full test-run — 3 fixes applied, 4 untracked items, kyberforge improvements needed #2

New Issue

Claude · 2026-06-21T01:07:01Z

Claude commented

2026-06-21 01:07:01 +00:00

Overview

A complete test audit was run on this session (2026-06-21). All automatable checks were executed via parallel subagents. Three issues were fixed inline. Four new issues were found that are not yet tracked. The kyberforge plugin has several follow-on improvement opportunities. This issue is the handoff document for a follow-up session.

Process followed

1. Discovery

Surveyed the repo to identify all runnable checks before launching agents:

tests/*.sh — shell test scripts
scripts/check-manifests.sh — manifest validator
.agents/evals/ and plugins/kyberforge/*/evals/eval.yaml — eval files
plugins/kyberforge/skills/marketplace-architect/scripts/ — plugin utility scripts
claude plugin validate — Claude Code plugin manifest validator (discovered as a gap mid-session)

2. Parallel execution

Launched 4 subagents in parallel (first wave):

shell-tests — ran all 5 automatable test scripts
manifest-checks — JSON validity, check-manifests.sh, bash -n syntax, shellcheck
skill-conformance — audited every SKILL.md against the 6-section / META.md / 3-field-frontmatter standard
secret-scan — gitleaks full scan + manual grep for credential patterns

Then a 5th agent (second wave) after user flagged the gap:

plugin-scripts — ran marketplace-architect/scripts/test_scripts.sh, inventory.sh, validate.sh, hitl-loop template syntax check

Then a 6th agent after confirming claude plugin validate had not been run:

claude-plugin-validate — ran claude plugin validate (normal + --strict) on kyberforge plugin and both marketplace manifests

3. Fix and verify

Applied 3 fixes inline
Re-ran claude plugin validate --strict after fixes to confirm both targets now pass

Results — what passed

Check	Result
Shell tests — 77 cases across 5 scripts	✅ All pass
`marketplace-architect/scripts/test_scripts.sh` — 18 cases	✅ All pass
`scripts/check-manifests.sh`	✅ Pass
JSON validity — all manifest files	✅ Pass
`bash -n` syntax — all scripts and test files	✅ Pass
Eval structure — 7 `eval.yaml` files	✅ Pass (all 5 required test types present)
`gitleaks detect --no-git` (working tree)	✅ Clean
`claude plugin validate plugins/kyberforge`	✅ Pass (after fixes)
`claude plugin validate .claude-plugin/marketplace.json`	✅ Pass (after fixes)
`claude plugin validate .github/plugin/marketplace.json`	✅ Pass (unchanged)

Tests skipped by design (require live Claude session or modify the system):

tests/test-install.sh — modifies ~/.claude/; needs a clean isolated environment
tests/test-governance-layer.sh — requires a live interactive Claude session
tests/test-instructions-and-docs.sh — requires a live interactive Claude session

Fixes applied this session

Fix 1 — `install.sh` SC2115: `rm -rf` without `:?` guard

File: scripts/install.sh line 42
Severity: Real risk (low probability, high blast radius)

Before:

rm -rf "$skills_dest/$skill_name"

After:

rm -rf "${skills_dest:?}/${skill_name:?}"

Why it's a bug: If $skill_name were ever unset or empty (e.g. unexpected glob expansion behaviour), this expands to rm -rf "$skills_dest/" — wiping the entire skills destination directory. set -euo pipefail at the top of the script reduces the probability (unset variable would abort), but the :? guard is the correct defensive pattern and what shellcheck SC2115 requires. The cost of the fix is zero; the blast radius of the failure is not.

Fix 2 — `kyberforge/.claude-plugin/plugin.json` missing `version` field

File: plugins/kyberforge/.claude-plugin/plugin.json
Severity: Strict-mode validation failure

Before:

{
  "name": "kyberforge",
  "displayName": "Kyberforge",
  "description": "...",
  ...
}

After:

{
  "name": "kyberforge",
  "displayName": "Kyberforge",
  "version": "1.0.0",
  "description": "...",
  ...
}

Why it's a bug: claude plugin validate --strict treats a missing version field as an error (exit 1). Any CI pipeline using strict mode would fail. The field is documented as recommended (semver) in the plugin manifest spec.

Fix 3 — `agents/README.md` treated as an agent definition by the validator

File: plugins/kyberforge/agents/README.md → moved to plugins/kyberforge/docs/adding-agents.md
Severity: Strict-mode validation failure

Why it's a bug: claude plugin validate scans all .md files in the agents/ directory and expects them to be agent definitions with YAML frontmatter. The file was a contributor guide explaining how to add agents to the plugin — not an agent definition. Kyberforge currently ships no agents, so the agents/ directory is now correctly empty. Moving the file to docs/ preserves the content without confusing the validator.

Untracked issues — require follow-up

U1 — Gitleaks false positive blocks pre-commit hook on all machines

File: docs/research/ai-coding-factory/ai-coding-factory-session.md:90
Rule triggered: generic-api-key
Match: Token routing: Haiku/Sonnet/Opus
Why false positive: Plain prose in a research document describing Claude model tier routing. The word "Token" followed by a slash-delimited string triggers the entropy rule. No credential present.
Impact: The pre-commit gitleaks hook will block any commit on any machine where gitleaks is installed. First introduced in commit ace8b53 (2026-05-16).
Fix: Add an allowlist entry to scripts/gitleaks.toml:

[allowlist]
description = "False positive — 'Token routing' prose in research doc, not a credential"
paths = ['''docs/research/ai-coding-factory/ai-coding-factory-session\.md''']

Or use the fingerprint-based allowlist if the gitleaks version supports it.

U2 — `write-skill` eval asserts deprecated 8-section body format

File: plugins/kyberforge/skills/write-skill/evals/eval.yaml lines 41–49
Issue: The output-has-all-sections test case asserts that output contains ## Role and ## When to use / When not to use. Both sections were dropped from the authoring standard in the write-skill Phase 1 Refactor (2026-05-18). A skill that correctly follows the current standard will false-fail this test.
Fix: Update the test to assert the current 6-section structure: Required inputs, Constraints, Process, Output format, Failure handling, Self-check. Remove assertions for ## Role and ## When to use / When not to use.
Note: Pre-0019 cleanup item #3 says "run write-eval against write-skill to extend coverage" but does not specifically note this staleness. The stale assertion is a bug in the existing eval, not just a coverage gap.

U3 — `improve-codebase-architecture` skill uses `../` relative paths (plugin-safety risk)

File: .agents/skills/improve-codebase-architecture/SKILL.md lines 68 and 70
Paths referenced:

../grill-with-docs/CONTEXT-FORMAT.md
../grill-with-docs/ADR-FORMAT.md

Current status: Both files exist on disk (CONTEXT.md principle: verified, not assumed). No bug today.
Risk: CONTEXT.md states that skills inside plugins are self-contained and cannot reference files outside their own directory after install-time caching. If this skill is ever packaged into a plugin, these relative paths silently break — the referenced files would not be present. The inventory.sh cross-reference scanner caught this (11 ../ warnings across skill files).
Fix: Either make the skill self-contained (inline the relevant content from those files) or add a note to the Chunk 3 rebuild task for this skill flagging the dependency.

U4 — `improve-codebase-architecture/scripts/` cross-reference warnings (11 total)

inventory.sh from marketplace-architect reported 11 cross-reference warnings for ../ paths across .agents/skills/ files. U3 above is the most concrete example. A full audit of all 11 is needed to determine which are in files already scheduled for Chunk 3 rebuild (acceptable — they will be rewritten) vs. which are in post-refactor skills that should be self-contained.
Files to check: Run bash plugins/kyberforge/skills/marketplace-architect/scripts/inventory.sh and review the cross-reference section.

Already-tracked items confirmed in this audit

Item	Tracked in	Status
13/18 SKILL.md files non-conformant (pre-refactor stubs, no META.md, no category, custom sections)	Issue 0028 (Chunk 3 closure)	Expected; no action needed until rebuild
`write-docs` stale frontmatter + old 8-section structure	Issue 0028	Expected
`write-eval` stale frontmatter + old 8-section structure	Pre-0019 cleanup item #2	Expected
11/18 skills have no eval	Chunk 3 rebuild scope	Expected
`shellcheck` SC2034 false positives in `deploy-manifest.sh`	—	False positive; shellcheck cannot follow `source`; no fix needed

Kyberforge plugin — improvement opportunities identified

These are not bugs but quality gaps discovered during the audit. Relevant for the kyberforge improvement workstream the user mentioned.

write-eval needs refactor to 6-section standard — currently follows the old 8-section format with Role + When/Not sections and provenance fields in SKILL.md frontmatter. Pre-0019 cleanup item #2. Should use write-skill to author the refactored version.
write-skill eval stale assertions — see U2 above.
agents/ directory is now empty — Fix 3 moved the only file out. The root plugin.json still has "agents": "agents/". This reference should either be removed or the directory should house actual agent definitions if/when kyberforge adds agents. Low priority but leaves a dangling reference.
scripts/validate.sh and scripts/inventory.sh are not integrated into the shell test suite — test_scripts.sh covers unit tests for these scripts but does not run them against the live repo as part of tests/. The inventory.sh cross-reference output is useful for ongoing quality checks and could be a tests/test-inventory.sh that asserts zero ../ warnings in post-refactor skills.
No CI pipeline runs claude plugin validate --strict — All the plugin validation work done in this session was manual. A CI gate (Chunk 6 scope) should run this automatically on every push that touches plugins/ or .claude-plugin/.
plugin-create assets may need updating — The assets/plugin-template/agents/README.md (the template for new plugins) is a separate file from the one moved in Fix 3. Verify it still reflects the correct convention for agent directories and that new plugins scaffolded by plugin-create will also pass claude plugin validate --strict.

Recommended follow-up actions (priority order)

Fix U1 — add gitleaks allowlist entry; unblocks pre-commit hook on all machines. 5-minute fix.
Fix U2 — update write-skill eval stale assertions; prevents false eval failures before 0019 work begins.
Validate fixes 1–3 — open a fresh session, run bash scripts/install.sh in an isolated environment, run claude plugin validate --strict on kyberforge, confirm all three fixes hold.
Audit U4 — run inventory.sh, triage the 11 ../ cross-reference warnings; close or create follow-up issues per skill.
Kyberforge improvements — address items 3–6 from the improvements section above as a small focused workstream.
Add claude plugin validate --strict to the CI spec — note in docs/research/governance_principles/CONTROLS.md or the Chunk 6 grill as a required gate for plugins/ changes.

Session context

Branch: main at commit 117e07f
Checks run: 2026-06-21
Agents used: 6 parallel subagents (shell-tests, manifest-checks, skill-conformance, secret-scan, plugin-scripts, claude-plugin-validate)
Tests not run (require interactive Claude session): test-governance-layer.sh, test-instructions-and-docs.sh
Tests not run (system-modifying): test-install.sh
claude plugin validate was not in the initial check sweep — discovered as a gap when user flagged it; added in second wave

## Overview A complete test audit was run on this session (2026-06-21). All automatable checks were executed via parallel subagents. Three issues were fixed inline. Four new issues were found that are not yet tracked. The kyberforge plugin has several follow-on improvement opportunities. This issue is the handoff document for a follow-up session. --- ## Process followed ### 1. Discovery Surveyed the repo to identify all runnable checks before launching agents: - `tests/*.sh` — shell test scripts - `scripts/check-manifests.sh` — manifest validator - `.agents/evals/` and `plugins/kyberforge/*/evals/eval.yaml` — eval files - `plugins/kyberforge/skills/marketplace-architect/scripts/` — plugin utility scripts - `claude plugin validate` — Claude Code plugin manifest validator (discovered as a gap mid-session) ### 2. Parallel execution Launched 4 subagents in parallel (first wave): - **shell-tests** — ran all 5 automatable test scripts - **manifest-checks** — JSON validity, `check-manifests.sh`, `bash -n` syntax, shellcheck - **skill-conformance** — audited every SKILL.md against the 6-section / META.md / 3-field-frontmatter standard - **secret-scan** — gitleaks full scan + manual grep for credential patterns Then a 5th agent (second wave) after user flagged the gap: - **plugin-scripts** — ran `marketplace-architect/scripts/test_scripts.sh`, `inventory.sh`, `validate.sh`, hitl-loop template syntax check Then a 6th agent after confirming `claude plugin validate` had not been run: - **claude-plugin-validate** — ran `claude plugin validate` (normal + `--strict`) on kyberforge plugin and both marketplace manifests ### 3. Fix and verify - Applied 3 fixes inline - Re-ran `claude plugin validate --strict` after fixes to confirm both targets now pass --- ## Results — what passed | Check | Result | |---|---| | Shell tests — 77 cases across 5 scripts | ✅ All pass | | `marketplace-architect/scripts/test_scripts.sh` — 18 cases | ✅ All pass | | `scripts/check-manifests.sh` | ✅ Pass | | JSON validity — all manifest files | ✅ Pass | | `bash -n` syntax — all scripts and test files | ✅ Pass | | Eval structure — 7 `eval.yaml` files | ✅ Pass (all 5 required test types present) | | `gitleaks detect --no-git` (working tree) | ✅ Clean | | `claude plugin validate plugins/kyberforge` | ✅ Pass (after fixes) | | `claude plugin validate .claude-plugin/marketplace.json` | ✅ Pass (after fixes) | | `claude plugin validate .github/plugin/marketplace.json` | ✅ Pass (unchanged) | **Tests skipped by design** (require live Claude session or modify the system): - `tests/test-install.sh` — modifies `~/.claude/`; needs a clean isolated environment - `tests/test-governance-layer.sh` — requires a live interactive Claude session - `tests/test-instructions-and-docs.sh` — requires a live interactive Claude session --- ## Fixes applied this session ### Fix 1 — `install.sh` SC2115: `rm -rf` without `:?` guard **File:** `scripts/install.sh` line 42 **Severity:** Real risk (low probability, high blast radius) **Before:** ```bash rm -rf "$skills_dest/$skill_name" ``` **After:** ```bash rm -rf "${skills_dest:?}/${skill_name:?}" ``` **Why it's a bug:** If `$skill_name` were ever unset or empty (e.g. unexpected glob expansion behaviour), this expands to `rm -rf "$skills_dest/"` — wiping the entire skills destination directory. `set -euo pipefail` at the top of the script reduces the probability (unset variable would abort), but the `:?` guard is the correct defensive pattern and what shellcheck SC2115 requires. The cost of the fix is zero; the blast radius of the failure is not. --- ### Fix 2 — `kyberforge/.claude-plugin/plugin.json` missing `version` field **File:** `plugins/kyberforge/.claude-plugin/plugin.json` **Severity:** Strict-mode validation failure **Before:** ```json { "name": "kyberforge", "displayName": "Kyberforge", "description": "...", ... } ``` **After:** ```json { "name": "kyberforge", "displayName": "Kyberforge", "version": "1.0.0", "description": "...", ... } ``` **Why it's a bug:** `claude plugin validate --strict` treats a missing `version` field as an error (exit 1). Any CI pipeline using strict mode would fail. The field is documented as recommended (semver) in the plugin manifest spec. --- ### Fix 3 — `agents/README.md` treated as an agent definition by the validator **File:** `plugins/kyberforge/agents/README.md` → moved to `plugins/kyberforge/docs/adding-agents.md` **Severity:** Strict-mode validation failure **Why it's a bug:** `claude plugin validate` scans all `.md` files in the `agents/` directory and expects them to be agent definitions with YAML frontmatter. The file was a contributor guide explaining how to add agents to the plugin — not an agent definition. Kyberforge currently ships no agents, so the agents/ directory is now correctly empty. Moving the file to `docs/` preserves the content without confusing the validator. --- ## Untracked issues — require follow-up ### U1 — Gitleaks false positive blocks pre-commit hook on all machines **File:** `docs/research/ai-coding-factory/ai-coding-factory-session.md:90` **Rule triggered:** `generic-api-key` **Match:** `Token routing: Haiku/Sonnet/Opus` **Why false positive:** Plain prose in a research document describing Claude model tier routing. The word "Token" followed by a slash-delimited string triggers the entropy rule. No credential present. **Impact:** The pre-commit gitleaks hook will block any commit on any machine where gitleaks is installed. First introduced in commit `ace8b53` (2026-05-16). **Fix:** Add an allowlist entry to `scripts/gitleaks.toml`: ```toml [allowlist] description = "False positive — 'Token routing' prose in research doc, not a credential" paths = ['''docs/research/ai-coding-factory/ai-coding-factory-session\.md'''] ``` Or use the fingerprint-based allowlist if the gitleaks version supports it. --- ### U2 — `write-skill` eval asserts deprecated 8-section body format **File:** `plugins/kyberforge/skills/write-skill/evals/eval.yaml` lines 41–49 **Issue:** The `output-has-all-sections` test case asserts that output contains `## Role` and `## When to use / When not to use`. Both sections were dropped from the authoring standard in the write-skill Phase 1 Refactor (2026-05-18). A skill that correctly follows the current standard will **false-fail** this test. **Fix:** Update the test to assert the current 6-section structure: Required inputs, Constraints, Process, Output format, Failure handling, Self-check. Remove assertions for `## Role` and `## When to use / When not to use`. **Note:** Pre-0019 cleanup item #3 says "run write-eval against write-skill to extend coverage" but does not specifically note this staleness. The stale assertion is a bug in the existing eval, not just a coverage gap. --- ### U3 — `improve-codebase-architecture` skill uses `../` relative paths (plugin-safety risk) **File:** `.agents/skills/improve-codebase-architecture/SKILL.md` lines 68 and 70 **Paths referenced:** - `../grill-with-docs/CONTEXT-FORMAT.md` - `../grill-with-docs/ADR-FORMAT.md` **Current status:** Both files exist on disk (`CONTEXT.md` principle: verified, not assumed). No bug today. **Risk:** CONTEXT.md states that skills inside plugins are self-contained and cannot reference files outside their own directory after install-time caching. If this skill is ever packaged into a plugin, these relative paths silently break — the referenced files would not be present. The `inventory.sh` cross-reference scanner caught this (11 `../` warnings across skill files). **Fix:** Either make the skill self-contained (inline the relevant content from those files) or add a note to the Chunk 3 rebuild task for this skill flagging the dependency. --- ### U4 — `improve-codebase-architecture/scripts/` cross-reference warnings (11 total) `inventory.sh` from marketplace-architect reported 11 cross-reference warnings for `../` paths across `.agents/skills/` files. U3 above is the most concrete example. A full audit of all 11 is needed to determine which are in files already scheduled for Chunk 3 rebuild (acceptable — they will be rewritten) vs. which are in post-refactor skills that should be self-contained. **Files to check:** Run `bash plugins/kyberforge/skills/marketplace-architect/scripts/inventory.sh` and review the cross-reference section. --- ## Already-tracked items confirmed in this audit | Item | Tracked in | Status | |---|---|---| | 13/18 SKILL.md files non-conformant (pre-refactor stubs, no META.md, no category, custom sections) | Issue 0028 (Chunk 3 closure) | Expected; no action needed until rebuild | | `write-docs` stale frontmatter + old 8-section structure | Issue 0028 | Expected | | `write-eval` stale frontmatter + old 8-section structure | Pre-0019 cleanup item #2 | Expected | | 11/18 skills have no eval | Chunk 3 rebuild scope | Expected | | `shellcheck` SC2034 false positives in `deploy-manifest.sh` | — | False positive; shellcheck cannot follow `source`; no fix needed | --- ## Kyberforge plugin — improvement opportunities identified These are not bugs but quality gaps discovered during the audit. Relevant for the kyberforge improvement workstream the user mentioned. 1. **`write-eval` needs refactor to 6-section standard** — currently follows the old 8-section format with Role + When/Not sections and provenance fields in SKILL.md frontmatter. Pre-0019 cleanup item #2. Should use `write-skill` to author the refactored version. 2. **`write-skill` eval stale assertions** — see U2 above. 3. **`agents/` directory is now empty** — Fix 3 moved the only file out. The root `plugin.json` still has `"agents": "agents/"`. This reference should either be removed or the directory should house actual agent definitions if/when kyberforge adds agents. Low priority but leaves a dangling reference. 4. **`scripts/validate.sh` and `scripts/inventory.sh` are not integrated into the shell test suite** — `test_scripts.sh` covers unit tests for these scripts but does not run them against the live repo as part of `tests/`. The `inventory.sh` cross-reference output is useful for ongoing quality checks and could be a `tests/test-inventory.sh` that asserts zero `../` warnings in post-refactor skills. 5. **No CI pipeline runs `claude plugin validate --strict`** — All the plugin validation work done in this session was manual. A CI gate (Chunk 6 scope) should run this automatically on every push that touches `plugins/` or `.claude-plugin/`. 6. **`plugin-create` assets may need updating** — The `assets/plugin-template/agents/README.md` (the template for new plugins) is a separate file from the one moved in Fix 3. Verify it still reflects the correct convention for agent directories and that new plugins scaffolded by `plugin-create` will also pass `claude plugin validate --strict`. --- ## Recommended follow-up actions (priority order) 1. **Fix U1** — add gitleaks allowlist entry; unblocks pre-commit hook on all machines. 5-minute fix. 2. **Fix U2** — update `write-skill` eval stale assertions; prevents false eval failures before 0019 work begins. 3. **Validate fixes 1–3** — open a fresh session, run `bash scripts/install.sh` in an isolated environment, run `claude plugin validate --strict` on kyberforge, confirm all three fixes hold. 4. **Audit U4** — run `inventory.sh`, triage the 11 `../` cross-reference warnings; close or create follow-up issues per skill. 5. **Kyberforge improvements** — address items 3–6 from the improvements section above as a small focused workstream. 6. **Add `claude plugin validate --strict` to the CI spec** — note in `docs/research/governance_principles/CONTROLS.md` or the Chunk 6 grill as a required gate for `plugins/` changes. --- ## Session context - Branch: `main` at commit `117e07f` - Checks run: 2026-06-21 - Agents used: 6 parallel subagents (shell-tests, manifest-checks, skill-conformance, secret-scan, plugin-scripts, claude-plugin-validate) - Tests not run (require interactive Claude session): `test-governance-layer.sh`, `test-instructions-and-docs.sh` - Tests not run (system-modifying): `test-install.sh` - `claude plugin validate` was not in the initial check sweep — discovered as a gap when user flagged it; added in second wave

Claude commented

2026-06-21 01:20:39 +00:00

Commits from this session

All fixes described in this issue have been committed and pushed to main.

Commit	Fix
`ce7dd15`	`fix(hooks)`: pass `-x` to shellcheck and fix `source=` directive path — pre-existing bug exposed when `install.sh` was first staged post-audit
`247bd4a`	`fix(kyberforge)`: add `version` field to claude-code plugin manifest
`a3ff72c`	`fix(kyberforge)`: move `agents/README.md` to `docs/` — not an agent definition
`34c93d9`	`chore(docs)`: remove `.gitkeep` placeholders from `docs/ard/` and `docs/bug/`

Note on Fix 1 (install.sh SC2115 :? guard): this change was folded into commit ce7dd15 alongside the shellcheck -x fix, since both touched install.sh and both were required to get the hook to pass.

Post-push CI: all remote hooks passed — 126 install tests, 23 governance checks, 5 manifest checks, 34 hook setup tests. All green.

Still open from this issue: U1 (gitleaks allowlist), U2 (write-skill eval stale assertions), U3/U4 (../ cross-reference audit), and the kyberforge improvement items. None are blocking current work.

## Commits from this session All fixes described in this issue have been committed and pushed to `main`. | Commit | Fix | |---|---| | [`ce7dd15`](https://git.dev.rkdr.net/Defame1297/holocron/commit/ce7dd15) | `fix(hooks)`: pass `-x` to shellcheck and fix `source=` directive path — pre-existing bug exposed when `install.sh` was first staged post-audit | | [`247bd4a`](https://git.dev.rkdr.net/Defame1297/holocron/commit/247bd4a) | `fix(kyberforge)`: add `version` field to claude-code plugin manifest | | [`a3ff72c`](https://git.dev.rkdr.net/Defame1297/holocron/commit/a3ff72c) | `fix(kyberforge)`: move `agents/README.md` to `docs/` — not an agent definition | | [`34c93d9`](https://git.dev.rkdr.net/Defame1297/holocron/commit/34c93d9) | `chore(docs)`: remove `.gitkeep` placeholders from `docs/ard/` and `docs/bug/` | **Note on Fix 1 (`install.sh` SC2115 `:?` guard):** this change was folded into commit `ce7dd15` alongside the shellcheck `-x` fix, since both touched `install.sh` and both were required to get the hook to pass. **Post-push CI:** all remote hooks passed — 126 install tests, 23 governance checks, 5 manifest checks, 34 hook setup tests. All green. **Still open from this issue:** U1 (gitleaks allowlist), U2 (write-skill eval stale assertions), U3/U4 (`../` cross-reference audit), and the kyberforge improvement items. None are blocking current work.

Claude commented

2026-06-21 01:30:40 +00:00

Test suite gap analysis — handoff for follow-up session

A structural audit was run against docs/research/governance_principles/CONTROLS.md to identify what the test suite is missing. Seven gaps are immediately actionable (no CI required). Five more are Chunk 6 scope. docs/ROADMAP.md has been updated with this breakdown under the Governance workstream section.

Immediately actionable — 7 items, no Chunk 6 dependency

Work these in order. Items 2–4 depend on item 1 being resolved first.

1. Fix U1: gitleaks false positive before anything else

Why first: any gitleaks scan test will fail until this is resolved. Pre-commit hook blocks commits on all machines with gitleaks installed.

File: scripts/gitleaks.toml
Match: Token routing: Haiku/Sonnet/Opus in docs/research/ai-coding-factory/ai-coding-factory-session.md:90 triggers generic-api-key (entropy match on "Token"). Not a credential.

Fix:

[allowlist]
description = "False positive — 'Token routing' prose in research doc, not a credential"
paths = ['''docs/research/ai-coding-factory/ai-coding-factory-session\.md''']

Verify: gitleaks detect --source . --config scripts/gitleaks.toml exits 0.

2. `tests/test-gitleaks-scan.sh` — run scan against the live repo

Gap: test-setup-gitleaks.sh tests that the setup script works; nothing tests that the scan itself passes. The U1 false positive went undetected until a manual audit.

What to write:

Run gitleaks detect --source . --config scripts/gitleaks.toml --no-git and assert exit 0
Run gitleaks detect --source . --config scripts/gitleaks.toml (with git history) and assert exit 0
Assert scripts/gitleaks.toml has at least one [allowlist] block (regression guard — a config with no allowlist is a config that hasn't been validated)

Requires: item 1 done first.

3. `tests/test-plugin-validate.sh` — `claude plugin validate --strict` on all targets

Gap: Discovered mid-session as a manual check. Not in tests/ and not in the pre-push hook.

What to write:

claude plugin validate plugins/kyberforge --strict → assert exit 0
claude plugin validate .claude-plugin/marketplace.json --strict → assert exit 0
claude plugin validate .github/plugin/marketplace.json --strict → assert exit 0

Also: add claude plugin validate .claude-plugin/marketplace.json --strict to the pre-push hook in scripts/setup-hooks.sh (and update .git/hooks/pre-push). Current pre-push only runs check-manifests.sh.

4. `tests/test-hook-integrity.sh` — verify installed hook, not setup script

Gap: test-setup-hooks.sh tests that setup-hooks.sh produces the correct hook. Nothing tests that the hook is actually installed and active — the control CONTROLS.md calls out for monthly verification.

What to write:

Assert .git/hooks/pre-commit exists and is executable
Assert .git/hooks/pre-commit contains the gitleaks marker (# managed by setup-gitleaks.sh)
Assert .git/hooks/pre-commit contains the shellcheck/jq/yq/SKILL.md marker (# managed by setup-hooks.sh)
Assert .git/hooks/commit-msg exists and is executable
Assert .git/hooks/pre-push exists and is executable
Assert .git/hooks/pre-push references check-manifests.sh

5. `tests/test-inventory-crossrefs.sh` — assert clean cross-reference output

Gap: marketplace-architect/scripts/test_scripts.sh unit-tests inventory.sh but nothing runs it against the live repo to catch ../ path warnings.

What to write:

Run bash plugins/kyberforge/skills/marketplace-architect/scripts/inventory.sh from repo root
Capture the cross-reference warning section
For each warning: assert the file is in a known pre-refactor skill (i.e. scheduled for Chunk 3 rebuild — the 11 listed in issue 0028). If a post-refactor skill appears, FAIL.
Post-refactor skills (must be cross-ref-clean): gitleaks, neuledge-context, write-docs, write-skill (in .agents/skills/), plus all kyberforge plugin skills.
Pre-refactor stubs (warnings acceptable, skip): caveman, diagnose, grill-me, grill-with-docs, improve-codebase-architecture, prototype, tdd, to-issues, to-prd, triage, zoom-out.

Also resolves U4 from the original issue — the triage of the 11 ../ warnings.

6. Extend `test-governance-layer.sh` — verify controls are enforced, not just files present

Gap: Current checks verify governance.md, ai-constitution.md, HUMANS.md exist and contain expected content. CONTROLS.md requires that the controls themselves are in place.

Add to the existing script:

Assert .git/hooks/pre-commit exists and is executable (gitleaks wired)
Assert gitleaks is in $PATH (pre-commit hook is effective, not silently skipped)
Assert claude plugin validate .claude-plugin/marketplace.json --strict exits 0 (plugin manifests meet the spec)
Assert scripts/gitleaks.toml has at least one [allowlist] block (scan has been validated)
Add a CONTROLS.md reference block to the test output so the human reviewer can see what requirements each check enforces

7. `tests/run-all-tests.sh` — single entry point for all tests

Gap: No way to run everything with one command. Required for CI integration (Chunk 6) and useful now.

What to write:

#!/usr/bin/env bash
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
PASS=0; FAIL=0
run() {
  echo "--- $1 ---"
  if bash "$REPO_ROOT/tests/$1"; then PASS=$((PASS+1))
  else FAIL=$((FAIL+1)); fi
}
run test-check-manifests.sh
run test-setup-hooks.sh
run test-setup-gitleaks.sh
run test-gitleaks-scan.sh        # new
run test-plugin-validate.sh      # new
run test-hook-integrity.sh       # new
run test-inventory-crossrefs.sh  # new
run test-neuledge-context.sh
run test-statusline.sh
# skipped: test-install.sh (system-modifying), test-governance-layer.sh / test-instructions-and-docs.sh (require live Claude session)
echo ""
echo "Results: $PASS passed, $FAIL failed"
[[ $FAIL -eq 0 ]]

Chunk 6 CI gaps — 5 items, blocked on CI pipeline

These require a .gitea/workflows/ CI pipeline. Document them now so the Chunk 6 grill starts with a concrete list.

Requirement	CONTROLS.md quote	Status
Secret scanning in CI	"Pre-commit hooks can be bypassed; CI cannot. Both layers are required."	No CI pipeline
Dependency/security scanning in CI	"Every repository CI pipeline must include dependency vulnerability scanning"	No CI pipeline
Licence scanning in CI	"Must cover code content, not just declared deps — for AI-assisted contributions"	No CI pipeline
Human approval gate in CI/CD	"Any pipeline applying production changes must include an explicit human approval step"	No CI/CD
Audit logging for agentic workflows	"Every state-modifying agentic workflow must produce a tamper-evident log"	No agentic workflows yet

When the Chunk 6 grill begins, read this issue alongside docs/research/governance_principles/CONTROLS.md and docs/ROADMAP.md (Governance workstream section).

Session state when this comment was written

Branch: main at commit 34c93d9
docs/ROADMAP.md updated with pre-Chunk 6 test work and Chunk 6 CI gaps (same session, not yet committed)
docs/ard/ and docs/bug/ directories removed (.gitkeep committed in 34c93d9)
Pre-0019 work (0018 phase 3, write-eval refactor, eval updates) is the current active workstream — these test suite items do not block it

## Test suite gap analysis — handoff for follow-up session A structural audit was run against `docs/research/governance_principles/CONTROLS.md` to identify what the test suite is missing. Seven gaps are immediately actionable (no CI required). Five more are Chunk 6 scope. `docs/ROADMAP.md` has been updated with this breakdown under the Governance workstream section. --- ## Immediately actionable — 7 items, no Chunk 6 dependency Work these in order. Items 2–4 depend on item 1 being resolved first. --- ### 1. Fix U1: gitleaks false positive before anything else **Why first:** any gitleaks scan test will fail until this is resolved. Pre-commit hook blocks commits on all machines with gitleaks installed. **File:** `scripts/gitleaks.toml` **Match:** `Token routing: Haiku/Sonnet/Opus` in `docs/research/ai-coding-factory/ai-coding-factory-session.md:90` triggers `generic-api-key` (entropy match on "Token"). Not a credential. **Fix:** ```toml [allowlist] description = "False positive — 'Token routing' prose in research doc, not a credential" paths = ['''docs/research/ai-coding-factory/ai-coding-factory-session\.md'''] ``` **Verify:** `gitleaks detect --source . --config scripts/gitleaks.toml` exits 0. --- ### 2. `tests/test-gitleaks-scan.sh` — run scan against the live repo **Gap:** `test-setup-gitleaks.sh` tests that the setup script works; nothing tests that the scan itself passes. The U1 false positive went undetected until a manual audit. **What to write:** - Run `gitleaks detect --source . --config scripts/gitleaks.toml --no-git` and assert exit 0 - Run `gitleaks detect --source . --config scripts/gitleaks.toml` (with git history) and assert exit 0 - Assert `scripts/gitleaks.toml` has at least one `[allowlist]` block (regression guard — a config with no allowlist is a config that hasn't been validated) **Requires:** item 1 done first. --- ### 3. `tests/test-plugin-validate.sh` — `claude plugin validate --strict` on all targets **Gap:** Discovered mid-session as a manual check. Not in `tests/` and not in the pre-push hook. **What to write:** - `claude plugin validate plugins/kyberforge --strict` → assert exit 0 - `claude plugin validate .claude-plugin/marketplace.json --strict` → assert exit 0 - `claude plugin validate .github/plugin/marketplace.json --strict` → assert exit 0 **Also:** add `claude plugin validate .claude-plugin/marketplace.json --strict` to the pre-push hook in `scripts/setup-hooks.sh` (and update `.git/hooks/pre-push`). Current pre-push only runs `check-manifests.sh`. --- ### 4. `tests/test-hook-integrity.sh` — verify installed hook, not setup script **Gap:** `test-setup-hooks.sh` tests that `setup-hooks.sh` *produces* the correct hook. Nothing tests that the hook is actually *installed and active* — the control CONTROLS.md calls out for monthly verification. **What to write:** - Assert `.git/hooks/pre-commit` exists and is executable - Assert `.git/hooks/pre-commit` contains the gitleaks marker (`# managed by setup-gitleaks.sh`) - Assert `.git/hooks/pre-commit` contains the shellcheck/jq/yq/SKILL.md marker (`# managed by setup-hooks.sh`) - Assert `.git/hooks/commit-msg` exists and is executable - Assert `.git/hooks/pre-push` exists and is executable - Assert `.git/hooks/pre-push` references `check-manifests.sh` --- ### 5. `tests/test-inventory-crossrefs.sh` — assert clean cross-reference output **Gap:** `marketplace-architect/scripts/test_scripts.sh` unit-tests `inventory.sh` but nothing runs it against the live repo to catch `../` path warnings. **What to write:** - Run `bash plugins/kyberforge/skills/marketplace-architect/scripts/inventory.sh` from repo root - Capture the cross-reference warning section - For each warning: assert the file is in a known pre-refactor skill (i.e. scheduled for Chunk 3 rebuild — the 11 listed in issue 0028). If a post-refactor skill appears, FAIL. - Post-refactor skills (must be cross-ref-clean): `gitleaks`, `neuledge-context`, `write-docs`, `write-skill` (in `.agents/skills/`), plus all kyberforge plugin skills. - Pre-refactor stubs (warnings acceptable, skip): `caveman`, `diagnose`, `grill-me`, `grill-with-docs`, `improve-codebase-architecture`, `prototype`, `tdd`, `to-issues`, `to-prd`, `triage`, `zoom-out`. **Also resolves U4** from the original issue — the triage of the 11 `../` warnings. --- ### 6. Extend `test-governance-layer.sh` — verify controls are enforced, not just files present **Gap:** Current checks verify `governance.md`, `ai-constitution.md`, `HUMANS.md` exist and contain expected content. CONTROLS.md requires that the *controls themselves* are in place. **Add to the existing script:** - Assert `.git/hooks/pre-commit` exists and is executable (gitleaks wired) - Assert `gitleaks` is in `$PATH` (pre-commit hook is effective, not silently skipped) - Assert `claude plugin validate .claude-plugin/marketplace.json --strict` exits 0 (plugin manifests meet the spec) - Assert `scripts/gitleaks.toml` has at least one `[allowlist]` block (scan has been validated) - Add a CONTROLS.md reference block to the test output so the human reviewer can see what requirements each check enforces --- ### 7. `tests/run-all-tests.sh` — single entry point for all tests **Gap:** No way to run everything with one command. Required for CI integration (Chunk 6) and useful now. **What to write:** ```bash #!/usr/bin/env bash set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" PASS=0; FAIL=0 run() { echo "--- $1 ---" if bash "$REPO_ROOT/tests/$1"; then PASS=$((PASS+1)) else FAIL=$((FAIL+1)); fi } run test-check-manifests.sh run test-setup-hooks.sh run test-setup-gitleaks.sh run test-gitleaks-scan.sh # new run test-plugin-validate.sh # new run test-hook-integrity.sh # new run test-inventory-crossrefs.sh # new run test-neuledge-context.sh run test-statusline.sh # skipped: test-install.sh (system-modifying), test-governance-layer.sh / test-instructions-and-docs.sh (require live Claude session) echo "" echo "Results: $PASS passed, $FAIL failed" [[ $FAIL -eq 0 ]] ``` --- ## Chunk 6 CI gaps — 5 items, blocked on CI pipeline These require a `.gitea/workflows/` CI pipeline. Document them now so the Chunk 6 grill starts with a concrete list. | Requirement | CONTROLS.md quote | Status | |---|---|---| | Secret scanning in CI | "Pre-commit hooks can be bypassed; CI cannot. Both layers are required." | No CI pipeline | | Dependency/security scanning in CI | "Every repository CI pipeline must include dependency vulnerability scanning" | No CI pipeline | | Licence scanning in CI | "Must cover code content, not just declared deps — for AI-assisted contributions" | No CI pipeline | | Human approval gate in CI/CD | "Any pipeline applying production changes must include an explicit human approval step" | No CI/CD | | Audit logging for agentic workflows | "Every state-modifying agentic workflow must produce a tamper-evident log" | No agentic workflows yet | When the Chunk 6 grill begins, read this issue alongside `docs/research/governance_principles/CONTROLS.md` and `docs/ROADMAP.md` (Governance workstream section). --- ## Session state when this comment was written - Branch: `main` at commit `34c93d9` - `docs/ROADMAP.md` updated with pre-Chunk 6 test work and Chunk 6 CI gaps (same session, not yet committed) - `docs/ard/` and `docs/bug/` directories removed (`.gitkeep` committed in `34c93d9`) - Pre-0019 work (0018 phase 3, write-eval refactor, eval updates) is the current active workstream — these test suite items do not block it

Defame1297 referenced this issue from a commit

2026-06-21 01:33:36 +00:00

fix(gitleaks): suppress Token routing false positive; sync allowlists; update roadmap

Claude commented

2026-06-21 01:33:52 +00:00

U1 resolved — `ac8235c`

U1 (gitleaks false positive) fixed in commit ac8235c.

What was fixed:

docs/ROADMAP.md now documents the Token routing: Haiku/Sonnet/Opus pattern, which itself triggered the same generic-api-key rule during the commit that added the test suite gap analysis. The false positive became self-blocking.
Both scripts/gitleaks.toml (source) and .gitleaks.toml (deployed root copy read by the hook) were out of sync — the deployed file already had a docs/research/.* path allowlist (covering the original finding) but the source did not. Both files are now aligned with a shared allowlist covering docs/research/.* and docs/ROADMAP\.md.

Side-effect discovered: scripts/gitleaks.toml and .gitleaks.toml were tracking different allowlist states. Anyone running setup-gitleaks.sh would have overwritten the deployed file with the stale source, losing the existing docs/research/.* allowlist. Now in sync.

U1 is fully closed. The test-gitleaks-scan.sh test (item 2 in the previous comment) can now be written without a prerequisite blocker.

## U1 resolved — [`ac8235c`](https://git.dev.rkdr.net/Defame1297/holocron/commit/ac8235c) **U1 (gitleaks false positive)** fixed in commit `ac8235c`. **What was fixed:** - `docs/ROADMAP.md` now documents the `Token routing: Haiku/Sonnet/Opus` pattern, which itself triggered the same `generic-api-key` rule during the commit that added the test suite gap analysis. The false positive became self-blocking. - Both `scripts/gitleaks.toml` (source) and `.gitleaks.toml` (deployed root copy read by the hook) were out of sync — the deployed file already had a `docs/research/.*` path allowlist (covering the original finding) but the source did not. Both files are now aligned with a shared allowlist covering `docs/research/.*` and `docs/ROADMAP\.md`. **Side-effect discovered:** `scripts/gitleaks.toml` and `.gitleaks.toml` were tracking different allowlist states. Anyone running `setup-gitleaks.sh` would have overwritten the deployed file with the stale source, losing the existing `docs/research/.*` allowlist. Now in sync. **U1 is fully closed.** The `test-gitleaks-scan.sh` test (item 2 in the previous comment) can now be written without a prerequisite blocker.

Claude commented

2026-06-21 01:37:09 +00:00

Additional finding: gitleaks config sync is a recurring risk

Discovered while committing the ROADMAP update (the commit itself was blocked by the false positive pattern appearing in ROADMAP.md).

The structural problem:

scripts/gitleaks.toml (source, tracked in git) and .gitleaks.toml (deployed root copy, also tracked in git) are two separate files that are supposed to stay in sync — setup-gitleaks.sh deploys the source to the root. Found in this session with different allowlist states: the deployed .gitleaks.toml had a docs/research/.* path allowlist (suppressing the original false positive); the source scripts/gitleaks.toml did not.

Current risk: anyone running setup-gitleaks.sh in this repo will overwrite .gitleaks.toml with the stale source, silently removing the allowlist and re-exposing the false positive as a blocking pre-commit failure. The two files were synced in commit ac8235c — but the underlying mechanism that caused the drift is still in place.

Root cause: setup-gitleaks.sh is designed for deploying to other repos (project setup). When run in this repo, it overwrites a tracked file. There's no divergence check and no merge — pure overwrite.

Options for the follow-up session (pick one):

Merge rather than overwrite — update setup-gitleaks.sh to detect when .gitleaks.toml already exists and merge rather than overwrite. Git-style: warn on divergence, require explicit --force to overwrite. Cleanest long-term fix.
Stop tracking .gitleaks.toml in git — add .gitleaks.toml to .gitignore; treat it as a generated file that always comes from setup-gitleaks.sh. Pro: no sync burden. Con: every machine must run setup-gitleaks.sh before the allowlist is active. The pre-commit hook would be the backstop — and the pre-commit gitleaks block references the config by auto-discovery (no --config flag), so the root file must exist.
Remove scripts/gitleaks.toml as a separate source — put the canonical config directly at .gitleaks.toml (tracked), and update setup-gitleaks.sh to copy from the repo root rather than from scripts/. Con: couples the global install script to the repo layout.

Recommended: Option 1. It's the only one that preserves both the deploy-to-other-repos use case and the tracked-source use case without a sync burden.

Also note: the pre-commit hook generated by setup-gitleaks.sh runs gitleaks git --staged --redact -v without a --config flag. Gitleaks auto-discovers .gitleaks.toml at the repo root — which is why the allowlist works. But this auto-discovery assumption is fragile: if .gitleaks.toml is absent (e.g. first checkout before running setup), the hook runs with the default config, and the false positive blocks the commit. A --config scripts/gitleaks.toml flag in the hook would be more explicit and wouldn't depend on the root file existing.

## Additional finding: gitleaks config sync is a recurring risk Discovered while committing the ROADMAP update (the commit itself was blocked by the false positive pattern appearing in ROADMAP.md). **The structural problem:** `scripts/gitleaks.toml` (source, tracked in git) and `.gitleaks.toml` (deployed root copy, also tracked in git) are two separate files that are supposed to stay in sync — `setup-gitleaks.sh` deploys the source to the root. Found in this session with **different allowlist states**: the deployed `.gitleaks.toml` had a `docs/research/.*` path allowlist (suppressing the original false positive); the source `scripts/gitleaks.toml` did not. **Current risk:** anyone running `setup-gitleaks.sh` in this repo will overwrite `.gitleaks.toml` with the stale source, silently removing the allowlist and re-exposing the false positive as a blocking pre-commit failure. The two files were synced in commit `ac8235c` — but the underlying mechanism that caused the drift is still in place. **Root cause:** `setup-gitleaks.sh` is designed for deploying to *other* repos (project setup). When run in this repo, it overwrites a tracked file. There's no divergence check and no merge — pure overwrite. **Options for the follow-up session (pick one):** 1. **Merge rather than overwrite** — update `setup-gitleaks.sh` to detect when `.gitleaks.toml` already exists and merge rather than overwrite. Git-style: warn on divergence, require explicit `--force` to overwrite. Cleanest long-term fix. 2. **Stop tracking `.gitleaks.toml` in git** — add `.gitleaks.toml` to `.gitignore`; treat it as a generated file that always comes from `setup-gitleaks.sh`. Pro: no sync burden. Con: every machine must run `setup-gitleaks.sh` before the allowlist is active. The pre-commit hook would be the backstop — and the pre-commit gitleaks block references the config by auto-discovery (no `--config` flag), so the root file must exist. 3. **Remove `scripts/gitleaks.toml` as a separate source** — put the canonical config directly at `.gitleaks.toml` (tracked), and update `setup-gitleaks.sh` to copy from the repo root rather than from `scripts/`. Con: couples the global install script to the repo layout. **Recommended:** Option 1. It's the only one that preserves both the deploy-to-other-repos use case and the tracked-source use case without a sync burden. **Also note:** the pre-commit hook generated by `setup-gitleaks.sh` runs `gitleaks git --staged --redact -v` without a `--config` flag. Gitleaks auto-discovers `.gitleaks.toml` at the repo root — which is why the allowlist works. But this auto-discovery assumption is fragile: if `.gitleaks.toml` is absent (e.g. first checkout before running setup), the hook runs with the default config, and the false positive blocks the commit. A `--config scripts/gitleaks.toml` flag in the hook would be more explicit and wouldn't depend on the root file existing.

Defame1297 added the

Reviewed

Won't Fix

3

label 2026-06-22 20:06:29 +00:00

Defame1297 closed this issue

2026-06-22 20:06:46 +00:00

Claude referenced this issue

2026-08-08 19:29:30 +00:00

feat(lint): wire Vale as deterministic prefilter for skill-audit/agent-audit #85

Claude referenced this issue

2026-08-08 20:48:43 +00:00