Defame1297/holocron
0
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases 3 Wiki Activity

fix(lint): flatten every multi-line description form in vale-wrap.sh

Browse Source 
Vale locates a frontmatter description by matching the parsed YAML value back
against the source text, so any scalar whose value is not spelled out verbatim
loses the `text.frontmatter.description` scope entirely. The wrapper only
flattened `>` folded scalars, so plain, double-quoted and single-quoted
multi-line descriptions silently reported zero alerts and exit 0 — a clean pass
indistinguishable from a real one, in a prefilter whose callers are instructed
not to re-derive its verdict by judgment.

Implementation notes:

- Classify the scalar kind after `^description:[ \t]*` and reuse one shared
  continuation-line generator for every form; `|` literal blocks keep their
  line breaks, stay verbatim-matchable, and are still left untouched.
- Emit the flattened value in whichever scalar form needs no escape at all
  (plain, then single-quoted, then double-quoted), because any escape breaks
  the verbatim match. The old blanket `'` -> U+2019 substitution silently made
  apostrophe-bearing rule tokens unmatchable across 63% of the corpus; it now
  survives only for the one combination no YAML scalar can carry verbatim.
- Terminate continuations at a line flush with the key, not only on a shallower
  indent — a `description:` followed by a flush-left line previously swallowed
  the rest of the frontmatter.
- Route vale's value-taking flags explicitly instead of inferring targets by
  file existence, and absolutize relative `--output`/`--path` values the way
  `--config` already was, since the run `cd`s into the scratch mirror.
- Fail loudly on a nonexistent path instead of inheriting bare vale's fallback
  to stdin, which rendered a typo'd path as `0 errors ... in stdin`, exit 0 —
  a form the callers' `0 files` NOT-RUN guard cannot match.
- Follow symlinks when walking a directory argument, matching bare vale.

Refs: #85
This commit is contained in:
Defame1297
2026-08-09 15:44:04 +00:00
parent afc2b7fdfd
commit aa8cc22695
3 changed files with 934 additions and 196 deletions
Download Patch File Download Diff File Expand all files Collapse all files

416
plugins/kyberforge/skills/agent-audit/scripts/vale-wrap.sh
View File

@@ -2,27 +2,45 @@
set -euo pipefail
# Works around a Vale limitation: the `text.frontmatter.description` NLP scope
# silently stops matching once the `description:` value is a YAML block scalar
# (`>`/`|`) spanning 2+ physical lines — the style used by most skills/agents in
# this repo. Flattens the description to one physical line in a scratch copy
# (padding with blank lines so every other line number is unchanged), then runs
# the real `vale` binary against the copies. Drop-in replacement for calling
# `vale` directly: same args, same exit code.
# silently stops matching once the `description:` value spans 2+ physical lines
# in any form YAML joins back into one string — a `>`/`>-`/`>+` folded block
# scalar (the style used by most skills/agents in this repo), a plain scalar
# wrapped onto continuation lines, or a double- or single-quoted scalar wrapped
# the same way. A `|`/`|-`/`|+` literal block scalar is NOT affected: its parsed
# value keeps exactly the line breaks the source has, and vale matches it fine
# (verified against vale 3.15.2), so literal blocks are deliberately left alone.
# This script flattens an affected description to one physical line in a scratch
# copy (padding with blank lines so every other line number is unchanged), then
# runs the real `vale` binary against the copies. Drop-in replacement for calling
# `vale` directly: same args, same exit code, bar the two documented divergences
# below.
#
# "Same args" means relative paths — `--config` values and path arguments alike
# — resolve against the caller's current directory, exactly as bare `vale`
# resolves them. (An earlier version resolved them against the repo root, an
# invented convention that hard-errored on `--config ../../.vale.ini` from a
# subdirectory and, worse, silently dropped file arguments that didn't happen to
# resolve from the repo root — skipping the flattening this script exists for.)
# "Same args" means relative paths — path arguments and the values of the
# path-valued flags (`--config`, `--output`, `--path`) alike — resolve against
# the caller's current directory, exactly as bare `vale` resolves them. The flag
# values are rewritten to absolute form because the run ends up `cd`'d into the
# scratch mirror, where a relative one would no longer resolve. (An earlier
# version resolved path arguments against the repo root, an invented convention
# that hard-errored on `--config ../../.vale.ini` from a subdirectory and, worse,
# silently dropped file arguments that didn't happen to resolve from the repo
# root — skipping the flattening this script exists for.)
#
# The one addition to bare `vale`'s argument handling: with no `--config` at
# all, this script's own sibling `assets/vale/.vale.ini` is used instead of
# vale's upward search. pre-commit prefixes only `entry[0]` with the hook-repo
# clone path, so a `--config` in `.pre-commit-hooks.yaml` would resolve against
# the *consuming* repo and hard-fail (E100) for every external consumer. The
# manifest therefore passes the script alone, and an explicit `--config` from
# any other caller still wins.
# Divergence 1: with no `--config` at all, this script's own sibling
# `assets/vale/.vale.ini` is used instead of vale's upward search. pre-commit
# prefixes only `entry[0]` with the hook-repo clone path, so a `--config` in
# `.pre-commit-hooks.yaml` would resolve against the *consuming* repo and
# hard-fail (E100) for every external consumer. The manifest therefore passes the
# script alone, and an explicit `--config` from any other caller still wins.
#
# Divergence 2: a path-shaped argument that does not exist is a hard error
# (exit 2). Bare vale drops it, falls back to reading stdin, and prints
# `0 errors ... in stdin` with exit 0 — a typo'd target is then indistinguishable
# from a clean run. Both audit skills treat a `0 files` report as NOT RUN rather
# than clean, and `in stdin` does not match that guard, so the silent form would
# read as "prefilter clean" and skip the LLM fallback. Erroring is the only way
# to keep that guard honest. Linting prose piped on stdin is therefore
# unsupported here — it already was, since the no-path handoff closes stdin so
# vale can't block on a pipe that will never carry content.
#
# Vale prints each path exactly as it was handed to it, so the scratch tree
# mirrors the caller's absolute cwd: a relative path argument is passed through
@@ -41,22 +59,44 @@ cwd="$(pwd -P)"
# later edit breaking that invariant, not a live fix.
vale_args=()
path_args=()
config_next=false
pending_flag=""
config_given=false
for arg in "$@"; do
if [[ "$config_next" == true ]]; then
config_next=false
if [[ "$arg" == /* ]]; then
vale_args+=("$arg")
else
vale_args+=("$cwd/$arg")
fi
if [[ -n "$pending_flag" ]]; then
# Value of a separated two-argv flag. It is never a lint target, however
# file-like it looks. The run ends up `cd`'d into the scratch mirror, so a
# value naming a file has to be absolutized here or it stops resolving.
case "$pending_flag" in
--config)
# Always a path, and required to exist.
if [[ "$arg" == /* ]]; then
vale_args+=("$arg")
else
vale_args+=("$cwd/$arg")
fi
;;
--output|--path)
# `--output` is either a built-in style name (`line`, `JSON`) or a
# template file; only the file form needs resolving. `--path` is
# likewise a path. Anything that names nothing is passed through and
# left for vale to interpret.
if [[ "$arg" != /* && -e "$arg" ]]; then
vale_args+=("$cwd/$arg")
else
vale_args+=("$arg")
fi
;;
*)
vale_args+=("$arg")
;;
esac
pending_flag=""
continue
fi
case "$arg" in
--config)
vale_args+=("$arg")
config_next=true
pending_flag="$arg"
config_given=true
continue
;;
@@ -70,24 +110,60 @@ for arg in "$@"; do
config_given=true
continue
;;
# Same cwd-relative resolution for the `--flag=value` spelling of the two
# other path-valued flags.
--output=*|--path=*)
flag_val="${arg#*=}"
if [[ "$flag_val" != /* && -n "$flag_val" && -e "$flag_val" ]]; then
vale_args+=("${arg%%=*}=$cwd/$flag_val")
else
vale_args+=("$arg")
fi
continue
;;
# Vale's remaining value-taking flags, per `vale --help` (3.x). In the
# separated two-argv form the value must not be classified as a lint target
# — `--output tmpl.tmpl` names a real template file, and treating it as
# input both lints the template and reorders argv so vale sees
# `--output --no-wrap`. The `--flag=value` form needs no entry here: it
# starts with `-` and falls through to vale untouched. A value flag added by
# some future vale release is simply absent from this list and lands back on
# today's behaviour, so this list going stale is never worse than not having
# it.
--ext|--filter|--glob|--minAlertLevel|--output|--path)
vale_args+=("$arg")
pending_flag="$arg"
continue
;;
# Vale's subcommands are bare words that name no file, so they would trip
# the not-found error below. A lint target literally named `sync` (no
# extension, no slash) is misread as the subcommand — accepted, because the
# alternative is failing every `vale-wrap.sh ls-config`.
ls-config|ls-dirs|ls-metrics|ls-vars|sync)
vale_args+=("$arg")
continue
;;
esac
# `-f`/`-d` resolve relative paths against the caller's cwd, same as vale does.
# A path that doesn't exist is left for vale to report on, exactly as bare
# vale would.
if [[ "$arg" != -* && ( -f "$arg" || -d "$arg" ) ]]; then
# An absolute path inside the caller's cwd is relativized so the report cites
# a path that resolves against the real tree. Left absolute, it would be
# rewritten to its scratch copy and printed as `/tmp/tmp.XXXX/...` — a real
# path to a file that is deleted on exit, which reads as a bug in any report
# quoting it. Absolute paths outside the cwd have no relative form and keep
# the scratch-path behaviour documented above.
if [[ "$arg" == "$cwd"/* ]]; then
path_args+=("${arg#"$cwd"/}")
else
path_args+=("$arg")
fi
else
if [[ "$arg" == -* ]]; then
vale_args+=("$arg")
continue
fi
# Everything left is a lint target: `vale [options] [input...]` has no third
# kind of argument. See divergence 2 above for why a missing one is fatal here.
if [[ ! -e "$arg" ]]; then
echo "vale-wrap.sh: no such file or directory: $arg" >&2
exit 2
fi
# An absolute path inside the caller's cwd is relativized so the report cites
# a path that resolves against the real tree. Left absolute, it would be
# rewritten to its scratch copy and printed as `/tmp/tmp.XXXX/...` — a real
# path to a file that is deleted on exit, which reads as a bug in any report
# quoting it. Absolute paths outside the cwd have no relative form and keep
# the scratch-path behaviour documented above.
if [[ "$arg" == "$cwd"/* ]]; then
path_args+=("${arg#"$cwd"/}")
else
path_args+=("$arg")
fi
done
@@ -119,55 +195,210 @@ src, dest = sys.argv[1], sys.argv[2]
with open(src, encoding='utf-8', errors='surrogateescape') as fh:
content = fh.read()
# YAML 1.2 double-quoted escapes (spec 5.7 / 7.3.1). `\<newline>` is handled
# separately in unescape_double because it also swallows the next indentation.
DQ_ESCAPES = {
'0': '\0', 'a': '\a', 'b': '\b', 't': '\t', '\t': '\t', 'n': '\n',
'v': '\v', 'f': '\f', 'r': '\r', 'e': '\x1b', ' ': ' ', '"': '"',
'/': '/', '\\': '\\', 'N': '\x85', '_': '\xa0', 'L': '\u2028',
'P': '\u2029',
}
# First characters that make a plain (unquoted) scalar mean something other than
# text: YAML's c-indicator set.
PLAIN_UNSAFE_FIRST = '-?:,[]{}#&*!|>\'"%@`'
def unescape_double(text):
"""Decode a double-quoted YAML scalar's body to the string YAML parses."""
out = []
i = 0
while i < len(text):
char = text[i]
if char != '\\':
out.append(char)
i += 1
continue
i += 1
if i >= len(text):
break
esc = text[i]
if esc == '\n':
i += 1
while i < len(text) and text[i] in ' \t':
i += 1
continue
if esc in 'xuU':
width = {'x': 2, 'u': 4, 'U': 8}[esc]
digits = text[i + 1:i + 1 + width]
if len(digits) == width:
try:
out.append(chr(int(digits, 16)))
except ValueError:
pass
else:
i += 1 + width
continue
out.append(DQ_ESCAPES.get(esc, esc))
i += 1
return ''.join(out)
def close_quote(text, quote):
"""Index of the closing `quote` in `text`, which starts just past the
opening one. None while the scalar is still unterminated."""
i = 0
while i < len(text):
char = text[i]
if quote == '"' and char == '\\':
i += 2
continue
if char == quote:
if quote == "'" and text[i + 1:i + 2] == "'":
i += 2
continue
return i
i += 1
return None
def continuation_lines(rest):
"""Yield the physical lines of `rest` that continue the value started on the
`description:` line. Indentation-based and blank-line-tolerant, per YAML:
a blank line (any amount of whitespace) always stays inside; the indent is
set by the first content line; the value ends at the first line indented
less than that, at any line flush with the key (that is the next mapping
key, not a continuation), or at EOF."""
indent = None
for line in rest.splitlines(keepends=True):
text = line.rstrip('\n')
if text.strip() == '':
yield line
continue
line_indent = len(text) - len(text.lstrip(' \t'))
if line_indent == 0:
return
if indent is None:
indent = line_indent
elif line_indent < indent:
return
yield line
def emit(value):
"""Render `value` as a one-line YAML scalar whose source text spells the
value out verbatim. Vale locates the description by matching the parsed
value back against the source, so a scalar carrying any escape — `''` in a
single-quoted scalar, `\\"` or `\\\\` in a double-quoted one — makes the
whole `text.frontmatter.description` scope vanish, the same failure this
script exists to work around. Verbatim forms only, therefore, tried in
descending order of fidelity."""
if (value
and value[0] not in PLAIN_UNSAFE_FIRST
and ': ' not in value
and not value.endswith(':')
and ' #' not in value):
return value # plain: nothing needs escaping at all
if "'" not in value:
return "'" + value + "'" # single-quoted: only `'` would escape
if '"' not in value and '\\' not in value:
return '"' + value + '"' # double-quoted: only `"`/`\` would
# Last resort: the value needs quoting AND holds an apostrophe AND a double
# quote or backslash, so no verbatim YAML scalar can carry it. Substituting
# U+2019 for the apostrophe keeps the scope alive, at the cost of any style
# rule whose token contains a literal ASCII apostrophe. Scratch-copy only —
# never written back to the real file.
return "'" + value.replace("'", '’') + "'"
fm_match = re.match(r'^(---\n)(.*?\n)(---\n)', content, re.DOTALL)
if fm_match:
fm = fm_match.group(2)
# Only `>`/`>-`/`>+` (folded) scalars break Vale's frontmatter-description
# scope. `|`/`|-`/`|+` (literal) scalars already work fine with bare vale,
# so they're deliberately left unmatched here.
header_m = re.search(r'^description:[ \t]*(>[+-]?)[ \t]*\n', fm, re.MULTILINE)
if header_m:
# Body capture is indentation-based and blank-line-tolerant, per YAML
# block-scalar rules: a blank line (any amount of whitespace) always
# stays inside the block; the indent is set by the first content line;
# the block ends at the first line indented less than that, or EOF.
rest = fm[header_m.end():]
indent = None
body_lines = []
for line in rest.splitlines(keepends=True):
text = line.rstrip('\n')
if text.strip() == '':
body_lines.append(line)
continue
line_indent = len(text) - len(text.lstrip(' \t'))
if indent is None:
indent = line_indent
elif line_indent < indent:
header_m = re.search(r'^description:[ \t]*', fm, re.MULTILINE)
else:
header_m = None
if header_m:
head_start = header_m.start()
value_start = header_m.end()
header_end = fm.find('\n', value_start)
header_end = len(fm) if header_end == -1 else header_end
first = fm[value_start:header_end]
body_start = header_end + 1
indicator = first.rstrip()
block_m = re.fullmatch(r'([|>])([+-]?[0-9]*|[0-9]*[+-]?)', indicator)
if block_m and block_m.group(1) == '|':
kind = None # literal blocks keep their line breaks; vale is fine
elif block_m:
kind = 'block' # folded (`>`): the value starts on the next line
elif indicator == '':
kind = 'block' # bare `description:`: a plain scalar on later lines
elif first[:1] == '"':
kind = 'double'
elif first[:1] == "'":
kind = 'single'
elif first[:1] in '#&*!':
kind = None # comment, anchor, alias or tag — not a plain scalar
else:
kind = 'plain'
text = ''
value_end = value_start
value_lines = 0
if kind in ('block', 'plain'):
body = ''.join(continuation_lines(fm[body_start:]))
value_end = body_start + len(body)
if kind == 'block':
text = body
value_lines = body.count('\n')
else:
text = fm[value_start:value_end]
value_lines = 1 + body.count('\n')
if ' #' in text or text.lstrip().startswith('#'):
# A `#` opens a comment inside a plain scalar. Folding it in
# would lint text YAML never treats as part of the value, so
# leave the file alone rather than lint the wrong string.
kind = None
elif kind in ('double', 'single'):
quote = '"' if kind == 'double' else "'"
inner_start = value_start + 1
acc = fm[inner_start:body_start]
idx = close_quote(acc, quote)
lines = continuation_lines(fm[body_start:])
while idx is None:
try:
acc += next(lines)
except StopIteration:
break
body_lines.append(line)
raw = ''.join(body_lines)
if raw.count('\n') >= 2:
flat = re.sub(r'\s+', ' ', raw).strip()
# YAML single-quoted scalars have no backslash-escape mechanism at
# all, so wrapping in single quotes sidesteps the backslash-escape
# bug entirely for embedded double quotes, backslashes, and
# non-ASCII text. The one YAML-spec-correct way to embed a literal
# apostrophe is to double it ('') — but Vale's own frontmatter
# scanner isn't a full YAML parser and doesn't understand that
# doubling: empirically, it silently truncates the value at the
# first ' it sees, hiding everything after it from the NLP scope
# (a different flavor of the same bug this whole script exists to
# work around). Since this copy is scratch-only and never written
# back, sidestep it by substituting a Unicode right single
# quotation mark (U+2019) for any literal apostrophe instead of
# doubling it — visually a smart quote, but never triggers a YAML
# escape sequence at all.
flat_q = "'" + flat.replace("'", "’") + "'"
pad = '\n' * raw.count('\n')
start = header_m.start()
end = header_m.end() + len(raw)
new_fm = fm[:start] + f'description: {flat_q}\n{pad}' + fm[end:]
content = fm_match.group(1) + new_fm + fm_match.group(3) + content[fm_match.end():]
idx = close_quote(acc, quote)
if idx is None:
kind = None # unterminated quote: invalid YAML, leave it to vale
else:
inner = acc[:idx]
value_end = inner_start + idx + 1
text = unescape_double(inner) if quote == '"' else inner.replace("''", "'")
value_lines = 1 + inner.count('\n')
flat = re.sub(r'\s+', ' ', text).strip()
if kind and flat and value_lines >= 2:
# `value_end` can land mid-line, just past a closing quote, so extend to
# the end of that physical line and carry whatever follows (a trailing
# comment) across unchanged.
if value_end > 0 and fm[value_end - 1] == '\n':
span_end = value_end
trailer = ''
else:
newline = fm.find('\n', value_end)
span_end = len(fm) if newline == -1 else newline + 1
trailer = fm[value_end:span_end].rstrip('\n')
# One line replaces the span, so the blank-line pad is one short of the
# newline count it displaced — every later line number is unchanged.
pad = '\n' * (fm[head_start:span_end].count('\n') - 1)
new_fm = (fm[:head_start] + 'description: ' + emit(flat) + trailer
+ '\n' + pad + fm[span_end:])
content = (fm_match.group(1) + new_fm + fm_match.group(3)
+ content[fm_match.end():])
with open(dest, 'w', encoding='utf-8', errors='surrogateescape') as fh:
fh.write(content)
@@ -206,11 +437,16 @@ for arg in ${path_args[@]+"${path_args[@]}"}; do
# the tree, so any file dropped here would be silently unlinted — and then
# every markdown file in the copy is flattened in place. `.git` is pruned:
# vale never lints it and copying it can dwarf the rest of the tree.
# `find -L` follows symlinks because vale does: it lints both a symlinked
# file and a file under a symlinked directory, and a bare `-type f` walk
# would report "0 files" where bare vale reports one. (A symlink loop makes
# `find` warn on stderr and carry on, which is also what vale does.) The
# second walk needs no `-L`: the mirror is all real files by construction.
mkdir -p "$dest"
while IFS= read -r -d '' rel; do
mkdir -p "$dest/$(dirname "$rel")"
cp "$arg/$rel" "$dest/$rel"
done < <(cd "$arg" && find . -name .git -prune -o -type f -print0)
done < <(cd "$arg" && find -L . -name .git -prune -o -type f -print0)
while IFS= read -r -d '' md; do
flatten "$md" "$md"
done < <(find "$dest" -type f -name '*.md' -print0)

416
plugins/kyberforge/skills/skill-audit/scripts/vale-wrap.sh
View File

@@ -2,27 +2,45 @@
set -euo pipefail
# Works around a Vale limitation: the `text.frontmatter.description` NLP scope
# silently stops matching once the `description:` value is a YAML block scalar
# (`>`/`|`) spanning 2+ physical lines — the style used by most skills/agents in
# this repo. Flattens the description to one physical line in a scratch copy
# (padding with blank lines so every other line number is unchanged), then runs
# the real `vale` binary against the copies. Drop-in replacement for calling
# `vale` directly: same args, same exit code.
# silently stops matching once the `description:` value spans 2+ physical lines
# in any form YAML joins back into one string — a `>`/`>-`/`>+` folded block
# scalar (the style used by most skills/agents in this repo), a plain scalar
# wrapped onto continuation lines, or a double- or single-quoted scalar wrapped
# the same way. A `|`/`|-`/`|+` literal block scalar is NOT affected: its parsed
# value keeps exactly the line breaks the source has, and vale matches it fine
# (verified against vale 3.15.2), so literal blocks are deliberately left alone.
# This script flattens an affected description to one physical line in a scratch
# copy (padding with blank lines so every other line number is unchanged), then
# runs the real `vale` binary against the copies. Drop-in replacement for calling
# `vale` directly: same args, same exit code, bar the two documented divergences
# below.
#
# "Same args" means relative paths — `--config` values and path arguments alike
# — resolve against the caller's current directory, exactly as bare `vale`
# resolves them. (An earlier version resolved them against the repo root, an
# invented convention that hard-errored on `--config ../../.vale.ini` from a
# subdirectory and, worse, silently dropped file arguments that didn't happen to
# resolve from the repo root — skipping the flattening this script exists for.)
# "Same args" means relative paths — path arguments and the values of the
# path-valued flags (`--config`, `--output`, `--path`) alike — resolve against
# the caller's current directory, exactly as bare `vale` resolves them. The flag
# values are rewritten to absolute form because the run ends up `cd`'d into the
# scratch mirror, where a relative one would no longer resolve. (An earlier
# version resolved path arguments against the repo root, an invented convention
# that hard-errored on `--config ../../.vale.ini` from a subdirectory and, worse,
# silently dropped file arguments that didn't happen to resolve from the repo
# root — skipping the flattening this script exists for.)
#
# The one addition to bare `vale`'s argument handling: with no `--config` at
# all, this script's own sibling `assets/vale/.vale.ini` is used instead of
# vale's upward search. pre-commit prefixes only `entry[0]` with the hook-repo
# clone path, so a `--config` in `.pre-commit-hooks.yaml` would resolve against
# the *consuming* repo and hard-fail (E100) for every external consumer. The
# manifest therefore passes the script alone, and an explicit `--config` from
# any other caller still wins.
# Divergence 1: with no `--config` at all, this script's own sibling
# `assets/vale/.vale.ini` is used instead of vale's upward search. pre-commit
# prefixes only `entry[0]` with the hook-repo clone path, so a `--config` in
# `.pre-commit-hooks.yaml` would resolve against the *consuming* repo and
# hard-fail (E100) for every external consumer. The manifest therefore passes the
# script alone, and an explicit `--config` from any other caller still wins.
#
# Divergence 2: a path-shaped argument that does not exist is a hard error
# (exit 2). Bare vale drops it, falls back to reading stdin, and prints
# `0 errors ... in stdin` with exit 0 — a typo'd target is then indistinguishable
# from a clean run. Both audit skills treat a `0 files` report as NOT RUN rather
# than clean, and `in stdin` does not match that guard, so the silent form would
# read as "prefilter clean" and skip the LLM fallback. Erroring is the only way
# to keep that guard honest. Linting prose piped on stdin is therefore
# unsupported here — it already was, since the no-path handoff closes stdin so
# vale can't block on a pipe that will never carry content.
#
# Vale prints each path exactly as it was handed to it, so the scratch tree
# mirrors the caller's absolute cwd: a relative path argument is passed through
@@ -41,22 +59,44 @@ cwd="$(pwd -P)"
# later edit breaking that invariant, not a live fix.
vale_args=()
path_args=()
config_next=false
pending_flag=""
config_given=false
for arg in "$@"; do
if [[ "$config_next" == true ]]; then
config_next=false
if [[ "$arg" == /* ]]; then
vale_args+=("$arg")
else
vale_args+=("$cwd/$arg")
fi
if [[ -n "$pending_flag" ]]; then
# Value of a separated two-argv flag. It is never a lint target, however
# file-like it looks. The run ends up `cd`'d into the scratch mirror, so a
# value naming a file has to be absolutized here or it stops resolving.
case "$pending_flag" in
--config)
# Always a path, and required to exist.
if [[ "$arg" == /* ]]; then
vale_args+=("$arg")
else
vale_args+=("$cwd/$arg")
fi
;;
--output|--path)
# `--output` is either a built-in style name (`line`, `JSON`) or a
# template file; only the file form needs resolving. `--path` is
# likewise a path. Anything that names nothing is passed through and
# left for vale to interpret.
if [[ "$arg" != /* && -e "$arg" ]]; then
vale_args+=("$cwd/$arg")
else
vale_args+=("$arg")
fi
;;
*)
vale_args+=("$arg")
;;
esac
pending_flag=""
continue
fi
case "$arg" in
--config)
vale_args+=("$arg")
config_next=true
pending_flag="$arg"
config_given=true
continue
;;
@@ -70,24 +110,60 @@ for arg in "$@"; do
config_given=true
continue
;;
# Same cwd-relative resolution for the `--flag=value` spelling of the two
# other path-valued flags.
--output=*|--path=*)
flag_val="${arg#*=}"
if [[ "$flag_val" != /* && -n "$flag_val" && -e "$flag_val" ]]; then
vale_args+=("${arg%%=*}=$cwd/$flag_val")
else
vale_args+=("$arg")
fi
continue
;;
# Vale's remaining value-taking flags, per `vale --help` (3.x). In the
# separated two-argv form the value must not be classified as a lint target
# — `--output tmpl.tmpl` names a real template file, and treating it as
# input both lints the template and reorders argv so vale sees
# `--output --no-wrap`. The `--flag=value` form needs no entry here: it
# starts with `-` and falls through to vale untouched. A value flag added by
# some future vale release is simply absent from this list and lands back on
# today's behaviour, so this list going stale is never worse than not having
# it.
--ext|--filter|--glob|--minAlertLevel|--output|--path)
vale_args+=("$arg")
pending_flag="$arg"
continue
;;
# Vale's subcommands are bare words that name no file, so they would trip
# the not-found error below. A lint target literally named `sync` (no
# extension, no slash) is misread as the subcommand — accepted, because the
# alternative is failing every `vale-wrap.sh ls-config`.
ls-config|ls-dirs|ls-metrics|ls-vars|sync)
vale_args+=("$arg")
continue
;;
esac
# `-f`/`-d` resolve relative paths against the caller's cwd, same as vale does.
# A path that doesn't exist is left for vale to report on, exactly as bare
# vale would.
if [[ "$arg" != -* && ( -f "$arg" || -d "$arg" ) ]]; then
# An absolute path inside the caller's cwd is relativized so the report cites
# a path that resolves against the real tree. Left absolute, it would be
# rewritten to its scratch copy and printed as `/tmp/tmp.XXXX/...` — a real
# path to a file that is deleted on exit, which reads as a bug in any report
# quoting it. Absolute paths outside the cwd have no relative form and keep
# the scratch-path behaviour documented above.
if [[ "$arg" == "$cwd"/* ]]; then
path_args+=("${arg#"$cwd"/}")
else
path_args+=("$arg")
fi
else
if [[ "$arg" == -* ]]; then
vale_args+=("$arg")
continue
fi
# Everything left is a lint target: `vale [options] [input...]` has no third
# kind of argument. See divergence 2 above for why a missing one is fatal here.
if [[ ! -e "$arg" ]]; then
echo "vale-wrap.sh: no such file or directory: $arg" >&2
exit 2
fi
# An absolute path inside the caller's cwd is relativized so the report cites
# a path that resolves against the real tree. Left absolute, it would be
# rewritten to its scratch copy and printed as `/tmp/tmp.XXXX/...` — a real
# path to a file that is deleted on exit, which reads as a bug in any report
# quoting it. Absolute paths outside the cwd have no relative form and keep
# the scratch-path behaviour documented above.
if [[ "$arg" == "$cwd"/* ]]; then
path_args+=("${arg#"$cwd"/}")
else
path_args+=("$arg")
fi
done
@@ -119,55 +195,210 @@ src, dest = sys.argv[1], sys.argv[2]
with open(src, encoding='utf-8', errors='surrogateescape') as fh:
content = fh.read()
# YAML 1.2 double-quoted escapes (spec 5.7 / 7.3.1). `\<newline>` is handled
# separately in unescape_double because it also swallows the next indentation.
DQ_ESCAPES = {
'0': '\0', 'a': '\a', 'b': '\b', 't': '\t', '\t': '\t', 'n': '\n',
'v': '\v', 'f': '\f', 'r': '\r', 'e': '\x1b', ' ': ' ', '"': '"',
'/': '/', '\\': '\\', 'N': '\x85', '_': '\xa0', 'L': '\u2028',
'P': '\u2029',
}
# First characters that make a plain (unquoted) scalar mean something other than
# text: YAML's c-indicator set.
PLAIN_UNSAFE_FIRST = '-?:,[]{}#&*!|>\'"%@`'
def unescape_double(text):
"""Decode a double-quoted YAML scalar's body to the string YAML parses."""
out = []
i = 0
while i < len(text):
char = text[i]
if char != '\\':
out.append(char)
i += 1
continue
i += 1
if i >= len(text):
break
esc = text[i]
if esc == '\n':
i += 1
while i < len(text) and text[i] in ' \t':
i += 1
continue
if esc in 'xuU':
width = {'x': 2, 'u': 4, 'U': 8}[esc]
digits = text[i + 1:i + 1 + width]
if len(digits) == width:
try:
out.append(chr(int(digits, 16)))
except ValueError:
pass
else:
i += 1 + width
continue
out.append(DQ_ESCAPES.get(esc, esc))
i += 1
return ''.join(out)
def close_quote(text, quote):
"""Index of the closing `quote` in `text`, which starts just past the
opening one. None while the scalar is still unterminated."""
i = 0
while i < len(text):
char = text[i]
if quote == '"' and char == '\\':
i += 2
continue
if char == quote:
if quote == "'" and text[i + 1:i + 2] == "'":
i += 2
continue
return i
i += 1
return None
def continuation_lines(rest):
"""Yield the physical lines of `rest` that continue the value started on the
`description:` line. Indentation-based and blank-line-tolerant, per YAML:
a blank line (any amount of whitespace) always stays inside; the indent is
set by the first content line; the value ends at the first line indented
less than that, at any line flush with the key (that is the next mapping
key, not a continuation), or at EOF."""
indent = None
for line in rest.splitlines(keepends=True):
text = line.rstrip('\n')
if text.strip() == '':
yield line
continue
line_indent = len(text) - len(text.lstrip(' \t'))
if line_indent == 0:
return
if indent is None:
indent = line_indent
elif line_indent < indent:
return
yield line
def emit(value):
"""Render `value` as a one-line YAML scalar whose source text spells the
value out verbatim. Vale locates the description by matching the parsed
value back against the source, so a scalar carrying any escape — `''` in a
single-quoted scalar, `\\"` or `\\\\` in a double-quoted one — makes the
whole `text.frontmatter.description` scope vanish, the same failure this
script exists to work around. Verbatim forms only, therefore, tried in
descending order of fidelity."""
if (value
and value[0] not in PLAIN_UNSAFE_FIRST
and ': ' not in value
and not value.endswith(':')
and ' #' not in value):
return value # plain: nothing needs escaping at all
if "'" not in value:
return "'" + value + "'" # single-quoted: only `'` would escape
if '"' not in value and '\\' not in value:
return '"' + value + '"' # double-quoted: only `"`/`\` would
# Last resort: the value needs quoting AND holds an apostrophe AND a double
# quote or backslash, so no verbatim YAML scalar can carry it. Substituting
# U+2019 for the apostrophe keeps the scope alive, at the cost of any style
# rule whose token contains a literal ASCII apostrophe. Scratch-copy only —
# never written back to the real file.
return "'" + value.replace("'", '’') + "'"
fm_match = re.match(r'^(---\n)(.*?\n)(---\n)', content, re.DOTALL)
if fm_match:
fm = fm_match.group(2)
# Only `>`/`>-`/`>+` (folded) scalars break Vale's frontmatter-description
# scope. `|`/`|-`/`|+` (literal) scalars already work fine with bare vale,
# so they're deliberately left unmatched here.
header_m = re.search(r'^description:[ \t]*(>[+-]?)[ \t]*\n', fm, re.MULTILINE)
if header_m:
# Body capture is indentation-based and blank-line-tolerant, per YAML
# block-scalar rules: a blank line (any amount of whitespace) always
# stays inside the block; the indent is set by the first content line;
# the block ends at the first line indented less than that, or EOF.
rest = fm[header_m.end():]
indent = None
body_lines = []
for line in rest.splitlines(keepends=True):
text = line.rstrip('\n')
if text.strip() == '':
body_lines.append(line)
continue
line_indent = len(text) - len(text.lstrip(' \t'))
if indent is None:
indent = line_indent
elif line_indent < indent:
header_m = re.search(r'^description:[ \t]*', fm, re.MULTILINE)
else:
header_m = None
if header_m:
head_start = header_m.start()
value_start = header_m.end()
header_end = fm.find('\n', value_start)
header_end = len(fm) if header_end == -1 else header_end
first = fm[value_start:header_end]
body_start = header_end + 1
indicator = first.rstrip()
block_m = re.fullmatch(r'([|>])([+-]?[0-9]*|[0-9]*[+-]?)', indicator)
if block_m and block_m.group(1) == '|':
kind = None # literal blocks keep their line breaks; vale is fine
elif block_m:
kind = 'block' # folded (`>`): the value starts on the next line
elif indicator == '':
kind = 'block' # bare `description:`: a plain scalar on later lines
elif first[:1] == '"':
kind = 'double'
elif first[:1] == "'":
kind = 'single'
elif first[:1] in '#&*!':
kind = None # comment, anchor, alias or tag — not a plain scalar
else:
kind = 'plain'
text = ''
value_end = value_start
value_lines = 0
if kind in ('block', 'plain'):
body = ''.join(continuation_lines(fm[body_start:]))
value_end = body_start + len(body)
if kind == 'block':
text = body
value_lines = body.count('\n')
else:
text = fm[value_start:value_end]
value_lines = 1 + body.count('\n')
if ' #' in text or text.lstrip().startswith('#'):
# A `#` opens a comment inside a plain scalar. Folding it in
# would lint text YAML never treats as part of the value, so
# leave the file alone rather than lint the wrong string.
kind = None
elif kind in ('double', 'single'):
quote = '"' if kind == 'double' else "'"
inner_start = value_start + 1
acc = fm[inner_start:body_start]
idx = close_quote(acc, quote)
lines = continuation_lines(fm[body_start:])
while idx is None:
try:
acc += next(lines)
except StopIteration:
break
body_lines.append(line)
raw = ''.join(body_lines)
if raw.count('\n') >= 2:
flat = re.sub(r'\s+', ' ', raw).strip()
# YAML single-quoted scalars have no backslash-escape mechanism at
# all, so wrapping in single quotes sidesteps the backslash-escape
# bug entirely for embedded double quotes, backslashes, and
# non-ASCII text. The one YAML-spec-correct way to embed a literal
# apostrophe is to double it ('') — but Vale's own frontmatter
# scanner isn't a full YAML parser and doesn't understand that
# doubling: empirically, it silently truncates the value at the
# first ' it sees, hiding everything after it from the NLP scope
# (a different flavor of the same bug this whole script exists to
# work around). Since this copy is scratch-only and never written
# back, sidestep it by substituting a Unicode right single
# quotation mark (U+2019) for any literal apostrophe instead of
# doubling it — visually a smart quote, but never triggers a YAML
# escape sequence at all.
flat_q = "'" + flat.replace("'", "’") + "'"
pad = '\n' * raw.count('\n')
start = header_m.start()
end = header_m.end() + len(raw)
new_fm = fm[:start] + f'description: {flat_q}\n{pad}' + fm[end:]
content = fm_match.group(1) + new_fm + fm_match.group(3) + content[fm_match.end():]
idx = close_quote(acc, quote)
if idx is None:
kind = None # unterminated quote: invalid YAML, leave it to vale
else:
inner = acc[:idx]
value_end = inner_start + idx + 1
text = unescape_double(inner) if quote == '"' else inner.replace("''", "'")
value_lines = 1 + inner.count('\n')
flat = re.sub(r'\s+', ' ', text).strip()
if kind and flat and value_lines >= 2:
# `value_end` can land mid-line, just past a closing quote, so extend to
# the end of that physical line and carry whatever follows (a trailing
# comment) across unchanged.
if value_end > 0 and fm[value_end - 1] == '\n':
span_end = value_end
trailer = ''
else:
newline = fm.find('\n', value_end)
span_end = len(fm) if newline == -1 else newline + 1
trailer = fm[value_end:span_end].rstrip('\n')
# One line replaces the span, so the blank-line pad is one short of the
# newline count it displaced — every later line number is unchanged.
pad = '\n' * (fm[head_start:span_end].count('\n') - 1)
new_fm = (fm[:head_start] + 'description: ' + emit(flat) + trailer
+ '\n' + pad + fm[span_end:])
content = (fm_match.group(1) + new_fm + fm_match.group(3)
+ content[fm_match.end():])
with open(dest, 'w', encoding='utf-8', errors='surrogateescape') as fh:
fh.write(content)
@@ -206,11 +437,16 @@ for arg in ${path_args[@]+"${path_args[@]}"}; do
# the tree, so any file dropped here would be silently unlinted — and then
# every markdown file in the copy is flattened in place. `.git` is pruned:
# vale never lints it and copying it can dwarf the rest of the tree.
# `find -L` follows symlinks because vale does: it lints both a symlinked
# file and a file under a symlinked directory, and a bare `-type f` walk
# would report "0 files" where bare vale reports one. (A symlink loop makes
# `find` warn on stderr and carry on, which is also what vale does.) The
# second walk needs no `-L`: the mirror is all real files by construction.
mkdir -p "$dest"
while IFS= read -r -d '' rel; do
mkdir -p "$dest/$(dirname "$rel")"
cp "$arg/$rel" "$dest/$rel"
done < <(cd "$arg" && find . -name .git -prune -o -type f -print0)
done < <(cd "$arg" && find -L . -name .git -prune -o -type f -print0)
while IFS= read -r -d '' md; do
flatten "$md" "$md"
done < <(find "$dest" -type f -name '*.md' -print0)