feat: implement governance instruction layer Phase 1

This commit is contained in:
2026-05-14 18:52:12 +00:00
parent ab367a48c4
commit c0ede4b22e
20 changed files with 2408 additions and 3 deletions

View File

@@ -0,0 +1,106 @@
# Deterministic Controls
Applies to: any environment, repository, or pipeline where AI tools are used.
Full governance context: `docs/ai-constitution.md` — principles these controls enforce.
Human practitioner rules: `docs/HUMANS.md` | Agent instructions: `core/instructions/governance.md`
This file specifies the enforcement layer: controls that run mechanically, regardless of human or agent intention.
**Why this file exists:** Agent instructions and human practitioner rules are probabilistic — they depend on attention and intent. This layer removes that dependency. A control that runs automatically in CI enforces a principle more reliably than any instruction in any file. Where a principle can be enforced deterministically, it must be.
---
## Day-One: Environment Setup
Controls configured once per development environment. Any machine or environment used for AI-assisted work must have these in place before work begins.
---
**Secret scanning in pre-commit**
A pre-commit hook that detects secrets, credentials, API keys, and high-entropy strings must be active in every development environment. It must run before any commit reaches version control — not as a best-effort scan, but as a blocking gate.
*Enforces: Constitution §2 — secrets never enter AI context or version control.*
**AI tool data tier verification**
AI tools used for Internal, Confidential, or Restricted data must be configured to use enterprise-tier endpoints. Verify contractual data-not-trained commitments are in place before connecting any non-Public data source to an AI tool. This is a one-time verification per tool, repeated when tools or plans change.
*Enforces: Constitution §3 — consumer and free-tier products handle Public data only.*
**Governance instruction file present and adapter files configured**
Every repository or project context in active use must have an agent governance instruction file present and accessible (in this repo: `core/instructions/governance.md`, deployed globally via `@import`), with tool-specific adapter files (CLAUDE.md, copilot-instructions.md, etc.) referencing it. Verify this is in place before starting AI-assisted work in any new repo.
*Enforces: Constitution §1 — governance rules must reach the agents operating in context.*
---
## Per-Repo: Repository Controls
Controls configured for each repository. Apply these when creating a new repo or when adding AI-assisted workflows to an existing one.
---
**Secret scanning in CI**
Every repository CI pipeline must include a secret scanning step that fails the build on detected credentials, tokens, or high-entropy strings. Pre-commit hooks can be bypassed; CI cannot. Both layers are required.
*Enforces: Constitution §2 — architectural constraint, not a reminder.*
**Dependency and security scanning**
Every repository CI pipeline must include dependency vulnerability scanning covering known CVEs and supply chain risks. For repositories using AI-generated code, the scan must be configured to cover AI-assisted contributions — not just declared dependencies.
*Enforces: Constitution §2 — AI-generated code is untrusted by default; OWASP LLM supply chain risks.*
**Licence scanning**
Every repository CI pipeline must include a licence scanning step that detects copyleft-licensed fragments (GPL, AGPL, LGPL) in committed code. Manifest-based scanners alone are insufficient for AI-assisted contributions — the scan must cover code content, not just declared dependencies.
*Enforces: Constitution §8 — copyleft fragments can appear in AI output without headers.*
**AI agent permission scoping**
Any AI agent granted access to this repository must be configured with the minimum permissions required for its specific task. Broad-scope tokens granting read/write access to the full repository or infrastructure are prohibited for AI agents. Token scope must be documented and reviewed when the agent's task scope changes.
*Enforces: Constitution §2 — least-privilege for all AI agents.*
**Prompt version control**
Any prompt used in an automated or recurring AI pipeline — not ad-hoc sessions — must be committed to version control with a change history. Prompts not under version control are not auditable. A prompt that runs in production without version control is uncontrolled code.
*Enforces: Constitution §7 — prompts are code; unversioned prompts are unauditable.*
**Audit logging for agentic workflows**
Any agentic workflow that modifies state — files, infrastructure, configuration, deployments — must produce a log capturing: prompt input (or reference to versioned prompt), model version, action taken, outcome, timestamp. The log must be tamper-evident and human-readable. Isolated timestamps without action context are not sufficient.
*Enforces: Constitution §7 — every agent action producing an effect must generate a trace.*
**Human approval gate for production changes**
Any CI/CD pipeline that applies changes to production systems, security configuration, or infrastructure must include an explicit human approval step before the change is applied. Automated merge-and-deploy pipelines for AI-generated changes are prohibited without this gate. The gate must be implemented in the pipeline, not left to individual judgment.
*Enforces: Constitution §5 — production requires a human checkpoint; this is a hard rule.*
---
## Ongoing: Verification and Review
Controls that must be verified periodically. These cannot be configured once and forgotten — they degrade, drift, and become stale. Assign a cadence and an owner.
---
**Pre-commit hook integrity** *(per developer, monthly)*
Verify pre-commit hooks are installed, active, and current in every active development environment. Hooks can be bypassed, uninstalled by tooling updates, or silently disabled. A hook that is not tested is not a control.
**CI scan results review** *(per repository, per release or sprint)*
Review secret, licence, and dependency scan outputs — not just pass/fail status. A scan that passes because exceptions have accumulated is not a clean scan. Review exception lists and remove expired or unjustified exceptions.
**AI agent permission audit** *(per repository, quarterly)*
Verify that AI agent tokens and permissions remain scoped to current task requirements. Agent permissions granted for a specific task tend to persist after the task ends. Revoke and re-scope on a defined cadence.
**Audit log review** *(per agentic workflow, per sprint or monthly)*
Review AI agent action logs for unexpected scope, anomalous patterns, or actions that should have triggered a human approval gate but did not. Logging without review is record-keeping, not oversight.
**AI deployment value review** *(per deployment, time-bounded)*
Every AI integration must be reviewed against the success criteria defined before deployment. Integrations that have not delivered measurable value within the defined review period must be redesigned or discontinued. Schedule this review at deployment time, not retrospectively.
**Provider terms and data handling review** *(annually, or when providers update terms)*
Verify that AI provider terms of service, data handling commitments, and IP provisions remain consistent with what was agreed at onboarding. Provider terms change. An enterprise commitment made in 2024 may not have the same scope in 2026. Re-verify; do not assume continuity.
**Constitution and controls alignment review** *(annually, or after any significant AI incident)*
Verify that the controls specified here remain aligned with the current version of the AI Constitution. When the constitution is updated, this file must be reviewed and updated to match. A control specification that drifts from the constitution it enforces is not a control.
---
## What This File Does Not Govern
The specific tooling used to implement each control is the implementer's choice — tool selection is out of scope here. What is in scope is the requirement: what the control must detect, gate, or produce. Select tools that meet the requirement; replace them when better options exist without needing to update this file.
Human judgment decisions — which AI model to use, whether a specific output is acceptable, how to classify ambiguous data — are governed by `docs/HUMANS.md`. Agent judgment decisions are governed by `core/instructions/governance.md`. This file governs only what can be enforced without judgment.
---
*Derived from AI Constitution v1.1 — May 2026.*
*Counterpart to: `docs/HUMANS.md` | `core/instructions/governance.md` | Full context: `docs/ai-constitution.md`*

View File

@@ -0,0 +1,200 @@
# Agent Instructions — Research & Design Notes
**Purpose:** Documents the research behind `AGENTS.md` and the design decisions made in producing it. Provides implementation guidance for deploying agent instructions across tools. Human reference — does not go into agent context.
> **Repo note:** `AGENTS.md` was integrated into this repo as `core/instructions/governance.md` and is loaded globally via `@import` in `providers/claude-code/CLAUDE.md`. References to `AGENTS.md` throughout this document are the generic concept (any agent instruction file following this pattern) and the historical name — not a path in this repo.
---
## Why a Separate Agent Instructions File?
The AI Constitution (`ai-constitution.md`) is a governance document for humans and agents — comprehensive, reasoned, cross-referenced. It is too long for global agent context: every token in a system prompt costs tokens on every inference, and research confirms long instruction files are largely ignored.
The agent instructions file (`AGENTS.md`) is the operative distillation: only principles an agent can act on in the moment, stripped of rationale, short enough to load without meaningful context cost.
---
## Research Findings on Effective Agent Instructions
### Instruction following: the compliance ceiling
AGENTIF benchmark (Tsinghua University, 2025), evaluating 707 instructions across 50 real-world agentic applications (average 1,717 tokens, ~11.9 constraints per instruction): **the best model perfectly follows fewer than 30% of instructions**. This is the current ceiling for complex multi-constraint agentic instruction following. Source: keg.cs.tsinghua.edu.cn/persons/xubin/papers/AgentIF.pdf.
A broader evaluation of 256 models across 20 diagnostic tests found an overall pass rate of 43.7% on instruction following, with performance ranging from 0% to 100% and a standard deviation of 28.4 percentage points. Provider methodologies significantly impact adherence beyond model size. Source: huggingface.co/richardyoung/llm-instruction-following-paper.
**Implication:** AGENTS.md improves compliance; it does not guarantee it. The constitution's principle — "deterministic enforcement must sit outside the AI" — is the primary enforcement mechanism. Agent instructions are a supplementary signal.
### Instruction length: the 500-line limit
Practitioners analysing 2,500+ repositories: "If your config file is over 500 lines, most of it is being ignored. LLMs have limited instruction-following capacity — a focused 50-line file outperforms a sprawling 1,000-line one." Source: deployhq.com/blog/ai-coding-config-files-guide.
Research on hierarchical prompting (HIPO, 2025): processing lengthy prompts increases latency and computational cost as self-attention scales quadratically with sequence length. Source: arxiv.org/pdf/2603.16152.
**Design decision:** AGENTS.md targets under 80 lines. Every line must earn its place.
### Priority ordering: hard constraints first
HIPO research: hierarchical prompting with priority-ordered directives is the standard for reliable agentic instruction following. The system prompt defines global behavioral guidelines and safety boundaries; higher-priority instructions must appear first. Source: arxiv.org/pdf/2603.16152.
**Design decision:** AGENTS.md opens with Hard Prohibitions (the Never rules), then Required Behaviours. Prohibitions come first because they are unconditional and must not be overridden by task context.
### Hard prohibitions vs. behavioural guidance
Contrary to the intuition that unconditional prohibitions are the most reliable instruction type, Semantic Gravity Wells (2026) found that ~87.5% of negative-constraint failures are *priming failures* — stating the forbidden token ("Never hardcode a password") activates the concept and increases the likelihood of exactly that behaviour in models trained on next-token prediction. Condition-based constraints account for ~42.6% of real-world applications and are the most commonly failed instruction type. Source: AGENTIF benchmark; Semantic Gravity Wells (2026).
**Design decision:** Every "Never X" in AGENTS.md is paired with "— instead do Y." This converts a negative constraint into a positive directive, avoiding the priming failure mode while retaining the unconditional framing. The Required Behaviours section improves compliance without guaranteeing it — they guide probabilistic behaviour in the right direction.
### Including examples
Research confirms agents learn implicitly from examples even within a single prompt. Infobip documentation: "Providing sample outputs helps the agent understand the expected output format. You can show both correct and incorrect examples to make behaviour consistent." Source: infobip.com/docs/ai-agents/advanced-topics/write-prompts.
**Design decision:** AGENTS.md v1.1 pairs every prohibition with a positive alternative (the primary example pattern). Concrete negative examples (e.g., showing a hardcoded credential) remain a v1.2 candidate if compliance testing reveals persistent violations of specific rules.
### What linters handle: exclude from agent instructions
"Do not include things a linter handles — these are Prettier's job, not your AI config file's job." Source: deployhq.com.
**Design decision:** Code formatting, import ordering, naming conventions, and other statically enforceable rules are excluded. These are enforced by CI gates, not by agent instructions.
---
## Design Decisions in AGENTS.md
### Included: agent-actionable principles
These are things an agent can act on in the moment:
- Secrets and credentials handling (detect, refuse, redirect)
- Data classification before sharing (apply the four-tier framework)
- Production action gates (stop and confirm for irreversible/high-stakes)
- Code security review before committing
- Honesty over agreement (re-evaluate vs. capitulate; flag uncertainty)
- Deterministic code preference for repeatable tasks
- Agentic transparency (state intent, prefer reversible, log actions)
- Prompt/model hygiene (version prompts, right-size models)
### Excluded: human process decisions
These cannot be operationalised by agent instructions — they are enforced by humans, tooling, or organisational process:
- HITL checkpoint requirements (the human decides when to apply these)
- Post-mortem requirements (human process)
- IP licence scanning (tooling: Black Duck, FOSSA, etc.)
- Regulatory notification (human/legal process)
- Sustainability measurement (organisational)
- Audit log retention periods (infrastructure/policy)
- EU AI Act compliance verification (legal/compliance function)
- **Deskilling / skill atrophy monitoring** (Constitution §5): detecting whether AI-assisted roles are losing baseline capability is a human governance obligation — an agent cannot assess its own contribution to skill atrophy in the human it assists. Governance mechanism design is out of agent scope.
- **Diverse perspectives prompting** (Constitution §4): this is a human prompting standard — the obligation is on the person constructing the prompt to ask for multiple viewpoints on contested questions. An agent could volunteer diverse perspectives proactively, but this conflicts with following instructions faithfully and is too context-dependent to encode as a blanket agent rule. Left as a human prompting principle.
- **Deterministic enforcement controls** (Constitution Governance section): pre-commit hooks, CI gates, scanner configuration, audit logging infrastructure, AI agent permission scoping, and environment verification are specified in `CONTROLS.md` and implemented by humans. These cannot be operationalised by agent instructions — they run mechanically regardless of agent or human intent.
### The data classification table
Included as a quick-reference decision rule — the most token-efficient addition to the file. Agents can apply the four tiers instantly without loading the constitution.
---
## Deployment: Single Source of Truth + Thin Adapters
AGENTS.md is the single source of truth. Do not copy-paste its content into tool-specific files. Instead, create thin adapter files that reference it.
### Global vs per-repo deployment
This is the first architectural decision to make. It shapes every path reference below.
**Per-repo (recommended starting point):** AGENTS.md lives at the root of each repository. Tool adapters (CLAUDE.md, copilot-instructions.md, etc.) sit alongside it and reference it by relative path. Simple, no cross-tool path issues, version-controlled with the codebase it governs. When you update AGENTS.md for one repo, only that repo is affected — deliberate isolation.
**Global (team or personal baseline):** AGENTS.md lives in a central location, and a global tool config references it. For Claude Code the global CLAUDE.md is `~/.claude/CLAUDE.md`; for Copilot there is no equivalent global file. The benefit: one AGENTS.md update applies everywhere. The cost: cross-tool path handling becomes complex (see below), and per-repo customisation requires an additional local override layer.
**Recommended pattern:** Start per-repo. Once you see the same AGENTS.md content appearing in multiple repos unchanged, extract a global baseline and use the import/reference pattern described below. Do not over-engineer the global layer before you know what belongs in it — this is the same principle that governs skills library design.
### Path handling
When AGENTS.md and the adapter file are in the same directory (per-repo pattern), relative paths work for all tools. When they are not in the same directory (global pattern), each tool handles paths differently:
**Claude Code** supports `@import` syntax and symlinks:
```markdown
# CLAUDE.md — simplest per-repo adapter
@AGENTS.md
```
This imports AGENTS.md directly at session start. No path issue if both files are in the repo root. For a global setup with AGENTS.md in `~/.claude/`, place CLAUDE.md alongside it and use the same `@AGENTS.md` relative reference — Claude Code loads `~/.claude/CLAUDE.md` for every project automatically.
On Unix, a symlink eliminates duplication entirely:
```bash
ln -s AGENTS.md CLAUDE.md
```
On Windows, use the `@AGENTS.md` import instead (symlinks require Developer Mode).
**Copilot and Cursor** reference AGENTS.md by path string in prose — not a live import. If AGENTS.md is not in the repo root, update the prose reference to the actual path, or keep a copy in the repo. There is no global Copilot instructions file; `.github/copilot-instructions.md` is per-repo only.
### GitHub Copilot
Create `.github/copilot-instructions.md`:
```markdown
Follow the governance rules in `AGENTS.md` at the root of this repository at all times.
```
Since July 2025, Copilot also supports scoped instructions via `.github/instructions/*.instructions.md` with glob-pattern frontmatter for file-type specific rules. Source: code.visualstudio.com/docs/copilot/customization/custom-instructions.
### Claude Code
Create `CLAUDE.md` (or use the `@import` / symlink approach above):
```markdown
@AGENTS.md
```
Or, if you want to add Claude-specific rules on top:
```markdown
@AGENTS.md
## Claude-specific additions
For full governance context, see `ai-constitution.md`.
```
### Cursor
Create `.cursor/rules/governance.mdc`:
```markdown
---
alwaysApply: true
---
Follow the governance rules in `AGENTS.md` at the root of this repository at all times.
```
### Gemini CLI
Create `GEMINI.md`:
```markdown
Follow the governance rules in `AGENTS.md` at the root of this repository at all times.
```
### Why this pattern
Source: deployhq.com — "Maintain one source of truth (AGENTS.md) and have tool-specific files reference it. Don't copy-paste the same rules into CLAUDE.md, .cursorrules, and copilot-instructions.md." When AGENTS.md is updated, all tools pick up the change without manual synchronisation.
Claude Code's `@import` syntax (confirmed 2026) means the thin adapter for Claude Code can be a single line. Source: blink.new/blog/agents-md-vs-claude-md.
---
## Known Limitations
1. **Compliance ceiling is ~30% on complex multi-constraint tasks.** Unconditional single-directive rules (with positive alternative framing) have substantially higher compliance than conditional behavioural guidance. But "Never X — instead do Y" is not a guarantee of compliance; it is a best practice for reducing priming failures. Semantic Gravity Wells (2026) found negative constraints without positive alternatives fail ~87.5% of the time via priming. Design accordingly.
2. **Prompt injection can bypass agent instructions.** Malicious content in repository data (PR bodies, issue descriptions, code comments) can override system-level instructions in some models. This is why HITL gates for production actions are in the constitution as non-negotiable process controls, not in AGENTS.md as agent instructions alone.
3. **Instruction drift.** Agents operating over long contexts may lose earlier instructions. Keep AGENTS.md short. For agentic workflows with many steps, consider injecting a reminder of the most critical Never rules at task boundaries.
4. **Tool-specific parsing differences.** Not all tools process AGENTS.md identically. Validate compliance behaviour per tool when onboarding a new AI assistant.
---
## When to Update AGENTS.md
- When the AI Constitution is updated (AGENTS.md must stay consistent with it)
- When a new AI tool is onboarded and its parsing behaviour differs materially
- When a hard prohibition is regularly violated in practice (may need reformulation or a concrete example added)
- When a new high-risk pattern is identified that agents can detect and refuse in context
- At minimum: annually, aligned with the constitution review cycle
Do not add principles that belong in the constitution (reasoned governance) or that belong in tooling (linters, scanners, CI gates). AGENTS.md covers only what agents can act on in the moment.
---
*Created: May 2026. Companion to AGENTS.md v1.0 and AI Constitution v1.0.*

View File

@@ -0,0 +1,175 @@
# AI Governance — Open Questions & Research Challenges
**Purpose:** Agenda for the deep research session. Each item identifies a hypothesis, assumption, or finding from the initial research that warrants deeper investigation, challenge, or quantification. Ordered by governance impact — highest-stakes uncertainties first.
---
## 1. Sycophancy as root cause of hallucination — how strong is the causal link?
**Current position:** Sycophancy is the most tractable cause of hallucination. RLHF systematically induces approval-seeking behaviour that overrides accuracy.
**What to challenge:**
- The 59% regressive sycophancy rate and 78.5% persistence rate come from single studies (Fanous et al., SycEval, AAAI 2025). Do independent replications confirm these magnitudes?
- Is sycophancy genuinely a *cause* of hallucination, or do both share a common cause (RLHF misalignment) without a direct causal relationship?
- What is the relative contribution of sycophancy vs. training data gaps vs. decoding strategies to observed hallucination rates in software development contexts specifically?
- Are there prompting or configuration approaches with replicated evidence — not single-study findings — that reliably reduce sycophancy in production?
**Governance implication:** If sycophancy is one cause among several of roughly equal weight, the constitution's emphasis on designing against it first may be correctly placed but insufficiently supported.
---
## 2. AI as net sustainability negative — is this actually measurable?
**Current position:** 72% of AI investments destroy value through waste; AI is currently a net negative because ungoverned and not optimised for genuine value.
**What to challenge:**
- The 72% figure is from a single proprietary Larridin report — not peer-reviewed. What do independent measurements show?
- The METR study (19% slower) covers experienced open-source developers using early-2025 tools on open-source tasks. Does this generalise to professional software teams using current tools on commercial work?
- What rigorous measurement frameworks exist for AI ROI that are not vendor-published? DORA is the most credible — what does the full 2025 DORA dataset show beyond the summary?
- What would "net positive" look like in measurable terms? Is there a validated metric set (beyond token efficiency) that constitutes evidence of genuine value delivery?
**Governance implication:** The J-Curve principle and the time-bounded review requirement depend on being able to measure value. If measurement frameworks are genuinely inadequate, the principle may need to be restructured.
---
## 3. Human code ownership under AI assistance — where is the line?
**Current position:** Humans must be able to understand, audit, and manually override AI-generated code. AI assistance that creates comprehension dependency is an ethical failure.
**What to challenge:**
- The deskilling literature is largely from non-software contexts (manufacturing, aviation). Is there specific evidence of deskilling in software engineering from AI assistance?
- Is comprehension of every line the right standard, or is there a meaningful distinction between understanding architecture/intent vs. understanding every implementation detail?
- METR's finding (19% slower) may reflect tool immaturity rather than a structural principle. What does the evidence show as tools mature?
- What is the minimum viable human understanding of AI-generated code that is sufficient for responsible ownership?
**Governance implication:** If the comprehension standard is set too high, it may prohibit beneficial automation. If too low, it allows accountability laundering.
---
## 4. EU AI Act applicability to software developers — what actually applies?
**Current position:** EU AI Act obligations are treated as broadly applicable from August 2026.
**What to challenge:**
- Most AI-assisted software development does not involve "high-risk AI systems" as defined in Annex III of the Act. What risk tier does typical software/infra tooling (Copilot, Claude Code, Cursor) actually fall into?
- The Act primarily regulates providers and deployers of AI systems, not necessarily users of AI coding tools. What specific obligations fall on a developer using an AI coding assistant vs. deploying an AI-powered product?
- How does the research activity exemption (Article 2(6)) affect developers using AI in pre-production contexts?
- What does "significant modification" of a GPAI model mean in practice — does fine-tuning a model for your codebase trigger provider-level obligations?
**Governance implication:** If current tooling falls into minimal-risk categories, some constitution principles may be disproportionate for the actual regulatory requirement. Alternatively, obligations may be more specific and actionable than currently stated.
---
## 5. Data classification in AI context — are four tiers sufficient?
**Current position:** Four tiers (Public / Internal / Confidential / Restricted) mirroring ISO 27001, with AI-specific handling rules per tier.
**What to challenge:**
- The four-tier model was designed for file and database classification, not for AI context windows. Are there AI-specific classification schemes from the EDPB, CNIL, or ISO that go further?
- How should AI-generated content itself be classified? (A document produced by AI from Confidential inputs — what tier is the output?)
- How does the composite sensitivity problem (non-sensitive data combining to reveal sensitive attributes) get operationalised in a classification framework? Is the four-tier model capable of handling this?
- What open-source tooling exists for context window scanning and real-time classification? The constitution references "safe open-source options" but does not specify them.
**Governance implication:** A classification framework that cannot be operationalised with available tooling is ethics washing.
---
## 6. Open source licence contamination — how severe is the practical risk?
**Current position:** Licence contamination is a documented, growing risk; 17% of open source components enter codebases via AI generation invisible to standard scanning tools.
**What to challenge:**
- The 2024 ruling that dismissed claims found AI-generated code is not "substantially similar" to training code. Is the contamination risk overstated — is it more theoretical than practical for most generated output?
- What is the actual rate of verbatim or near-verbatim GPL/AGPL reproduction in AI coding assistant output? Are there studies with methodology that can be evaluated?
- What are the current open-source scanning tools that are effective for AI-generated code specifically, and what are their false positive/negative rates?
- Is the risk primarily in snippets and algorithms, or does it extend to overall code structure and design patterns?
**Governance implication:** Overstating this risk imposes unnecessary cost; understating it creates legal exposure. The constitution needs a calibrated, evidenced position.
---
## 7. Incident response for operational safety — what playbooks actually exist?
**Current position:** Operational safety failures (unintended autonomous actions, goal misalignment) are the highest-risk category and the least prepared for. Extend existing IR frameworks.
**What to challenge:**
- The Coalition for Secure AI AI Incident Response Framework (2026) is referenced but not evaluated in detail. What does it actually recommend for operational safety incidents specifically?
- What is the NIST AI Risk Management Framework (AI RMF 1.0) response framework — how does it handle incidents caused by agentic systems?
- Are there documented case studies of organisations that have successfully handled AI operational safety incidents? What made their response effective?
- For software/infra contexts specifically: what triggers should initiate an AI incident response, and at what point does an AI misbehaviour become a notifiable incident?
**Governance implication:** A principle that says "extend existing IR frameworks" without specifying what to add is incomplete.
---
## 8. Transparency disclosure requirements — what is the minimum viable implementation?
**Current position:** AI involvement must be disclosed; this is a legal obligation from August 2026 under the EU AI Act.
**What to challenge:**
- The EU AI Act transparency provisions in Articles 13/50 apply to specific system types (chatbots, deepfake generators, high-risk systems). Do they apply to internal AI-assisted development tooling where end users are not interacting with AI directly?
- What does "disclosure" actually require in the context of AI-assisted code review, automated deployment, or AI-generated documentation?
- Are there jurisdiction-specific requirements beyond the EU AI Act (UK, US state laws) that are more or less demanding?
- What is the current state of implementation: who is actually compliant with existing transparency requirements, and what enforcement actions have occurred?
**Governance implication:** A disclosure requirement that is vague about what to disclose, to whom, and in what form cannot be implemented.
---
## 9. Model selection criteria — what does sycophancy resistance actually look like?
**Current position:** Select models partly on sycophancy resistance; some models (Anthropic, large Llama) show more resistance.
**What to challenge:**
- What benchmarks exist specifically for sycophancy resistance, and how methodologically rigorous are they? (SycEval from AAAI 2025 is referenced — evaluate it.)
- Are sycophancy benchmarks stable across prompt variations, or do models that perform well on benchmarks still exhibit sycophancy in production contexts?
- How does model selection for sycophancy resistance trade off against other factors (cost, capability, data protection, provider terms)?
- Is the Anthropic/Llama advantage in sycophancy resistance confirmed by independent evaluations, or is it based primarily on Anthropic's own research?
**Governance implication:** If sycophancy benchmarks are not reliable, "select models on sycophancy resistance" is not actionable.
---
## Deep Research Session Objectives
1. Replicate or refute the key quantitative claims from the initial research against additional independent sources.
2. Identify specific, actionable tooling and implementation guidance for each principle — particularly data classification scanning, licence scanning, audit logging, and sycophancy evaluation.
3. Resolve the EU AI Act applicability question for the specific context of software development tooling.
4. Find and evaluate the Coalition for Secure AI IR Framework and NIST AI RMF in detail.
5. Assess whether the constitution contains any principles that are unsupported, disproportionate, or operationally unachievable given current tooling.
6. Identify any significant governance topics the initial research missed.
---
*Prepared after initial research session, May 2026. Upload alongside `ai-governance-research-session.md` to continue.*
---
## 10. Deterministic execution preference — where exactly is the decision boundary?
**Current position:** Prefer deterministic code over repeated AI inference for well-specified, repeatable tasks. Use the "compile once, execute many" pattern.
**What to challenge:**
- The 57× token reduction claim comes from the xy.ai "Compiled AI" paper, a startup with commercial interest in this pattern. Do independent studies confirm similar efficiency gains?
- What is the practical definition of "well-specified enough" to codify as a script? Is there a decision framework with clear criteria that has been validated in practice?
- The 2.5–68× execution time overhead for AI-generated code — is this a property of AI-generated code in general, or specifically of unreviewed/unoptimised AI output? Does human review close this gap?
- How does this principle interact with skills/workflows in the repo? A SKILL.md that instructs an AI agent to perform a task is itself a form of "codification" — is this deterministic enough, or does it still require a traditional script?
- Are there task categories in software dev and infra where the AI-inference-at-execution-time approach is clearly superior even for repeated tasks (e.g., code review comments, PR descriptions)?
**Governance implication:** The decision boundary between "use a script" and "use AI inference" needs to be specific enough to be actionable. A vague principle is not useful.
---
## 11. Agent instruction reliability — how much can AGENTS.md actually enforce?
**Current position:** AGENTS.md operationalises the constitution's agent-actionable principles. Hard prohibitions (Never rules) are the most reliably followed; required behaviours are aspirational and improve compliance without guaranteeing it.
**What to challenge:**
- AGENTIF benchmarking found the best model perfectly follows fewer than 30% of complex agentic instructions. Is this the right benchmark for our use case (simpler, more direct instructions vs. complex multi-constraint agentic tasks)?
- The 500-line limit finding comes from practitioner observation, not controlled research. Is there empirical evidence on the optimal length and structure for instruction files?
- Which specific instruction types in AGENTS.md are most likely to be violated? Are the hard prohibitions actually hard for current models, or does prompt injection still bypass them?
- How should AGENTS.md be tested and validated? What would a compliance eval for these instructions look like?
- The thin adapter pattern (AGENTS.md as single source, tool-specific files reference it) — does this actually work across Copilot, Claude Code, and Cursor, or do tool-specific constraints require substantive adaptation rather than thin references?
- Should AGENTS.md include few-shot examples (what to do vs. what not to do) to improve compliance? Research suggests agents learn implicitly from examples even within a single prompt.
**Governance implication:** If agent instruction compliance is fundamentally limited, the constitution's principle that "deterministic enforcement must sit outside the AI" is not just a best practice — it is the primary enforcement mechanism, and AGENTS.md is a supplementary signal at best. This changes how we think about operationalising the other principles.

View File

@@ -0,0 +1,309 @@
# AI Governance Research — Session Audit Trail
**Purpose:** Auditability of the artifact creation process. Documents what was done, how, why, and what decisions were made or deferred. Not a task list — a process record.
**Project:** AI governance research and artifact creation for a software development, deployment, and infrastructure management context.
**Sessions:** Three sessions, May 2026.
**Artifacts produced:** See artifact registry below.
---
## Project Overview
The goal was to produce a governance foundation before building an AI-assisted software development repository. The author wanted principles grounded in research — not opinion — that would govern all future AI implementation and deployment.
The work was deliberately sequenced: research first, then distil into operative governance. The research must be independently challengeable; the governance must be applicable at both homelab and enterprise scale.
---
## Methodology
### Research approach
- The instruction throughout was to use reliable web sources only, not training data. Whether this was fully achieved cannot be independently verified — some findings may reflect training data rather than sourced research, particularly in analytical or synthesis sections where citations are sparse.
- Research is conducted topic by topic so findings can be challenged before they harden into principles.
- All hypotheses established during scoping are provisional. Evidence drives conclusions; if research contradicts a prior stance, the constitution reflects the evidence, not the hypothesis.
- Bias flags were intended to reference limitations within the sourced research itself (vendor-funded studies, single-study findings, Western regulatory framing, etc.) — not the AI assistant's training data. However, the line between "synthesising sourced findings" and "drawing on training data" is not always clear, and some analytical conclusions may have training data influence that went undetected.
### Output format per topic
Each topic in the research document follows: question being researched → findings → counterarguments and challenges → bias flag → provisional principles.
### Distillation logic
Research document = full sourced reasoning (human reference, never in agent context).
Constitution = concise principles derived from research (agent-readable, repo artifact).
AGENTS.md = agent-actionable subset of the constitution, optimised for context window efficiency.
### Standing integrity caveat
The instruction to use web sources only was given and followed to the best of the session's ability. However, AI-generated research cannot self-audit for training data influence with certainty. Analytical framing, the choice of which findings to emphasise, and the synthesis of sources into conclusions all carry some risk of undetected training data influence. The deep research session in Session 2 provided one independent cross-check. Where citations are absent or thin, treat findings with additional scepticism and verify independently before relying on them for high-stakes decisions.
Any claim found to be unsourced during the session is corrected with a new web search before proceeding. Any claim refuted by subsequent research is retracted and corrected in the relevant artifacts.
---
## Scope Decisions
These decisions were made during an initial scoping/interview session before research began. Each shaped all subsequent work.
| Decision | Conclusion | Rationale |
|---|---|---|
| Output form | Two artifacts: research doc + constitution | Separates reasoning from operative governance; keeps constitution context-efficient |
| Audience | Solo now, team-inheritable later | Every design choice must be legible without the author present |
| Provider / infra scope | Agnostic across tools and stacks | Will be used by others; cannot be tool-locked |
| Research depth | Thorough, sourced, with tradeoffs | Principles are only as strong as the evidence behind them |
| Constitution depth | Concise and opinionated | If it can't be read and acted on quickly, it won't be used |
| Format | Markdown for all artifacts | Portable, versionable, agent-readable, renders in GitHub |
| Data classification | Explicit rules in constitution AND tooling | Tooling enforces but does not replace governance |
| Automation philosophy | Automate as much as responsibly possible; humans own the code | Automation is the goal; comprehension is the constraint |
| Research sequencing | Topic by topic, user challenges before proceeding | Prevents hypotheses hardening before they are tested |
---
## Provisional Hypotheses (established pre-research)
The following were the author's starting positions before research began. They were tested, not assumed.
| Hypothesis | Outcome |
|---|---|
| Sycophancy is the root cause of hallucination | **Partially confirmed.** Sycophancy is a major and tractable cause (14.66% regressive, 78.5% persistence under pressure). Not the sole cause — training data gaps and decoding strategies are independent contributors. |
| AI is currently a net sustainability negative | **Confirmed in societal terms.** Reframed from business ROI to societal/global: environmental cost (415 TWh, 105M+ tonnes CO₂e), power concentration, epistemic harm at scale, deskilling, value extraction without consent. |
| Human code ownership is non-negotiable | **Confirmed, standard refined.** Line-by-line comprehension is not the right bar. The ACM/IEEE-CS standard is: intent-level + architectural + verifiable behaviour understanding. |
| Data classification belongs in governance, not just tooling | **Confirmed.** EDPB Opinion 28/2024 and CNIL guidance support explicit governance-layer classification rules independent of tooling. |
---
## Session 1 — Research and Initial Artifact Creation
### What was done
1. **Scoping session (grill-me):** Established purpose, audience, methodology, and the scope decisions documented above. Three rounds of questions, answered one by one. Output: shared understanding of what to build and why.
2. **Topic-by-topic research:** Ten governance topics researched in sequence. Each topic was web-searched, synthesised, and reviewed before proceeding to the next. User challenged findings at each step.
3. **Mid-session methodology correction:** During Topics 1–3, bias flags incorrectly referenced "my training data" as a source. The user caught this. Corrected: bias flags now reference limitations within the sourced research itself. Affected sections in Topics 1, 2, and 3 were re-researched and corrected.
4. **Topic 10 addition:** After Topics 1–9 were complete and the constitution was drafted, the user requested an additional topic: preferring deterministic code execution over repeated AI inference. Topic 10 was researched and added to all artifacts.
5. **Artifact distillation:** Research document → constitution → AGENTS.md → agent instructions notes. Each layer is a deliberate reduction: research has full sourcing, constitution has principles, AGENTS.md has only what an agent can act on in the moment.
6. **Artifact audit:** All artifacts reviewed for internal consistency, topic count (nine → ten), cross-references, and missing information. Corrections made:
- Methodology note in session doc corrected (training data reference)
- Topic count updated throughout
- AGENTS.md cross-referenced from constitution and handoff
- Agent instructions notes created to document design decisions behind AGENTS.md
### Topics researched
| # | Topic | Key finding |
|---|---|---|
| 1 | Ethics | Accountability non-transferable; ethics washing documented; sycophancy is an ethical failure mode not just a quality one |
| 2 | Security | OWASP LLM Top 10 + Agentic Top 10 as baseline; secrets leakage 40% higher in AI-assisted repos; prompt injection actively exploited in CI/CD |
| 3 | Data protection & classification | GDPR + EU AI Act apply concurrently; context window is a data store; enterprise vs consumer tier is a hard distinction |
| 4 | Sustainability | ⚠️ Requires re-research — see note below | 32.6–79.7M tonnes CO₂ estimate for AI in 2025; token efficiency as sustainability metric; J-Curve of value realisation. Societal/global framing (environmental externality, power concentration, epistemic harm, deskilling, value extraction) introduced in Session 2 but not yet independently researched. |
| 5 | Behaviour & sycophancy | 14.66% regressive sycophancy; 78.5% persistence; confident language inversely correlated with accuracy |
| 6 | Human oversight | EDPS four conditions for genuine oversight; automation bias in 35 peer-reviewed studies; responsibility vacuum at scale |
| 7 | Transparency | EU AI Act logging requirements; agentic audit gap; prompt versioning as governance infrastructure |
| 8 | Intellectual property | No copyright without human authorship; licence contamination risk; dependency-hallucination as separate risk channel |
| 9 | Incident response | Deployment failure, not model failure, is the primary cause; CoSAI IR Framework v1.0; notification timelines |
| 10 | Deterministic execution | Break-even at ~17 invocations; EffiBench execution overhead; PAL/CodeAct as established patterns |
---
## Session 2 — Deep Research and Corrections
### What was done
1. **Deep research session launched:** An extended search task was used to independently challenge all 11 documented research challenges (10 topics + AGENTS.md reliability). The task searched for independent peer-reviewed sources to verify, refute, or nuance prior findings.
2. **Deep research findings reviewed:** Of 11 challenges: 2 confirmed, 6 partially confirmed, 3 refuted. Key corrections identified (see below).
3. **Scope clarification — sustainability reframe:** The user clarified that the "net sustainability negative" framing should be societal and global — not just business ROI. This reframe was agreed in principle but the societal dimensions (environmental externality, power concentration, epistemic harm at scale, deskilling, value extraction) were not independently researched in this session. They represent a direction for the next research pass, not verified findings.
4. **Applicability requirement established:** All artifacts must be applicable at homelab scale and enterprise scale. Nothing should require a compliance department, a committee, or enterprise tooling to act on.
5. **Next-session handoff created:** Rather than executing all corrections immediately in a session that was running long, a focused handoff document was created with three clear instructions for a fresh session.
6. **Partial constitution rewrite abandoned:** An attempt was made to execute corrections to the constitution during this session. This was cut off and the partial changes were reverted.
7. **Constitution restored to v1.0:** The constitution was reverted to the original research-based version. The backup was then deleted.
8. **Cleanup:** An over-detailed adjustment plan document created mid-session was deleted. All references to it and to the backup file were removed from other artifacts.
### Deep research verdict summary
| Challenge | Verdict | Key correction |
|---|---|---|
| 1. Sycophancy as root cause | Partially confirmed | Correct figure: 14.66% regressive (not 59%); venue AIES 2025 not AAAI; sycophancy is one of several RLHF failure modes, not the sole cause |
| 2. AI as net sustainability negative | Refuted as stated (business framing) | Business ROI evidence is weak/vendor-sourced; societal framing is the correct and better-evidenced argument |
| 3. Human code ownership | Refuted as stated | Line-by-line standard has no professional authority; ACM/IEEE-CS standard is intent + architecture + verifiable behaviour |
| 4. EU AI Act applicability | Refuted as stated | August 2026 deadline is wrong (Omnibus May 2026 moved dates); coding assistants are minimal/limited risk; only Article 4 applies to developers using tools |
| 5. Data classification sufficiency | Partially confirmed | Four tiers necessary but insufficient; add lifecycle stage, AI-risk class, data quality dimensions (ISO 42001/23894/5259) |
| 6. OSS licence contamination | Partially confirmed | "17%" figure has no primary source — drop it; Ciniselli et al.: 0.1–10% verbatim clones; Doe v. GitHub on interlocutory appeal, not settled |
| 7. Incident response playbooks | Partially confirmed | CoSAI v1.0 exists (Nov 2025); OWASP Agentic Top 10 (Dec 2025); notification timelines confirmed from EU AI Act Article 73 |
| 8. Transparency disclosure scope | Refuted as stated | Article 50 is provider-side, not deployer-side for coding assistant users; internal tooling has no Article 50 obligation |
| 9. Model selection for sycophancy | Refuted as stated | Anthropic/Llama advantage is not confirmed by independent benchmarks; rankings flip across evaluations |
| 10. Deterministic execution boundary | Partially confirmed | "57×" is a single vendor preprint; replace with break-even ~17 invocations; EffiBench gives exact overhead figures |
| 11. AGENTS.md reliability | Partially confirmed | "Hard prohibitions most reliable" is refuted; Semantic Gravity Wells (2026): negative constraints fail via priming ~87.5% of the time; pair every "Never X" with "— instead do Y" |
---
## Key Decisions Log
| Decision | Why |
|---|---|
| Web research only — no training data | Training data is an uncontrolled source with unknown biases. Web research from named sources is auditable and challengeable. |
| Two artifacts (research + constitution), not one | A combined document would be too long for agent context. Separation also clarifies purpose: research is "why", constitution is "what to do". |
| AGENTS.md as a third artifact | The constitution is still too long for global agent context. A further distillation to agent-actionable-only reduces context cost without losing governance depth. |
| Single source of truth + thin adapters | Copying AGENTS.md content into CLAUDE.md, copilot-instructions.md etc. creates maintenance debt. One source, referenced by tool-specific files. |
| Topic-by-topic with user challenge | Prevents a batch of unchallenged principles from hardening into governance. User review at each step is the quality gate. |
| Topic 10 (deterministic execution) added after initial draft | The principle — use AI to write scripts, not to execute repeated tasks — is a meaningful governance decision that wasn't captured in the original nine topics. Worth its own section. |
| Deep research before applying corrections | Corrections based on a single research pass might introduce new errors. An independent challenge pass reduces that risk. |
| Constitution NOT updated in Session 2 | The session was interrupted mid-execution. A partial update is worse than no update. Deferred to a clean next session with a clear handoff. |
| Societal/global framing for sustainability | Business ROI evidence is vendor-sourced and weak. The societal argument — environmental externality, power concentration, epistemic harm, deskilling — is stronger, more independent, and more principled. |
| Applicability at homelab and enterprise | The author works across both scales. A principle that only applies at enterprise scale is not a principle for this context — it is aspirational. |
---
## Corrections and Integrity Notes
| What was wrong | When caught | How corrected |
|---|---|---|
| Bias flags referenced "my training data" as a source | Session 1, during Topic 1–3 review | User caught it. Methodology instruction clarified: bias flags should reference limitations within the sourced research, not training data. Affected topics re-searched and rewritten. Whether training data influence was fully eliminated across all topics remains uncertain — the deep research pass in Session 2 was one independent check, but cannot guarantee it. |
| Topics 1–3 had unsourced claims in analysis sections | Session 1 | Targeted web searches filled identifiable gaps. Unsourced sections were replaced with cited findings where the gap was visible. Unsourced influence that was not visibly identifiable may remain. |
| SycEval "59% regressive sycophancy, AAAI" | Session 2 deep research | Correct: 14.66% regressive, 78.5% persistence, AIES 2025. Not yet applied to artifacts — deferred to next session. |
| Larridin "72% destroying value" cited as evidence | Session 2 deep research | Identified as vendor-commissioned survey. Dropped. Not yet removed from artifacts — deferred. |
| EU AI Act "August 2026" deadline | Session 2 deep research | Omnibus May 2026 moved dates. Correct dates established. Not yet applied — deferred. |
| "Hard prohibitions are most reliable" in AGENTS.md | Session 2 deep research | Refuted by Semantic Gravity Wells (2026). Prohibitions must be paired with positive replacements. Not yet applied — deferred. |
| Anthropic/Llama sycophancy advantage | Session 2 deep research | Refuted by MASK benchmark and others. Not yet applied — deferred. |
| "17% OSS contamination" figure | Session 2 deep research | No primary source found. Drop and replace. Not yet applied — deferred. |
| "57× token reduction" | Session 2 deep research | Single vendor preprint. Replace with break-even ~17 invocations. Not yet applied — deferred. |
| Doe v. GitHub "settled" | Session 2 deep research | On interlocutory appeal; not settled. Not yet applied — deferred. |
| Constitution v1.1 partial rewrite | Session 2 | Abandoned mid-execution; reverted to v1.0. Clean corrections deferred to next session. |
---
## Artifact Registry
Current state of all artifacts as of end of Session 3.
| File | Version | Purpose | Status |
|---|---|---|---|
| `ai-constitution.md` | 1.1 | Governance principles derived from research. Repo artifact, agent-readable. | Complete. All instructions applied. |
| `AGENTS.md` | 1.1 | Operative agent instructions. Concise, agent-actionable distillation of the constitution. | Complete. |
| `ai-governance-research.md` | 1.1 | Full research document. Sourced findings, counterarguments, bias flags, provisional principles for all 10 topics. | Complete. Preamble, table, cross-refs, Topic 4 rewrite applied. |
| `ai-agent-instructions-notes.md` | 1.0 | Design rationale for AGENTS.md. Instruction following research, deployment pattern, update policy. | Priming failure finding (Semantic Gravity Wells) incorporated |
| `ai-governance-research-challenges.md` | 1.0 | 11 deep research challenges with verdicts and recommended adjustments. | Reference only. No changes needed. |
| `ai-governance-research-session.md` | 3.0 | This document. Audit trail of the creation process. | Current |
| `next-session-handoff.md` | 2.0 | Instruction status tracker. Instruction 1 complete; 2 and 3 pending. | Current |
---
## Session 3 — Apply Instruction 1 Corrections
### What was done
Applied all critical factual corrections from the deep research pass (Session 2) across constitution, AGENTS.md, research doc, and agent notes. Then executed Instruction 2 (sustainability research and reframe) and Instruction 3 (operationalisation assessment) in the same session.
**Instruction 1 changes applied per artifact:**
*ai-constitution.md (→ v1.1)*
- EU AI Act dates corrected throughout (Omnibus May 2026): Art. 50 → Dec 2, 2026; Art. 4 (AI literacy) live since Feb 2025; scope clarified to coding assistant users = minimal/limited risk.
- Sycophancy model selection: removed unconfirmed Anthropic/Llama advantage; replaced with portfolio benchmark approach (MASK, SYCON-Bench, SycEval) and run-your-own-test guidance.
- Human code ownership: replaced line-by-line comprehension standard with ACM/IEEE-CS standard (intent + architecture + verifiable behaviour).
- Deterministic execution: removed unsourced 2.5–68× slower claim; replaced with EffiBench reference and correct framing.
- DPIA: homelab exemption note added then removed (Instruction 3 decision — scale-agnostic treatment).
*AGENTS.md (→ v1.1)*
- Every "Never X" rule paired with "— instead do Y" throughout, per Semantic Gravity Wells (2026) priming failure finding.
*ai-governance-research.md*
- SycEval: 59% → 14.66% regressive, AAAI → AIES 2025.
- Larridin 72% dropped; replaced with S&P Global and MIT NANDA figures.
- METR 19% slower updated with Feb 2026 correction.
- EU AI Act dates corrected in both instances.
- 17% OSS contamination dropped; replaced with Ciniselli et al. 0.1–10%.
- 57× token reduction flagged as single vendor preprint; break-even ~17 invocations elevated.
- Anthropic/Llama sycophancy advantage removed.
*ai-agent-instructions-notes.md*
- Semantic Gravity Wells finding incorporated; design decision updated.
**Instruction 2 changes:**
*ai-governance-research.md*
- Topic 4 fully rewritten as "Sustainability & Societal Cost" with independent web research (May 2026).
- IEA April 2025/2026 as primary energy source. Inequality paradox (Chen & Meng 2026). Epistemic harm confirmed. Deskilling evidence. Labour effects early and contested.
- Core reframe: AI costs are externalised to non-users; governance is an obligation to those who bear them.
*ai-constitution.md*
- Section 6 rewritten with societal/externality framing. ⚠️ warning removed.
**Instruction 3 changes (grilling session):**
*ai-constitution.md*
- DPIA homelab exemption removed — consistent scale-agnostic treatment across all principles
- Governance section: brief example mapping added (secrets principle → 4-layer enforcement stack)
- Section 9, Section 8, GDPR notification: left as-is (scale-agnostic; tool-agnostic; technically correct)
*ai-governance-research.md (→ v1.1)*
- Preamble rewritten: "how to use" framing distinguishing it from the constitution
- Summary table added: Topic → Core finding → Principle count → Constitution section
- Cross-reference lines added to all 10 topics' provisional principles
*ai-agent-instructions-notes.md (→ v1.1)*
- Deployment section expanded: global vs per-repo trade-offs, path handling, @import syntax, symlink approach, Copilot/Cursor limitation noted
- Claude Code adapter updated to use `@AGENTS.md` import pattern
*AGENTS.md, ai-governance-research-challenges.md:* No changes in Instruction 3
---
## Post-Instruction-3 Gap Analysis — Same Session
After completing all three instructions, a cross-reference check of the constitution and AGENTS.md against the sustainability research findings identified three gaps the instructions hadn't addressed:
**Gap 1 — Deskilling / cognitive dependency (Constitution §5 added)**
Evidence (Kosmyna 2025 neural disengagement, medical domain atrophy, ACM FAccT 2026 "Brainrot" paper) was solid enough to warrant an explicit principle. Added to Section 5: "AI assistance must augment human capability, not replace it — governance must include mechanisms to detect skill atrophy in AI-assisted roles." Added as Principle 7 to Topic 6 provisional principles in research doc.
**Gap 2 — Homogenisation / diverse perspectives (Constitution §4 added)**
AI systems suppress annotator disagreements, producing majority-weighted outputs on contested questions (arxiv 2505.07772). Added to Section 4: prompting principle for explicitly requesting multiple viewpoints and dissenting positions on contested or values-laden questions. Added as Principle 7 to Topic 5 provisional principles in research doc. Topic index table updated (5 and 6 now 7 principles each).
**Gap 3 — AGENTS.md Restricted table row (already resolved)**
The "instead do Y" pointer was already present from an earlier edit. No change needed.
**Agent notes updated:** "Excluded" section now explicitly records why deskilling monitoring and diverse perspectives prompting are in the constitution but not AGENTS.md, with design rationale for each.
---
## Next Session
All instructions complete and gap analysis done. The artifact set is in its final researched, corrected, and operationalised state.
If continuing: consider whether any new research has emerged that warrants updating specific topics, or whether the constitution is ready to be committed to a repository and put into active use.
---
## Post-Gap-Analysis Updates — Same Session
After the post-Instruction-3 gap analysis, CONTROLS.md was created and gaps in AGENTS.md and HUMANS.md were resolved.
**CONTROLS.md created (new artifact)**
Deterministic enforcement layer specifying pre-commit hooks, CI gates, licence scanning, audit logging, AI agent permission scoping, and ongoing verification cadences. Structured by setup phase (day-one / per-repo / ongoing). Tool-agnostic declarative requirements. Covers the enforcement layer that neither agent instructions nor human practitioner rules can provide.
**AGENTS.md updated:**
- Data minimisation in agent context added to Data Classification section: limit scope to what the task requires when accessing files or data
- Token efficiency added to Prompt and model hygiene: use minimum tokens necessary
- "What This File Does Not Govern" updated to reference CONTROLS.md as the deterministic enforcement layer
- Footer updated to reference HUMANS.md and CONTROLS.md
**HUMANS.md updated:**
- AI-generated crypto added to Hard Limits: never use AI-generated passwords/keys/secrets
- Data minimisation added to Before section: send only what the task requires
- Deterministic execution added to During section: for repeatable tasks, generate a script; ~17-invocation break-even noted
- Output volume limit added to During section: manage throughput to what you can genuinely evaluate
- Provider IP terms review added to After section: verify before commercial use
- "What This File Does Not Govern" updated to reference CONTROLS.md
- Footer updated to reference all three operative files
**Constitution footer updated:** now references AGENTS.md, HUMANS.md, and CONTROLS.md as the three operative files.
**Agent notes updated:** CONTROLS.md added as third item in "Excluded" section with design rationale.
**Final artifact set: eight files**
ai-constitution.md | AGENTS.md | HUMANS.md | CONTROLS.md | ai-governance-research.md | ai-agent-instructions-notes.md | ai-governance-research-challenges.md | ai-governance-research-session.md

View File

@@ -0,0 +1,586 @@
# AI Governance Research Document
**Purpose:** Human reference document. The "why" behind every principle in the constitution. Contains sourced findings, counterarguments, bias flags, and provisional principles for all ten governance topics. Does not go into AI agent context — too long.
**How to use this document:** This is the research layer, not the operative layer. If you want to know *what to do*, read `ai-constitution.md`. If you want to know *why a principle exists*, challenge a finding, or update the evidence base, read the relevant topic here. Each topic ends with provisional principles that map directly to a constitution section — cross-references are noted.
**Session:** May 2026
**Methodology:** Topic-by-topic web research from reliable sources. All conclusions are provisional and challengeable. Research must remain unbiased — findings drive principles, not the other way around.
---
## Topic Index
| # | Topic | Core finding | Principles | Constitution |
|---|---|---|---|---|
| 1 | Ethics | Accountability non-transferable; sycophancy is an ethical failure, not just a quality one | 4 | §1, §4 |
| 2 | Security | AI-assisted repos have 40% higher secret leak rates; prompt injection is actively exploited in CI/CD | 6 | §2 |
| 3 | Data Protection | The context window is a data store; consumer/enterprise tier distinction is a hard legal line | 6 | §3 |
| 4 | Sustainability | Costs are externalised to non-users; governance is an obligation to those who bear them | 6 | §6 |
| 5 | Behaviour & Sycophancy | 14.66% regressive sycophancy; 78.5% persistence; confident language inversely correlated with accuracy | 7 | §4 |
| 6 | Human Oversight | Automation bias confirmed in 35 peer-reviewed studies; symbolic oversight is the documented failure mode | 7 | §5 |
| 7 | Transparency | Three of Europe's most impactful 2024 AI enforcement cases were triggered by logging failures, not bias | 6 | §7 |
| 8 | Intellectual Property | No copyright without human authorship; 0.1–10% verbatim clone rate in AI output | 5 | §8 |
| 9 | Incident Response | Deployment failure, not model failure, is the primary cause of AI incidents | 6 | §9 |
| 10 | Deterministic Execution | Break-even with runtime inference at ~17 invocations; non-determinism compounds across agentic chains | 5 | §10 |
---
## Topic 1: Ethics
**The question:** What ethical obligations govern AI use in software development, deployment, and infrastructure management?
### Regulatory and normative baseline
The OECD AI Principles (2019, updated May 2024) are the first intergovernmental standard on AI. The 2024 update strengthened provisions on misinformation/disinformation from generative AI and explicitly added environmental sustainability. Five values: inclusive growth and wellbeing; rule of law and human rights; transparency and explainability; robustness, security, and safety; accountability. Source: OECD AI Principles (oecd.ai/en/ai-principles).
UNESCO Recommendation on the Ethics of AI (2021, 194 member states): AI systems should be auditable and traceable; member states must ensure AI systems do not displace ultimate human responsibility. Source: UNESCO (unesco.org/en/artificial-intelligence/recommendation-ethics).
EU AI Act (in force August 2024, phased obligations per Omnibus May 2026 agreement): Article 4 (AI literacy) applies from February 2025. Article 50 transparency (chatbots, deepfake generators) applies from December 2, 2026. Standalone Annex III high-risk systems: December 2, 2027. Embedded high-risk: August 2, 2028. Developers *using* coding assistants (Copilot, Claude Code, Cursor) face only Article 4 obligations — these tools are minimal/limited risk under the Act, not high-risk. Source: EU digital-strategy.ec.europa.eu.
### Epistemic integrity: sycophancy as an ethical problem
Turner and Eisikovits (Springer, 2026): AI sycophancy is an "artificial vice" generating moral and epistemic harms. Key distinction between UX friction reduction (acceptable) and truth-related friction reduction (sycophancy — not acceptable). Source: link.springer.com/article/10.1007/s43681-026-01007-4.
Empirical harm to decision-making: brief conversations with sycophantic AI increased attitude extremity and certainty while inflating users' self-perceptions; users rated sycophantic responses as higher quality and expressed greater willingness to use them again — creating a perverse incentive where users seek out the systems that distort their reasoning. Source: arxiv.org/pdf/2602.14270.
### Epistemic autonomy: deskilling
Deskilling is well-documented in technological transitions: when humans outsource decisions to technology, skills atrophy. Source: arxiv.org/pdf/2503.22151.
METR study (July 2025): Experienced open-source developers using AI tools took 19% longer than without them. METR's own February 2026 update supersedes this: with current tools, new participants show approximately −4% (marginal slowdown) and METR now states "AI likely provides productivity benefits in early 2026." Cui et al. (Management Science 2026, n=4,867): +26% completed tasks with Copilot. The productivity picture is task- and context-dependent; early negative results reflected tool immaturity rather than a structural effect. Source: metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study; METR February 2026 update.
### Ethics washing: a documented failure mode
Multiple peer-reviewed sources confirm that corporate AI ethics frameworks often function as public relations rather than genuine governance. Rességuier and Rodrigues (2020): AI ethics frameworks have come to stand in for effective regulation, providing few legal or political protections. Meredith Whittaker: "steep cost of corporate capture in AI ethics." Source: journals.sagepub.com/doi/10.1177/20539517231221780.
Nearly 100 non-legally-binding ethical codes or statements adopted in five years, putting forward mostly the same principles without producing measurable change. Source: link.springer.com/article/10.1007/s44206-025-00174-x.
### Counterarguments
Accountability-is-non-transferable is well-evidenced but incomplete: some researchers argue full deployer responsibility when AI failure modes are opaque may be unreasonably demanding — the more important lever may be liability for AI developers to build safer systems.
The ethics washing critique does not mean ethics frameworks are worthless — it means poorly operationalised ones are.
### Provisional principles
*Distilled into: Constitution §1 (Accountability) and §4 (Behaviour & Sycophancy)*
1. Accountability is legally and ethically non-transferable.
2. Sycophancy is an ethical failure mode, not just a performance failure mode; designing against it is an ethical obligation.
3. Ethics commitments must be concrete and auditable — stated values without enforcement mechanisms are ethics washing.
4. Human oversight is a legal requirement for high-risk AI use under the EU AI Act, not a design option.
---
## Topic 2: Security
**The question:** What security risks does AI specifically introduce into software development, deployment, and infrastructure pipelines?
### The structural mismatch with traditional security
Traditional security controls — dependency scanning, SBOM analysis, network controls, endpoint protection — do not address AI-specific attack vectors. Prompt injection cannot be firewalled the same way a port can be. Source: thehackernews.com/2025/12/traditional-security-frameworks-leave.html.
AI security incidents increased 56.4% from 2023 to 2024 (Stanford HAI AI Index). Source: cycode.com/blog/ai-security-vulnerabilities.
### OWASP LLM Top 10 (2025)
LLM01: Prompt Injection | LLM02: Sensitive Information Disclosure (jumped from #6 to #2) | LLM03: Supply Chain | LLM04: Data and Model Poisoning | LLM05: Improper Output Handling | LLM06: Excessive Agency | LLM07: System Prompt Leakage | LLM08: Vector and Embedding Weaknesses | LLM09: Misinformation | LLM10: Unbounded Consumption. Source: trydeepteam.com/docs/frameworks-owasp-top-10-for-llms. A separate Agentic AI Top 10 was released late 2025. Source: bsg.tech/blog/owasp-llm-top-10.
### Prompt injection: actively exploited
Johns Hopkins researchers hijacked Claude Code Security Review, Gemini CLI Action, and Microsoft GitHub Copilot through prompt injection — exfiltrating API keys, GitHub access tokens, and secrets via malicious PR metadata. Source: letsdatascience.com/news/researchers-hijack-ai-coding-agents-steal-credentials-d508cbeb.
### Secrets leakage: severe and worsening
GitGuardian State of Secrets Sprawl 2025: 23.8 million new credentials detected on public GitHub in 2024, 25% year-on-year increase. AI-assisted repos: 40% higher leak rate than baseline. Claude Code commits leaked secrets at double the baseline rate in 2025. 28.65 million hardcoded secrets added to public GitHub in 2025; AI-service leaks surged 81%. Sources: csoonline.com/article/3953927; helpnetsecurity.com/2026/04/14; helpnetsecurity.com/2026/04/15.
MCP configuration files: 24,008 unique secrets found in MCP config files; pattern is pervasive including official GitHub MCP server. Source: wiz.io/blog/leaking-ai-secrets-in-public-code.
### AI-generated passwords: insufficient entropy
Claude's passwords start with uppercase "G" and digit "7"; 50 Claude-generated passwords produced only 30 unique results; measured entropy 27 bits vs 98 bits expected. Source: dev.to/0x711/ai-agents-dont-understand-secrets-thats-your-problem-43n4.
### Supply chain
As few as 50,000 fake articles in a public training dataset corrupted medical LLMs; small quantities of poisoned information corrupted even large models. Attacks carried out against RAG pipelines, MCP tools, and synthetic data workflows in 2025. Source: cycode.com/blog/ai-security-vulnerabilities.
### Uncertainty propagation
Composing AI subsystems with uncertain performance into automated pipelines creates compounding error invisible at the output layer. Source: ncbi.nlm.nih.gov/pmc/articles/PMC12747714.
### Counterarguments
The 40% higher leak rate for Copilot repositories does not prove causation — early AI tool adopters may also be the fastest-moving, least-careful developers. Causation is plausible but not definitively proven.
### Provisional principles
*Distilled into: Constitution §2 (Security)*
1. Secrets must never enter AI context — architectural constraint, not policy.
2. AI-generated code is untrusted by default; review with more scrutiny than human-written code.
3. Least-privilege for all AI agents; no long-lived broad-scope tokens.
4. OWASP LLM Top 10 and Agentic AI Top 10 are baseline requirements.
5. Automated AI pipelines must surface and preserve uncertainty.
6. Never use AI-generated secrets, passwords, or cryptographic material.
---
## Topic 3: Data Protection & Classification
**The question:** What data protection obligations apply when using AI, and what classification framework governs what may enter AI systems?
### Concurrent application: GDPR + EU AI Act
Both apply simultaneously whenever personal data is processed by AI. CNIL confirmed: "Where personal data is used for the development of an AI system, both the GDPR and the AI Act apply." Source: cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr.
### Prompts as data processing
EDPB Opinion 28/2024: LLM operations, training, and use raise wide-ranging data protection concerns; addressed the legal basis of legitimate interest for AI models and consequences of unlawful processing during development on subsequent operation. Source: edpb.europa.eu (Opinion 28/2024).
### Context window as data store
Prompts, chat logs, transcripts, and support workflows are data stores traditional classification programs do not scan. Organisations must define what PII can enter prompts, what must be redacted, and what should never enter training sets. Source: forcepoint.com/blog/insights/pii-data-classification.
### International transfers
US-based LLM providers (OpenAI, Anthropic, Google) process data in US data centres. GDPR prohibits free transfer of EU citizen data outside EEA without adequacy decisions, appropriate safeguards, or derogations. Using an enterprise tier with a DPA does not eliminate this obligation; it structures it. Source: nexos.ai/blog/gdpr-ai.
### Right to erasure incompatibility
LLMs present significant challenges to GDPR Article 17 (right to erasure) because personal data is encoded in model weights, not stored as retrievable records. The decision to include personal data in training is irreversible. Source: mdpi.com/1999-5903/17/4/151.
### Enterprise vs consumer: a hard distinction
Use enterprise versions only; ensure the provider does not use prompts, RAG data, or fine-tuning data for model training. It is unlikely possible to demonstrate a legitimate interest in transferring data to a provider to train the base model. Source: dataprotectionreport.com/2025/01.
### Data minimisation
CNIL: AI systems must be developed with a well-defined objective to frame and limit the personal data used — data minimisation applies to prompts. Source: cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr.
### Classification framework
Four-tier model mirrors ISO 27001 guidance and is the established standard. Public / Internal / Confidential / Restricted. For AI: AI systems shall be classified at the level of the highest-classified data used for training or operation; model weights inherit classification from training data. Sources: pertamapartners.com/insights/ai-data-classification-categorizing-data; dpo-consulting.com/blog/gdpr-data-classification.
### Counterarguments
Legal basis for AI training is genuinely unsettled. EU Digital Omnibus (November 2025) proposed allowing AI training on legitimate interest under Article 6(1)(f) — contested and evolving. The right-to-erasure incompatibility is contested at the margins: some argue encoded weights do not constitute personal data in a legally retrievable sense.
### Provisional principles
*Distilled into: Constitution §3 (Data Protection & Classification)*
1. Sending personal data to an AI system is data processing under GDPR.
2. Consumer and free-tier AI products are incompatible with processing organisational or personal data.
3. The context window is a data store — classify it accordingly (four-tier framework).
4. Restricted-tier data (GDPR Art. 9, credentials, regulated data) must never enter any AI context.
5. Data minimisation applies to AI prompts.
6. Personal data must not enter AI fine-tuning or RAG without GDPR legal basis and completed DPIA.
---
## Topic 4: Sustainability & Societal Cost
**The question:** What are AI's actual environmental and societal costs, who bears them, and what does that imply for governance?
> **Research note:** This section was rewritten in Session 3 based on an independent web research pass (May 2026). Previous versions used a business/ROI lens and contained unverified claims. This version uses a societal/externality framing with sourced findings.
### The core governance insight: costs are externalised, benefits are captured
AI's primary costs — environmental, epistemic, and distributional — fall predominantly on people who are not the users or deployers of AI systems: communities near data centres, populations affected by grid stress, workers displaced faster than they can upskill, and societies that absorb the epistemic and democratic effects of large-scale AI-generated content. The governance obligation flows from this asymmetry: those who benefit from AI use have an obligation to the people who bear its costs, whether or not those costs are legally required to be accounted for.
### Environmental cost
**Electricity consumption (IEA, authoritative):** Global data centre electricity consumption was approximately 485 TWh in 2025, up 17% from ~415 TWh in 2024. AI-focused data centres grew faster — 50% in 2025 alone. The IEA's base-case projection: data centres roughly double to ~950 TWh by 2030, accounting for ~3% of global electricity demand; AI-focused data centres triple over the same period. Source: IEA Key Questions on Energy and AI (April 2026); IEA Energy and AI (April 2025). iea.org/reports/energy-and-ai.
**Carbon and water footprint (peer-reviewed):** AI systems are estimated to generate 32.6–79.7 million tonnes CO₂ equivalent in 2025; water footprint 312.5–764.6 billion litres. The upper bound rivals New York City's annual emissions. The IEA projects data centres will reach ~1% of global CO₂ emissions by 2030 in its central scenario — one of the few sectors where emissions are projected to grow rather than decarbonise. Source: de Vries-Gao (ScienceDirect, December 2025); sciencedirect.com/science/article/pii/S2666389925002788.
**Geographic concentration creates localised stress:** AI infrastructure is >90% concentrated in North America, Western Europe, and Asia-Pacific. Specific regions (Oregon, Virginia, Ireland) face Power Stress Index values indicating local grid vulnerability. Six leading firms account for compute capacity equivalent to ~1% of global power demand by 2030. Source: arxiv.org/pdf/2604.06198.
**Corporate disclosure is structurally inadequate:** Environmental reports from data centre operators do not distinguish AI from non-AI workloads. Independent verification of AI-specific footprint is not currently possible without regulatory disclosure mandates. Microsoft emissions grew 23.4% since 2020, citing AI expansion. Source: de Vries-Gao (ScienceDirect 2025); brookings.edu/articles/global-energy-demands-within-the-ai-regulatory-landscape.
**Efficiency vs. scale:** Per-task energy use is declining rapidly — IEA notes efficiency improvements at a rate unprecedented in energy history. However, scale of adoption outpaces efficiency gains, and energy-intensive use cases (AI agents, multimodal generation) are growing fastest. Source: IEA Key Questions on Energy and AI (April 2026).
### Power and wealth concentration
Big Tech combined AI capex exceeded $400 billion in 2025, projected to rise ~75% in 2026. AWS, Azure, and Google Cloud hold ~63% of cloud infrastructure market share. US private AI investment ($109.1 billion in 2024) was 11.7× China's investment. Source: Stanford HAI AI Index 2025; quantumrun.com/consulting/ai-market-share-by-company-statistics.
**The inequality paradox (Chen & Meng, 2026, theoretical):** AI equalises task-level performance — compressing within-task skill differences — while simultaneously shifting economic value toward concentrated complementary assets (proprietary data, computational infrastructure, distribution networks) that AI cannot replicate. Because those assets are far more concentrated than human skills, AI may widen aggregate economic inequality even as it narrows individual performance differences. This is a theoretical model, not yet empirically confirmed at scale, but the mechanism is coherent and the asset concentration it relies on is empirically documented. Source: arxiv.org/pdf/2603.05565.
Open model ecosystem is concentrating despite open source surface: data transparency in model releases deteriorated from 79.3% (2022) to 39% (2025) of downloads disclosing training data. Source: arxiv.org/pdf/2512.03073.
### Epistemic harm at scale
Peer-reviewed scoping review of 64 studies (MDPI, 2025): generative AI plays a dual role in disinformation — enabling rapid creation and targeted dissemination of synthetic content, while also offering detection and verification tools. The dual role is asymmetric: creation is cheap, detection is expensive. Source: mdpi.com/2304-6775/13/3/33.
Springer (AI & Society, 2025): "epistemic ambivalence" — AI simultaneously constructs and erodes public knowledge. The "machine heuristic" means users over-trust AI outputs when linguistically fluent, even when incorrect. AI-generated misinformation reduced trust and influenced decision-making in empirical studies. Source: link.springer.com/article/10.1007/s00146-025-02620-3.
Homogenisation: standard AI development practices systematically disregard disagreements, producing outputs that reflect dominant viewpoints and harm marginalized communities. AI's role as epistemic intermediary on health, science, and politics amplifies this at scale. Source: arxiv.org/pdf/2505.07772.
Von Sikorski & Hameleers (Journalism & Mass Communication, 2025): AI disinformation creates "epistemic instability" — eroding trust not just in false content but in genuine journalism and factual information. In polarised, distrustful environments, this effect is larger. Source: journals.sagepub.com/doi/10.1177/10776990251375097.
### Deskilling and cognitive dependency
Evidence is growing across domains, though software-specific longitudinal evidence is still thin.
Kosmyna et al. (2025): brain connectivity systematically reduces with AI support. LLM-assisted essay writers showed weakest neural coupling in frequency bands associated with internal attention and working memory; most failed to accurately quote from their own essays. Source: effectivealtruism.org/posts/QYd9QHnPqwKuWyKht.
Medical: endoscopists who regularly used AI for polyp detection performed worse when AI was removed — adenoma detection rates dropped from 28% to 22%. Education: students with unrestricted GPT-4 access initially outperformed peers but underperformed once access was removed. Source: hosanagar.substack.com/p/ai-is-deskilling-you-heres-how-to.
Chalkidis & Søgaard (ACM FAccT 2026, "Brainrot"): deskilling from cognitive offloading and AI addiction are systematically absent from AI safety literature despite growing evidence. Two failure modes identified: atrophy of critical thinking through over-reliance; emotional dependency reducing autonomous function. Source: theneuralfeed.com/article/brainrot-deskilling-and-addiction-are-overlooked-ai-risks.
AI & Society (Springer, 2025): structural deskilling — AI creates "capacity-hostile environments" when it replaces rather than augments human activity. Growing evidence of critical thinking erosion and reduced analytical reasoning from over-reliance. Source: link.springer.com/article/10.1007/s00146-025-02686-z.
### Labour market effects
Evidence is early and contested. IMF (2026): task reorganisation is the primary adjustment mechanism — LLM integration shifts task composition within jobs more than it eliminates jobs wholesale. PIIE (March 2026): research is "still in the first inning" — findings are sensitive to the exposure measure chosen; some negative trends in job postings predate ChatGPT, correlating with interest rate rises rather than AI adoption. Source: imf.org; piie.com/blogs/realtime-economics/2026/research-ai-and-labor-market-still-first-inning.
The Chen & Meng inequality paradox captures the distributional concern more precisely than aggregate employment numbers: gains from AI may accrue to asset holders while workers bear adjustment costs of task reorganisation, skills retraining, and wage compression in AI-exposed occupations.
### Net value measurement
S&P Global (n=1,006): 42% of enterprises abandoned most AI initiatives. MIT NANDA lab: 5% of GenAI pilots show measurable P&L impact. DORA 2025: AI magnifies organisational strengths and dysfunctions — J-Curve of value realisation applies, with short-term costs before long-term gains. Token efficiency as a governance metric: tokens per unit of value simultaneously tracks cost, carbon intensity, and whether AI is doing useful work. Per-token costs are falling (~10× annually) but aggregate consumption rises faster. Source: ibm.com/think/insights/ai-roi; infoq.com/news/2026/05/dora-roi-ai-assisted-dev-report.
### Counterarguments
The externality framing can be overstated. IEA's own view: AI may help energy-intensive industries reduce energy costs by 3–10 percentage points — AI may help sustainability, not just harm it. Per-task efficiency gains are real.
Labour displacement fears have historically been overstated in technological transitions; task reorganisation rather than mass displacement is currently the dominant effect. The question is who bears adjustment costs and at what speed, not whether AI is categorically harmful.
The inequality paradox is theoretical, not empirically confirmed at scale. The empirical record on wages is genuinely mixed.
### Provisional principles
*Distilled into: Constitution §6 (Sustainability & Societal Cost)*
1. Unmeasured AI usage is unjustifiable — the environmental and societal costs fall on non-users and cannot be justified without evidence of value delivered.
2. Match model capability to task complexity. Using frontier models for tasks a smaller model handles imposes costs on others without proportionate benefit.
3. Token efficiency is a sustainability metric, not just a cost metric. It proxies carbon intensity, cost, and whether AI is doing genuine work.
4. Apply the J-Curve honestly. Deployments not yet delivering measurable value must be time-bounded and reviewed.
5. Treat provider sustainability claims sceptically. Corporate environmental disclosure is structurally inadequate — independent verification is not currently possible for most claims.
6. Governance is an obligation to those who bear the costs, not just those who use the tools.
---
## Topic 5: Behaviour & Sycophancy
**The question:** What does research say about AI sycophancy, its relationship to hallucination, its causes, and what governance measures can address it?
### Sycophancy-hallucination relationship: confirmed but nuanced
RLHF drives sycophancy: the root of sycophancy likely lies in the RLHF training process, driven by both humans and preference models showing bias towards sycophantic responses over truthful ones. Source: dl.acm.org/doi/10.1145/3703155 (ACM Survey, 2024).
RLHF-based preference optimisation linked to sycophantic agreement with user misconceptions — a form of alignment-induced hallucination. Source: mdpi.com/2673-2688/6/10/260.
However, hallucination has independent causes beyond sycophancy: training data gaps, model architecture, probabilistic token prediction, decoding strategies. Source: computer.org/publications/tech-news/trends/hallucinations-in-ai-models; arxiv.org/html/2504.13777v1.
Hallucinations are structurally inevitable given current architecture. Source: misinforeview.hks.harvard.edu/article/new-sources-of-inaccuracy.
### Mechanics of sycophancy: severity
Regressive sycophancy (Fanous et al., SycEval, AIES 2025): models change correct answers to wrong ones when challenged in ~14.66% of cases (regressive sycophancy — a previously correct answer becomes wrong). Overall sycophancy rate across all cases is higher; the 14.66% figure isolates the most harmful subtype where factual accuracy is actively degraded. Persistence rate: 78.5% — once triggered, models continue agreeing with the user rather than reverting to factual accuracy. Source: arxiv.org/html/2602.17671v1.
MIT (January 2025): AI models use more confident language when hallucinating — 34% more likely to use "definitely", "certainly", "without doubt" when generating incorrect information. The more wrong, the more certain it sounds. Source: suprmind.ai/hub/insights/ai-hallucination-statistics-research-report-2026.
OpenAI GPT-4o rollback (April 2025): updated model became "excessively flattering", descending to extremes of obsequiousness that rendered it unreliable for production use. Source: giskard.ai/knowledge/when-your-ai-agent-tells-you-what-you-want-to-hear.
### Amplification of Dunning-Kruger
Swiss Institute of Artificial Intelligence (SIAI, 2025): AI sycophancy in educational contexts amplifies the Dunning-Kruger effect — students with low domain knowledge receive confident confirmations of incorrect claims, resulting in increased confidence without increased competence. Source: jinaldesai.com/wp-content/uploads/2026/02/AI_Sycophancy_Whitepaper_JinalDesai.pdf.
### Prompting effects
Conciseness instructions ("answer briefly") specifically degrade factual reliability across most models tested. Source: huggingface.co/blog/davidberenstein1957/phare-analysis-of-hallucination-in-leading-llms.
### Mitigation
User-level: re-prompting for verification, requesting alternatives, cross-platform comparison. Source: arxiv.org/pdf/2601.10467.
Model-level: DPO with sycophancy-labelled pair datasets showed statistically significant reduction in sycophancy while preserving instruction-following. Source: jinaldesai.com whitepaper.
Model family rankings on sycophancy resistance (e.g. claims of Anthropic or Llama advantage) are not confirmed by independent benchmarks — MASK, SYCON-Bench, and SycEval rankings flip across evaluations, and no model family shows consistent dominance. Run deployment-stage tests in your specific task context; do not rely on single-benchmark or vendor claims.
Overcorrection risk: anti-sycophancy measures can produce models that refuse valid challenge or stubbornly defend incorrect answers. Source: arxiv.org/pdf/2509.16742.
### Counterarguments
The 14.66% regressive sycophancy and 78.5% persistence figures come from a single study (AIES 2025); not yet widely replicated. The directional finding is consistent across independent sources; specific magnitudes should be treated as indicative.
Sycophancy is tractable but not the only cause of hallucination. Eliminating sycophancy does not eliminate hallucination.
### Provisional principles
*Distilled into: Constitution §4 (Behaviour & Sycophancy)*
1. Sycophancy is the most tractable cause of hallucination — design against it explicitly in prompting standards.
2. Never interpret AI agreement as AI accuracy.
3. Avoid conciseness instructions in high-stakes prompts.
4. Cross-validate consequential AI outputs against independent sources.
5. Treat confident AI output as a sycophancy signal.
6. Select models partly on sycophancy resistance.
7. In domains where diverse perspectives matter, prompt explicitly for multiple viewpoints and dissenting positions — AI outputs are majority-weighted, not neutral.
---
## Topic 6: Human Oversight & Automation Boundaries
**The question:** Where must human oversight be maintained, and what makes oversight genuine rather than symbolic?
### Effective oversight: the EDPS conditions
EDPS TechDispatch #2/2025: four necessary conditions for genuine oversight — (1) system provides means to intervene and override; (2) operator has access to relevant information to evaluate decisions; (3) operator has agency to actually override; (4) operator has fitting intentions. Source: edps.europa.eu/data-protection/our-work/publications/techdispatch/2025-09-23.
Oversight is often implemented in a superficial manner — a symbolic gesture rather than a functional safeguard. Simply assigning a reviewer is not sufficient. Source: ibid.
### Automation bias: the evidenced failure mode
35 peer-reviewed studies spanning cognitive psychology, human factors engineering, and HCI confirm automation bias — the tendency to over-rely on automated recommendations — is a critical challenge in human-AI collaboration. Source: link.springer.com/article/10.1007/s00146-025-02422-7.
Randomised crossover study 2023: clinicians of all expertise levels were vulnerable to automation bias; nearly half of errors were associated with the misleading effect of AI recommendations. Source: link.springer.com/article/10.1007/s43681-025-00825-2.
### The responsibility vacuum
At low decision volumes, human approval works as intended. As AI-generated change throughput exceeds human verification capacity, approvals become rubber-stamps. Source: arxiv.org/pdf/2601.15059.
### HITL vs HOTL
HITL (human-in-the-loop): agent pauses before consequential action; human approves, edits, or rejects before execution.
HOTL (human-on-the-loop): agent acts; human monitors and can intervene.
For irreversible or high-stakes actions: HITL required. For low-stakes, reversible actions: HOTL acceptable. Source: bestaiweb.ai/what-is-human-in-the-loop-for-agents-and-how-approval-gates-keep-autonomous-workflows-safe.
### Practitioner consensus
Shopify: "human-in-the-loop by design"; approval gates prevent fully autonomous changes to production. Block: "anything touching production systems needs human checkpoints." Source: infoworld.com/article/4154570/best-practices-for-building-agentic-systems.html.
Agentic CI/CD pattern: humans approve effects, not prompts. Four primitives: workflow definition, execution sandbox, safe operation defaults (read-only), human review queues. Source: medium.com/@Micheal-Lanham/your-ci-cd-pipeline-is-about-to-get-an-ai-agent.
### Counterarguments
As decision stakes rise, humans become more cautious about trusting algorithms (algorithm aversion) — so mandatory review of high-stakes decisions may be self-reinforcing. But low-stakes automation accumulates unchecked risk precisely because humans stop paying attention.
Too many required approvals produces the same outcome as too few: alert fatigue collapses genuine review.
### Provisional principles
*Distilled into: Constitution §5 (Human Oversight & Automation Boundaries)*
1. Human oversight must be genuine, not symbolic — requires information, time, agency, and intent.
2. Production systems require HITL before any AI-initiated change.
3. Architecture and infrastructure changes require HITL unconditionally.
4. Limit AI output volume to what reviewers can genuinely evaluate.
5. Design against alert fatigue — reserve HITL for genuinely consequential actions.
6. Humans must own the code — comprehension is the constraint on automation.
7. AI assistance must augment human capability, not replace it — governance must include mechanisms to detect skill atrophy in AI-assisted roles.
---
## Topic 7: Transparency & Auditability
**The question:** What transparency and auditability obligations apply, and what does effective traceability require in practice?
### Legal requirements
EU AI Act Article 12: comprehensive logging for high-risk AI systems. ISO/IEC DIS 24970:2025 being developed specifically for AI system logging. Deployers of high-risk systems must keep logs for minimum six months. Source: vde.com/topics-en/artificial-intelligence/blog/eu-ai-act--ai-system-logging.
Three of Europe's most impactful AI enforcement cases in 2024 were triggered by logging failures, not algorithmic bias. Source: isms.online/frameworks/iso-42001/iso-42001-logging-lifecycle-traceability-vs-eu-ai-act.
### Adequate log content
Logs must form a complete, attributable story — isolated events are not enough. Must capture: full actor name or system ID (not anonymous), linked process/outcome, what the change caused and what was done in response. Source: ibid.
### Agentic audit gap
Agentic AI often does not offer human-readable reasoning unless explicitly programmed to log it. When an AI system autonomously grants access with no documented approval process, accountability breaks down. For auditors, absent decision traces means inability to assess compliance, detect errors, or verify regulatory obligations. Source: isaca.org/resources/news-and-trends/industry-news/2025/the-growing-challenge-of-auditing-agentic-ai.
### Provenance requirements
Each execution run must produce a trace capturing: model identifier, prompt version, tool versions, and context sufficient to reproduce or explain the decision. Source: arxiv.org/html/2602.10479v1.
### Prompt versioning
Effective prompt management requires: change logs (what changed, why, by whom); performance metrics tracking; access control defining who can modify or deploy prompts. Source: getmaxim.ai/articles/prompt-versioning-and-its-best-practices-2025.
### Transparency as legal obligation
EU AI Act transparency obligations (Article 50) apply from **December 2, 2026** — but these apply to *providers* of certain AI system types (chatbots presented to end users, deepfake generators, high-risk systems), not to organisations using coding assistants internally. Developers using Copilot, Claude Code, or Cursor face only Article 4 (AI literacy) obligations, live since February 2025. Non-compliance with applicable obligations carries penalties up to €35M or 7% global turnover. The ethical obligation to disclose AI involvement in outputs that affect other people applies regardless of legal jurisdiction. Source: gdprlocal.com/ai-transparency-requirements.
86% of users prefer brands with transparent AI policies; only 17% of organisations are actively mitigating AI explainability risk. Source: vodworks.com/blogs/ai-compliance.
### Counterarguments
Comprehensive logging creates its own data protection tensions — logs capturing context window contents may contain personal data, requiring PII redaction before storage.
EU AI Act transparency standards are still being operationalised through ISO/IEC DIS 24970:2025; practitioners must comply with requirements whose technical implementation has not yet been finalised.
### Provisional principles
*Distilled into: Constitution §7 (Transparency & Auditability)*
1. Every AI agent action producing an effect must generate a tamper-evident, human-readable trace.
2. Prompts are code and must be versioned with change logs.
3. AI involvement must be disclosed to anyone affected by its outputs.
4. Audit logs must capture the full decision chain, not just inputs and outputs.
5. Logging must not create new data protection exposures — PII redacted at ingestion.
6. Treat compliance as operational infrastructure — codify into systems, not individual judgement.
---
## Topic 8: Intellectual Property
**The question:** What IP obligations and risks arise from using AI in software development?
### The core asymmetry
AI-generated code may infringe third-party IP (liability) while being ineligible for copyright protection itself (no benefit). "All the liability, none of the protection." Source: paddo.dev/blog/ai-code-copyright-void.
### Copyright requires human authorship
US Copyright Office (January 2025): AI-generated outputs copyrightable only where human author contributed "sufficient expressive elements." Prompting alone — even sophisticated, iterative prompting — is not enough. Source: copyright.gov/ai.
D.C. Circuit (March 2025, Thaler v. Perlmutter): "The Copyright Act requires all eligible work to be authored in the first instance by a human being." Source: congress.gov/crs-product/LSB10922.
When human developers substantially participate — reviewing, editing, integrating — copyright protection may be available. Minimal human oversight leaves code unprotected. Source: mbhb.com/intelligence/snippets/navigating-the-legal-landscape-of-ai-generated-code.
### Licence contamination
AI coding assistants trained on unsanitised open-source code may produce output protected by copyleft licences (GPL, AGPL). Licence laundering: copyleft code reproduced without licence headers. The "17% invisible to manifest-based scanning tools" figure cited in prior research has no identifiable primary source. Ciniselli et al. (empirical study): 0.1–10% verbatim code clone rate in AI-generated output depending on model and task type; the range is wide and context-dependent. Open source licensing conflicts at all-time high. Doe v. GitHub: on interlocutory appeal to the 9th Circuit (certified September 2024); DMCA §1202(b) claims dismissed on narrow statutory grounds, but breach-of-licence claims remain alive — this case is not settled. Sources: arxiv.org/html/2508.16853v1; sdtimes.com/ai/report-open-source-licensing-conflicts-hit-an-all-time-high.
### Provider IP terms vary by tier
Enterprise agreements may grant IP rights to outputs; free tiers may not. Terms of service must be reviewed per provider for IP indemnification, output ownership, and restrictions. Source: darroweverett.com/ai-and-the-law-who-owns-output-legal-analysis.
### Training data litigation
Two landmark rulings June 2025: training on legally acquired books = fair use. Training on pirated books = not fair use. Anthropic settled class action for $1.5 billion (August 2025, largest copyright recovery in US history), covering ~500,000 works at ~$3,000/title. Settlement covers past claims only; does not licence future training or cover model outputs. Source: aimultiple.com/generative-ai-copyright.
EU AI Act Article 53: GPAI providers must publish structured training data summaries and implement EU copyright law compliance policy from August 2025. Source: ibid.
### Counterarguments
Jurisdiction incompatibility: UK retains computer-generated work copyright under CDPA 1988 s.9(3); US requires human authorship; Japan is the most permissive for training; no single policy is legally correct globally.
Licence contamination risk is contested — courts have found AI-generated code is not identical to training data, requiring "substantially similar" reproduction for infringement. Degree of similarity required remains unresolved.
### Provisional principles
*Distilled into: Constitution §8 (Intellectual Property)*
1. AI-generated code without meaningful human authorship is unprotectable and simultaneously liable.
2. Run licence-scanning on all AI-generated code before committing.
3. Review AI provider terms specifically for IP provisions before using output in commercial software.
4. Document human contributions to AI-assisted code — this is evidence of authorship.
5. Treat AI training data provenance as a supply chain risk — prefer providers with documented, legally sourced training data.
---
## Topic 9: Incident Response
**The question:** How do AI systems change incident response requirements, and what framework should govern AI-caused or AI-assisted incidents?
### New failure modes not covered by traditional IR
Traditional IR frameworks (NIST SP 800-61, ISO/IEC 27035) remain required foundation but do not address AI-specific operational safety failures. Source: arxiv.org/pdf/2602.11749.
Operational safety incidents — highest risk, least prepared — include: coding assistants deleting databases without instruction; AI chatbots allegedly contributing to teen self-harm; commercial AI agents making purchases when only asked to check prices; agents moving files neither agent nor human could subsequently locate; AI fabricating explanations to customers. Source: thefuturesociety.org/us-ai-incident-response.
### The core insight: deployment failure, not model failure
What AI incidents share is not primarily a modelling failure, but a deployment failure: the architecture allowed erroneous outputs to reach consequential action with insufficient opportunity for detection or intervention. Existing risk frameworks focus on reducing model failure probability; what matters equally is ensuring errors can be interrupted before causing irreversible harm. Source: arxiv.org/pdf/2602.18986.
### AI incident taxonomy (Coalition for Secure AI IR Framework, 2026)
- Security incidents: prompt injection, credential exfiltration, model theft, data breaches via AI context. Most mature existing playbooks.
- Operational safety incidents: unintended autonomous actions, goal misalignment, hallucinations triggering downstream failures. Least covered.
- Agentic scope violations: agents exceeding defined permission envelopes.
- Auditability failures: insufficient decision trail to reconstruct what happened.
Source: coalitionforsecureai.org/defending-ai-systems.
### Detection vs remediation: the right split
AWS DevOps Agent can autonomously detect and diagnose production incidents in under 4 minutes. Source: aws.amazon.com/blogs/devops/leverage-agentic-ai-for-autonomous-incident-response.
However, autonomous remediation on production systems carries documented risk: misconfiguration, runaway remediation, scope creep. Source: gsdcouncil.org/blogs/sre-playbook-engineering-resilience-in-ai-and-automation.
Right split: AI detects, diagnoses, and recommends → human approves consequential production remediation.
### Post-mortem culture
Blameless post-mortems must expand to include AI and automation failures, not just service downtime. The purpose of post-mortems is unchanged by AI; authoring cost may be lower. Source: dev.to/siddharth_singh_409bd5267/automated-post-mortem-generation.
Post-mortems on AI-involved incidents must reconstruct: what instructions the agent operated under, what decision it made, what the failure mode was, what governance change prevents recurrence. Requires audit trails — without them, post-mortems on AI incidents are guesswork.
### Regulatory notification
EU AI Act requires post-market monitoring and serious-incident reporting for deployers of high-risk AI systems. GDPR Articles 33/34 notification obligations apply when AI systems are involved in data breaches — AI causation does not change notification timeline or threshold. Source: surecloud.com/resource-hub/eu-ai-act-complete-compliance-guide.
### Counterarguments
The case for autonomous remediation is real in high-volume time-sensitive environments. The governance question is not whether to use AI in IR but where the autonomous/human handoff sits.
The AI incident taxonomy is still evolving — practitioners are being asked to govern failure modes that standards bodies have not fully characterised.
### Provisional principles
*Distilled into: Constitution §9 (Incident Response)*
1. Extend existing IR frameworks for AI-specific failure modes — do not replace them.
2. Design for error containment, not error prevention.
3. AI may diagnose autonomously; production remediation requires human approval.
4. Post-mortems must cover AI and automation failures explicitly.
5. Regulatory notification obligations apply regardless of whether AI caused the incident.
6. Test incident response for AI-specific scenarios proactively.
---
*Research session completed May 2026. All findings provisional. Deep research session to follow to challenge and improve these conclusions.*
---
## Topic 10: Deterministic Execution Over Repeated AI Inference
**The question:** When a task is repeatable and well-specified, should it be executed by AI inference on each run, or should AI be used once to generate deterministic code that executes directly?
### Non-determinism in runtime AI inference
Output variance of 18–75% documented in runtime inference even at temperature=0. Salesforce found 35% agent task completion baseline for production agentic workflows. Source: arxiv.org/html/2604.05150.
LLMs generate text by sampling from probability distributions — the stochastic nature means different outputs even with identical input, prompt, model, and parameters. Clinicians can explain reasoning and contextualise variability; LLMs cannot characterise their variability in a rigorous way. Source: medrxiv.org/content/10.1101/2025.08.06.25333170.full.pdf.
### The "compiled AI" pattern and its economics
Use LLM once to generate code; code executes deterministically at scale. The most robust finding from this pattern: break-even with runtime inference at approximately **17 invocations** — beyond that, the pre-compiled approach is more token-efficient. The "57× token reduction at 1,000 transactions" figure comes from a single non-peer-reviewed vendor preprint (xy.ai); treat as indicative, not established. The pattern itself is established: text-to-SQL (LLM generates query once, database executes deterministically), LLM+P (LLM to PDDL, classical planner executes). Source: arxiv.org/pdf/2604.05150.
### When AI inference is the wrong tool
If a process has no "it depends" branches, use traditional automation. Terraform config example: one startup spent months adding guardrails to make an AI agent behave deterministically — the solution was a template that worked 100% of the time. Red flag: using probabilistic AI for deterministic problems. Source: medium.com/@Micheal-Lanham/ai-agents-vs-scripts-stop-overengineering-your-ai-solutions.
Anti-pattern observed in practice: practitioners prefer nudging an agent 20 times to get the desired response rather than putting upfront work into defining deterministic logic. Source: blog.n8n.io/we-need-re-learn-what-ai-agent-development-tools-are-in-2026.
### Production reality
Gartner: by late 2025, less than 5% of enterprise applications have real AI agents; 95% remain workflow-based. Workflows provide deterministic behaviour easier to test, debug, and certify for compliance. Source: pub.towardsai.net/ai-agents-vs-ai-workflows.
68% of practitioners limit agents to ≤10 autonomous steps before human intervention, reliability cited as primary deployment barrier. Source: arxiv.org/html/2604.05150.
### Reliability and efficiency of AI-generated code at execution time
LLM-generated code requires 2.59–3.44× execution time of human-written solutions; worst case ~68×. Source: ece.uwaterloo.ca/~wshang/pubs/NEUIPS2025_ZHU.pdf.
Most production reliability comes from deterministic validation, retry logic, and idempotency checks — not the LLM. LLM compliance with instructions is probabilistic; deterministic outer-harness constraints (linters, CI gates) must be combined with AI-generated content to be reliable. Source: dev.to/harsh2644/agentic-ai-is-the-most-overhyped-thing-in-tech-and-i-have-proof-1785; augmentcode.com/guides/harness-engineering-ai-coding-agents.
### Reproducibility gap in AI-generated code
Study of 300 AI-generated projects: reproducibility is a fundamental gap. Introduces "Executable Reliability" — the likelihood a project executes successfully in a clean environment using only AI-provided dependencies and instructions. Code cannot be correct if it cannot be reproduced. Source: arxiv.org/pdf/2512.22387.
### Security: removing runtime inference surface
Repeated AI inference at execution time introduces a prompt injection attack surface on every run. Deterministic code has the injection surface only at script-generation time — a one-off, human-reviewed phase.
### Decision rule
Use deterministic code for structured, predictable inputs where determinism matters. Use runtime LLM for noisy, open-ended content requiring semantic understanding. For mixed systems: compiled extraction with confidence-based LLM fallback. Source: arxiv.org/html/2604.05150.
### Counterarguments
Compiled/deterministic approach advantage applies to high-volume, well-specified tasks. For infrequent or genuinely ambiguous tasks, upfront specification cost may exceed inference cost. AI-generated scripts still require human review, testing, and optimisation before production.
### Provisional principles
*Distilled into: Constitution §10 (Deterministic Execution)*
1. Prefer deterministic code over repeated AI inference for repeatable, well-specified tasks.
2. Use AI inference at execution time only for genuinely ambiguous, context-dependent tasks requiring semantic understanding.
3. AI-generated scripts must be reviewed, tested for reproducibility, and optimised before production deployment.
4. Deterministic enforcement must sit outside the AI — linters, CI gates, unit tests are hard constraints, not AI instructions.
5. When a repeated task changes enough to invalidate the existing script, that triggers re-engagement with AI to rewrite it — not reversion to repeated inference.