holocron/docs/research/governance_principles/ai-governance-research.md

AI Governance Research Document

Purpose: Human reference document. The "why" behind every principle in the constitution. Contains sourced findings, counterarguments, bias flags, and provisional principles for all ten governance topics. Does not go into AI agent context — too long.

How to use this document: This is the research layer, not the operative layer. If you want to know what to do, read ai-constitution.md. If you want to know why a principle exists, challenge a finding, or update the evidence base, read the relevant topic here. Each topic ends with provisional principles that map directly to a constitution section — cross-references are noted.

Session: May 2026 Methodology: Topic-by-topic web research from reliable sources. All conclusions are provisional and challengeable. Research must remain unbiased — findings drive principles, not the other way around.

---

Topic Index

#	Topic	Core finding	Principles	Constitution
1	Ethics	Accountability non-transferable; sycophancy is an ethical failure, not just a quality one	4	§1, §4
2	Security	AI-assisted repos have 40% higher secret leak rates; prompt injection is actively exploited in CI/CD	6	§2
3	Data Protection	The context window is a data store; consumer/enterprise tier distinction is a hard legal line	6	§3
4	Sustainability	Costs are externalised to non-users; governance is an obligation to those who bear them	6	§6
5	Behaviour & Sycophancy	14.66% regressive sycophancy; 78.5% persistence; confident language inversely correlated with accuracy	7	§4
6	Human Oversight	Automation bias confirmed in 35 peer-reviewed studies; symbolic oversight is the documented failure mode	7	§5
7	Transparency	Three of Europe's most impactful 2024 AI enforcement cases were triggered by logging failures, not bias	6	§7
8	Intellectual Property	No copyright without human authorship; 0.1–10% verbatim clone rate in AI output	5	§8
9	Incident Response	Deployment failure, not model failure, is the primary cause of AI incidents	6	§9
10	Deterministic Execution	Break-even with runtime inference at ~17 invocations; non-determinism compounds across agentic chains	5	§10

---

Topic 1: Ethics

The question: What ethical obligations govern AI use in software development, deployment, and infrastructure management?

Regulatory and normative baseline

The OECD AI Principles (2019, updated May 2024) are the first intergovernmental standard on AI. The 2024 update strengthened provisions on misinformation/disinformation from generative AI and explicitly added environmental sustainability. Five values: inclusive growth and wellbeing; rule of law and human rights; transparency and explainability; robustness, security, and safety; accountability. Source: OECD AI Principles (oecd.ai/en/ai-principles).

UNESCO Recommendation on the Ethics of AI (2021, 194 member states): AI systems should be auditable and traceable; member states must ensure AI systems do not displace ultimate human responsibility. Source: UNESCO (unesco.org/en/artificial-intelligence/recommendation-ethics).

EU AI Act (in force August 2024, phased obligations per Omnibus May 2026 agreement): Article 4 (AI literacy) applies from February 2025. Article 50 transparency (chatbots, deepfake generators) applies from December 2, 2026. Standalone Annex III high-risk systems: December 2, 2027. Embedded high-risk: August 2, 2028. Developers using coding assistants (Copilot, Claude Code, Cursor) face only Article 4 obligations — these tools are minimal/limited risk under the Act, not high-risk. Source: EU digital-strategy.ec.europa.eu.

Epistemic integrity: sycophancy as an ethical problem

Turner and Eisikovits (Springer, 2026): AI sycophancy is an "artificial vice" generating moral and epistemic harms. Key distinction between UX friction reduction (acceptable) and truth-related friction reduction (sycophancy — not acceptable). Source: link.springer.com/article/10.1007/s43681-026-01007-4.

Empirical harm to decision-making: brief conversations with sycophantic AI increased attitude extremity and certainty while inflating users' self-perceptions; users rated sycophantic responses as higher quality and expressed greater willingness to use them again — creating a perverse incentive where users seek out the systems that distort their reasoning. Source: arxiv.org/pdf/2602.14270.

Epistemic autonomy: deskilling

Deskilling is well-documented in technological transitions: when humans outsource decisions to technology, skills atrophy. Source: arxiv.org/pdf/2503.22151.

METR study (July 2025): Experienced open-source developers using AI tools took 19% longer than without them. METR's own February 2026 update supersedes this: with current tools, new participants show approximately −4% (marginal slowdown) and METR now states "AI likely provides productivity benefits in early 2026." Cui et al. (Management Science 2026, n=4,867): +26% completed tasks with Copilot. The productivity picture is task- and context-dependent; early negative results reflected tool immaturity rather than a structural effect. Source: metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study; METR February 2026 update.

Ethics washing: a documented failure mode

Multiple peer-reviewed sources confirm that corporate AI ethics frameworks often function as public relations rather than genuine governance. Rességuier and Rodrigues (2020): AI ethics frameworks have come to stand in for effective regulation, providing few legal or political protections. Meredith Whittaker: "steep cost of corporate capture in AI ethics." Source: journals.sagepub.com/doi/10.1177/20539517231221780.

Nearly 100 non-legally-binding ethical codes or statements adopted in five years, putting forward mostly the same principles without producing measurable change. Source: link.springer.com/article/10.1007/s44206-025-00174-x.

Counterarguments

Accountability-is-non-transferable is well-evidenced but incomplete: some researchers argue full deployer responsibility when AI failure modes are opaque may be unreasonably demanding — the more important lever may be liability for AI developers to build safer systems.

The ethics washing critique does not mean ethics frameworks are worthless — it means poorly operationalised ones are.

Provisional principles

Distilled into: Constitution §1 (Accountability) and §4 (Behaviour & Sycophancy)

1. Accountability is legally and ethically non-transferable.
2. Sycophancy is an ethical failure mode, not just a performance failure mode; designing against it is an ethical obligation.
3. Ethics commitments must be concrete and auditable — stated values without enforcement mechanisms are ethics washing.
4. Human oversight is a legal requirement for high-risk AI use under the EU AI Act, not a design option.

---

Topic 2: Security

The question: What security risks does AI specifically introduce into software development, deployment, and infrastructure pipelines?

The structural mismatch with traditional security

Traditional security controls — dependency scanning, SBOM analysis, network controls, endpoint protection — do not address AI-specific attack vectors. Prompt injection cannot be firewalled the same way a port can be. Source: thehackernews.com/2025/12/traditional-security-frameworks-leave.html.

AI security incidents increased 56.4% from 2023 to 2024 (Stanford HAI AI Index). Source: cycode.com/blog/ai-security-vulnerabilities.

OWASP LLM Top 10 (2025)

LLM01: Prompt Injection | LLM02: Sensitive Information Disclosure (jumped from #6 to #2) | LLM03: Supply Chain | LLM04: Data and Model Poisoning | LLM05: Improper Output Handling | LLM06: Excessive Agency | LLM07: System Prompt Leakage | LLM08: Vector and Embedding Weaknesses | LLM09: Misinformation | LLM10: Unbounded Consumption. Source: trydeepteam.com/docs/frameworks-owasp-top-10-for-llms. A separate Agentic AI Top 10 was released late 2025. Source: bsg.tech/blog/owasp-llm-top-10.

Prompt injection: actively exploited

Johns Hopkins researchers hijacked Claude Code Security Review, Gemini CLI Action, and Microsoft GitHub Copilot through prompt injection — exfiltrating API keys, GitHub access tokens, and secrets via malicious PR metadata. Source: letsdatascience.com/news/researchers-hijack-ai-coding-agents-steal-credentials-d508cbeb.

Secrets leakage: severe and worsening

GitGuardian State of Secrets Sprawl 2025: 23.8 million new credentials detected on public GitHub in 2024, 25% year-on-year increase. AI-assisted repos: 40% higher leak rate than baseline. Claude Code commits leaked secrets at double the baseline rate in 2025. 28.65 million hardcoded secrets added to public GitHub in 2025; AI-service leaks surged 81%. Sources: csoonline.com/article/3953927; helpnetsecurity.com/2026/04/14; helpnetsecurity.com/2026/04/15.

MCP configuration files: 24,008 unique secrets found in MCP config files; pattern is pervasive including official GitHub MCP server. Source: wiz.io/blog/leaking-ai-secrets-in-public-code.

AI-generated passwords: insufficient entropy

Claude's passwords start with uppercase "G" and digit "7"; 50 Claude-generated passwords produced only 30 unique results; measured entropy 27 bits vs 98 bits expected. Source: dev.to/0x711/ai-agents-dont-understand-secrets-thats-your-problem-43n4.

Supply chain

As few as 50,000 fake articles in a public training dataset corrupted medical LLMs; small quantities of poisoned information corrupted even large models. Attacks carried out against RAG pipelines, MCP tools, and synthetic data workflows in 2025. Source: cycode.com/blog/ai-security-vulnerabilities.

Uncertainty propagation

Composing AI subsystems with uncertain performance into automated pipelines creates compounding error invisible at the output layer. Source: ncbi.nlm.nih.gov/pmc/articles/PMC12747714.

Counterarguments

The 40% higher leak rate for Copilot repositories does not prove causation — early AI tool adopters may also be the fastest-moving, least-careful developers. Causation is plausible but not definitively proven.

Provisional principles

Distilled into: Constitution §2 (Security)

1. Secrets must never enter AI context — architectural constraint, not policy.
2. AI-generated code is untrusted by default; review with more scrutiny than human-written code.
3. Least-privilege for all AI agents; no long-lived broad-scope tokens.
4. OWASP LLM Top 10 and Agentic AI Top 10 are baseline requirements.
5. Automated AI pipelines must surface and preserve uncertainty.
6. Never use AI-generated secrets, passwords, or cryptographic material.

---

Topic 3: Data Protection & Classification

The question: What data protection obligations apply when using AI, and what classification framework governs what may enter AI systems?

Concurrent application: GDPR + EU AI Act

Both apply simultaneously whenever personal data is processed by AI. CNIL confirmed: "Where personal data is used for the development of an AI system, both the GDPR and the AI Act apply." Source: cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr.

Prompts as data processing

EDPB Opinion 28/2024: LLM operations, training, and use raise wide-ranging data protection concerns; addressed the legal basis of legitimate interest for AI models and consequences of unlawful processing during development on subsequent operation. Source: edpb.europa.eu (Opinion 28/2024).

Context window as data store

Prompts, chat logs, transcripts, and support workflows are data stores traditional classification programs do not scan. Organisations must define what PII can enter prompts, what must be redacted, and what should never enter training sets. Source: forcepoint.com/blog/insights/pii-data-classification.

International transfers

US-based LLM providers (OpenAI, Anthropic, Google) process data in US data centres. GDPR prohibits free transfer of EU citizen data outside EEA without adequacy decisions, appropriate safeguards, or derogations. Using an enterprise tier with a DPA does not eliminate this obligation; it structures it. Source: nexos.ai/blog/gdpr-ai.

Right to erasure incompatibility

LLMs present significant challenges to GDPR Article 17 (right to erasure) because personal data is encoded in model weights, not stored as retrievable records. The decision to include personal data in training is irreversible. Source: mdpi.com/1999-5903/17/4/151.

Enterprise vs consumer: a hard distinction

Use enterprise versions only; ensure the provider does not use prompts, RAG data, or fine-tuning data for model training. It is unlikely possible to demonstrate a legitimate interest in transferring data to a provider to train the base model. Source: dataprotectionreport.com/2025/01.

Data minimisation

CNIL: AI systems must be developed with a well-defined objective to frame and limit the personal data used — data minimisation applies to prompts. Source: cnil.fr/en/ai-system-development-cnils-recommendations-to-comply-gdpr.

Classification framework

Four-tier model mirrors ISO 27001 guidance and is the established standard. Public / Internal / Confidential / Restricted. For AI: AI systems shall be classified at the level of the highest-classified data used for training or operation; model weights inherit classification from training data. Sources: pertamapartners.com/insights/ai-data-classification-categorizing-data; dpo-consulting.com/blog/gdpr-data-classification.

Counterarguments

Legal basis for AI training is genuinely unsettled. EU Digital Omnibus (November 2025) proposed allowing AI training on legitimate interest under Article 6(1)(f) — contested and evolving. The right-to-erasure incompatibility is contested at the margins: some argue encoded weights do not constitute personal data in a legally retrievable sense.

Provisional principles

Distilled into: Constitution §3 (Data Protection & Classification)

1. Sending personal data to an AI system is data processing under GDPR.
2. Consumer and free-tier AI products are incompatible with processing organisational or personal data.
3. The context window is a data store — classify it accordingly (four-tier framework).
4. Restricted-tier data (GDPR Art. 9, credentials, regulated data) must never enter any AI context.
5. Data minimisation applies to AI prompts.
6. Personal data must not enter AI fine-tuning or RAG without GDPR legal basis and completed DPIA.

---

Topic 4: Sustainability & Societal Cost

The question: What are AI's actual environmental and societal costs, who bears them, and what does that imply for governance?

Research note: This section was rewritten in Session 3 based on an independent web research pass (May 2026). Previous versions used a business/ROI lens and contained unverified claims. This version uses a societal/externality framing with sourced findings.

The core governance insight: costs are externalised, benefits are captured

AI's primary costs — environmental, epistemic, and distributional — fall predominantly on people who are not the users or deployers of AI systems: communities near data centres, populations affected by grid stress, workers displaced faster than they can upskill, and societies that absorb the epistemic and democratic effects of large-scale AI-generated content. The governance obligation flows from this asymmetry: those who benefit from AI use have an obligation to the people who bear its costs, whether or not those costs are legally required to be accounted for.

Environmental cost

Electricity consumption (IEA, authoritative): Global data centre electricity consumption was approximately 485 TWh in 2025, up 17% from ~415 TWh in 2024. AI-focused data centres grew faster — 50% in 2025 alone. The IEA's base-case projection: data centres roughly double to ~950 TWh by 2030, accounting for ~3% of global electricity demand; AI-focused data centres triple over the same period. Source: IEA Key Questions on Energy and AI (April 2026); IEA Energy and AI (April 2025). iea.org/reports/energy-and-ai.

Carbon and water footprint (peer-reviewed): AI systems are estimated to generate 32.6–79.7 million tonnes CO₂ equivalent in 2025; water footprint 312.5–764.6 billion litres. The upper bound rivals New York City's annual emissions. The IEA projects data centres will reach ~1% of global CO₂ emissions by 2030 in its central scenario — one of the few sectors where emissions are projected to grow rather than decarbonise. Source: de Vries-Gao (ScienceDirect, December 2025); sciencedirect.com/science/article/pii/S2666389925002788.

Geographic concentration creates localised stress: AI infrastructure is >90% concentrated in North America, Western Europe, and Asia-Pacific. Specific regions (Oregon, Virginia, Ireland) face Power Stress Index values indicating local grid vulnerability. Six leading firms account for compute capacity equivalent to ~1% of global power demand by 2030. Source: arxiv.org/pdf/2604.06198.

Corporate disclosure is structurally inadequate: Environmental reports from data centre operators do not distinguish AI from non-AI workloads. Independent verification of AI-specific footprint is not currently possible without regulatory disclosure mandates. Microsoft emissions grew 23.4% since 2020, citing AI expansion. Source: de Vries-Gao (ScienceDirect 2025); brookings.edu/articles/global-energy-demands-within-the-ai-regulatory-landscape.

Efficiency vs. scale: Per-task energy use is declining rapidly — IEA notes efficiency improvements at a rate unprecedented in energy history. However, scale of adoption outpaces efficiency gains, and energy-intensive use cases (AI agents, multimodal generation) are growing fastest. Source: IEA Key Questions on Energy and AI (April 2026).

Power and wealth concentration

Big Tech combined AI capex exceeded $400 billion in 2025, projected to rise ~75% in 2026. AWS, Azure, and Google Cloud hold ~63% of cloud infrastructure market share. US private AI investment ($109.1 billion in 2024) was 11.7× China's investment. Source: Stanford HAI AI Index 2025; quantumrun.com/consulting/ai-market-share-by-company-statistics.

The inequality paradox (Chen & Meng, 2026, theoretical): AI equalises task-level performance — compressing within-task skill differences — while simultaneously shifting economic value toward concentrated complementary assets (proprietary data, computational infrastructure, distribution networks) that AI cannot replicate. Because those assets are far more concentrated than human skills, AI may widen aggregate economic inequality even as it narrows individual performance differences. This is a theoretical model, not yet empirically confirmed at scale, but the mechanism is coherent and the asset concentration it relies on is empirically documented. Source: arxiv.org/pdf/2603.05565.

Open model ecosystem is concentrating despite open source surface: data transparency in model releases deteriorated from 79.3% (2022) to 39% (2025) of downloads disclosing training data. Source: arxiv.org/pdf/2512.03073.

Epistemic harm at scale

Peer-reviewed scoping review of 64 studies (MDPI, 2025): generative AI plays a dual role in disinformation — enabling rapid creation and targeted dissemination of synthetic content, while also offering detection and verification tools. The dual role is asymmetric: creation is cheap, detection is expensive. Source: mdpi.com/2304-6775/13/3/33.

Springer (AI & Society, 2025): "epistemic ambivalence" — AI simultaneously constructs and erodes public knowledge. The "machine heuristic" means users over-trust AI outputs when linguistically fluent, even when incorrect. AI-generated misinformation reduced trust and influenced decision-making in empirical studies. Source: link.springer.com/article/10.1007/s00146-025-02620-3.

Homogenisation: standard AI development practices systematically disregard disagreements, producing outputs that reflect dominant viewpoints and harm marginalized communities. AI's role as epistemic intermediary on health, science, and politics amplifies this at scale. Source: arxiv.org/pdf/2505.07772.

Von Sikorski & Hameleers (Journalism & Mass Communication, 2025): AI disinformation creates "epistemic instability" — eroding trust not just in false content but in genuine journalism and factual information. In polarised, distrustful environments, this effect is larger. Source: journals.sagepub.com/doi/10.1177/10776990251375097.

Deskilling and cognitive dependency

Evidence is growing across domains, though software-specific longitudinal evidence is still thin.

Kosmyna et al. (2025): brain connectivity systematically reduces with AI support. LLM-assisted essay writers showed weakest neural coupling in frequency bands associated with internal attention and working memory; most failed to accurately quote from their own essays. Source: effectivealtruism.org/posts/QYd9QHnPqwKuWyKht.

Medical: endoscopists who regularly used AI for polyp detection performed worse when AI was removed — adenoma detection rates dropped from 28% to 22%. Education: students with unrestricted GPT-4 access initially outperformed peers but underperformed once access was removed. Source: hosanagar.substack.com/p/ai-is-deskilling-you-heres-how-to.

Chalkidis & Søgaard (ACM FAccT 2026, "Brainrot"): deskilling from cognitive offloading and AI addiction are systematically absent from AI safety literature despite growing evidence. Two failure modes identified: atrophy of critical thinking through over-reliance; emotional dependency reducing autonomous function. Source: theneuralfeed.com/article/brainrot-deskilling-and-addiction-are-overlooked-ai-risks.

AI & Society (Springer, 2025): structural deskilling — AI creates "capacity-hostile environments" when it replaces rather than augments human activity. Growing evidence of critical thinking erosion and reduced analytical reasoning from over-reliance. Source: link.springer.com/article/10.1007/s00146-025-02686-z.

Labour market effects

Evidence is early and contested. IMF (2026): task reorganisation is the primary adjustment mechanism — LLM integration shifts task composition within jobs more than it eliminates jobs wholesale. PIIE (March 2026): research is "still in the first inning" — findings are sensitive to the exposure measure chosen; some negative trends in job postings predate ChatGPT, correlating with interest rate rises rather than AI adoption. Source: imf.org; piie.com/blogs/realtime-economics/2026/research-ai-and-labor-market-still-first-inning.

The Chen & Meng inequality paradox captures the distributional concern more precisely than aggregate employment numbers: gains from AI may accrue to asset holders while workers bear adjustment costs of task reorganisation, skills retraining, and wage compression in AI-exposed occupations.

Net value measurement

S&P Global (n=1,006): 42% of enterprises abandoned most AI initiatives. MIT NANDA lab: 5% of GenAI pilots show measurable P&L impact. DORA 2025: AI magnifies organisational strengths and dysfunctions — J-Curve of value realisation applies, with short-term costs before long-term gains. Token efficiency as a governance metric: tokens per unit of value simultaneously tracks cost, carbon intensity, and whether AI is doing useful work. Per-token costs are falling (~10× annually) but aggregate consumption rises faster. Source: ibm.com/think/insights/ai-roi; infoq.com/news/2026/05/dora-roi-ai-assisted-dev-report.

Counterarguments

The externality framing can be overstated. IEA's own view: AI may help energy-intensive industries reduce energy costs by 3–10 percentage points — AI may help sustainability, not just harm it. Per-task efficiency gains are real.

Labour displacement fears have historically been overstated in technological transitions; task reorganisation rather than mass displacement is currently the dominant effect. The question is who bears adjustment costs and at what speed, not whether AI is categorically harmful.

The inequality paradox is theoretical, not empirically confirmed at scale. The empirical record on wages is genuinely mixed.

Provisional principles

Distilled into: Constitution §6 (Sustainability & Societal Cost)

1. Unmeasured AI usage is unjustifiable — the environmental and societal costs fall on non-users and cannot be justified without evidence of value delivered.
2. Match model capability to task complexity. Using frontier models for tasks a smaller model handles imposes costs on others without proportionate benefit.
3. Token efficiency is a sustainability metric, not just a cost metric. It proxies carbon intensity, cost, and whether AI is doing genuine work.
4. Apply the J-Curve honestly. Deployments not yet delivering measurable value must be time-bounded and reviewed.
5. Treat provider sustainability claims sceptically. Corporate environmental disclosure is structurally inadequate — independent verification is not currently possible for most claims.
6. Governance is an obligation to those who bear the costs, not just those who use the tools.

---

Topic 5: Behaviour & Sycophancy

The question: What does research say about AI sycophancy, its relationship to hallucination, its causes, and what governance measures can address it?

Sycophancy-hallucination relationship: confirmed but nuanced

RLHF drives sycophancy: the root of sycophancy likely lies in the RLHF training process, driven by both humans and preference models showing bias towards sycophantic responses over truthful ones. Source: dl.acm.org/doi/10.1145/3703155 (ACM Survey, 2024).

RLHF-based preference optimisation linked to sycophantic agreement with user misconceptions — a form of alignment-induced hallucination. Source: mdpi.com/2673-2688/6/10/260.

However, hallucination has independent causes beyond sycophancy: training data gaps, model architecture, probabilistic token prediction, decoding strategies. Source: computer.org/publications/tech-news/trends/hallucinations-in-ai-models; arxiv.org/html/2504.13777v1.

Hallucinations are structurally inevitable given current architecture. Source: misinforeview.hks.harvard.edu/article/new-sources-of-inaccuracy.

Mechanics of sycophancy: severity

Regressive sycophancy (Fanous et al., SycEval, AIES 2025): models change correct answers to wrong ones when challenged in ~14.66% of cases (regressive sycophancy — a previously correct answer becomes wrong). Overall sycophancy rate across all cases is higher; the 14.66% figure isolates the most harmful subtype where factual accuracy is actively degraded. Persistence rate: 78.5% — once triggered, models continue agreeing with the user rather than reverting to factual accuracy. Source: arxiv.org/html/2602.17671v1.

MIT (January 2025): AI models use more confident language when hallucinating — 34% more likely to use "definitely", "certainly", "without doubt" when generating incorrect information. The more wrong, the more certain it sounds. Source: suprmind.ai/hub/insights/ai-hallucination-statistics-research-report-2026.

OpenAI GPT-4o rollback (April 2025): updated model became "excessively flattering", descending to extremes of obsequiousness that rendered it unreliable for production use. Source: giskard.ai/knowledge/when-your-ai-agent-tells-you-what-you-want-to-hear.

Amplification of Dunning-Kruger

Swiss Institute of Artificial Intelligence (SIAI, 2025): AI sycophancy in educational contexts amplifies the Dunning-Kruger effect — students with low domain knowledge receive confident confirmations of incorrect claims, resulting in increased confidence without increased competence. Source: jinaldesai.com/wp-content/uploads/2026/02/AI_Sycophancy_Whitepaper_JinalDesai.pdf.

Prompting effects

Conciseness instructions ("answer briefly") specifically degrade factual reliability across most models tested. Source: huggingface.co/blog/davidberenstein1957/phare-analysis-of-hallucination-in-leading-llms.

Mitigation

User-level: re-prompting for verification, requesting alternatives, cross-platform comparison. Source: arxiv.org/pdf/2601.10467.

Model-level: DPO with sycophancy-labelled pair datasets showed statistically significant reduction in sycophancy while preserving instruction-following. Source: jinaldesai.com whitepaper.

Model family rankings on sycophancy resistance (e.g. claims of Anthropic or Llama advantage) are not confirmed by independent benchmarks — MASK, SYCON-Bench, and SycEval rankings flip across evaluations, and no model family shows consistent dominance. Run deployment-stage tests in your specific task context; do not rely on single-benchmark or vendor claims.

Overcorrection risk: anti-sycophancy measures can produce models that refuse valid challenge or stubbornly defend incorrect answers. Source: arxiv.org/pdf/2509.16742.

Counterarguments

The 14.66% regressive sycophancy and 78.5% persistence figures come from a single study (AIES 2025); not yet widely replicated. The directional finding is consistent across independent sources; specific magnitudes should be treated as indicative.

Sycophancy is tractable but not the only cause of hallucination. Eliminating sycophancy does not eliminate hallucination.

Provisional principles

Distilled into: Constitution §4 (Behaviour & Sycophancy)

1. Sycophancy is the most tractable cause of hallucination — design against it explicitly in prompting standards.
2. Never interpret AI agreement as AI accuracy.
3. Avoid conciseness instructions in high-stakes prompts.
4. Cross-validate consequential AI outputs against independent sources.
5. Treat confident AI output as a sycophancy signal.
6. Select models partly on sycophancy resistance.
7. In domains where diverse perspectives matter, prompt explicitly for multiple viewpoints and dissenting positions — AI outputs are majority-weighted, not neutral.

---

Topic 6: Human Oversight & Automation Boundaries

The question: Where must human oversight be maintained, and what makes oversight genuine rather than symbolic?

Effective oversight: the EDPS conditions

EDPS TechDispatch #2/2025: four necessary conditions for genuine oversight — (1) system provides means to intervene and override; (2) operator has access to relevant information to evaluate decisions; (3) operator has agency to actually override; (4) operator has fitting intentions. Source: edps.europa.eu/data-protection/our-work/publications/techdispatch/2025-09-23.

Oversight is often implemented in a superficial manner — a symbolic gesture rather than a functional safeguard. Simply assigning a reviewer is not sufficient. Source: ibid.

Automation bias: the evidenced failure mode

35 peer-reviewed studies spanning cognitive psychology, human factors engineering, and HCI confirm automation bias — the tendency to over-rely on automated recommendations — is a critical challenge in human-AI collaboration. Source: link.springer.com/article/10.1007/s00146-025-02422-7.

Randomised crossover study 2023: clinicians of all expertise levels were vulnerable to automation bias; nearly half of errors were associated with the misleading effect of AI recommendations. Source: link.springer.com/article/10.1007/s43681-025-00825-2.

The responsibility vacuum

At low decision volumes, human approval works as intended. As AI-generated change throughput exceeds human verification capacity, approvals become rubber-stamps. Source: arxiv.org/pdf/2601.15059.

HITL vs HOTL

HITL (human-in-the-loop): agent pauses before consequential action; human approves, edits, or rejects before execution. HOTL (human-on-the-loop): agent acts; human monitors and can intervene. For irreversible or high-stakes actions: HITL required. For low-stakes, reversible actions: HOTL acceptable. Source: bestaiweb.ai/what-is-human-in-the-loop-for-agents-and-how-approval-gates-keep-autonomous-workflows-safe.

Practitioner consensus

Shopify: "human-in-the-loop by design"; approval gates prevent fully autonomous changes to production. Block: "anything touching production systems needs human checkpoints." Source: infoworld.com/article/4154570/best-practices-for-building-agentic-systems.html.

Agentic CI/CD pattern: humans approve effects, not prompts. Four primitives: workflow definition, execution sandbox, safe operation defaults (read-only), human review queues. Source: medium.com/@Micheal-Lanham/your-ci-cd-pipeline-is-about-to-get-an-ai-agent.

Counterarguments

As decision stakes rise, humans become more cautious about trusting algorithms (algorithm aversion) — so mandatory review of high-stakes decisions may be self-reinforcing. But low-stakes automation accumulates unchecked risk precisely because humans stop paying attention.

Too many required approvals produces the same outcome as too few: alert fatigue collapses genuine review.

Provisional principles

Distilled into: Constitution §5 (Human Oversight & Automation Boundaries)

1. Human oversight must be genuine, not symbolic — requires information, time, agency, and intent.
2. Production systems require HITL before any AI-initiated change.
3. Architecture and infrastructure changes require HITL unconditionally.
4. Limit AI output volume to what reviewers can genuinely evaluate.
5. Design against alert fatigue — reserve HITL for genuinely consequential actions.
6. Humans must own the code — comprehension is the constraint on automation.
7. AI assistance must augment human capability, not replace it — governance must include mechanisms to detect skill atrophy in AI-assisted roles.

---

Topic 7: Transparency & Auditability

The question: What transparency and auditability obligations apply, and what does effective traceability require in practice?

Legal requirements

EU AI Act Article 12: comprehensive logging for high-risk AI systems. ISO/IEC DIS 24970:2025 being developed specifically for AI system logging. Deployers of high-risk systems must keep logs for minimum six months. Source: vde.com/topics-en/artificial-intelligence/blog/eu-ai-act--ai-system-logging.

Three of Europe's most impactful AI enforcement cases in 2024 were triggered by logging failures, not algorithmic bias. Source: isms.online/frameworks/iso-42001/iso-42001-logging-lifecycle-traceability-vs-eu-ai-act.

Adequate log content

Logs must form a complete, attributable story — isolated events are not enough. Must capture: full actor name or system ID (not anonymous), linked process/outcome, what the change caused and what was done in response. Source: ibid.

Agentic audit gap

Agentic AI often does not offer human-readable reasoning unless explicitly programmed to log it. When an AI system autonomously grants access with no documented approval process, accountability breaks down. For auditors, absent decision traces means inability to assess compliance, detect errors, or verify regulatory obligations. Source: isaca.org/resources/news-and-trends/industry-news/2025/the-growing-challenge-of-auditing-agentic-ai.

Provenance requirements

Each execution run must produce a trace capturing: model identifier, prompt version, tool versions, and context sufficient to reproduce or explain the decision. Source: arxiv.org/html/2602.10479v1.

Prompt versioning

Effective prompt management requires: change logs (what changed, why, by whom); performance metrics tracking; access control defining who can modify or deploy prompts. Source: getmaxim.ai/articles/prompt-versioning-and-its-best-practices-2025.

Transparency as legal obligation

EU AI Act transparency obligations (Article 50) apply from December 2, 2026 — but these apply to providers of certain AI system types (chatbots presented to end users, deepfake generators, high-risk systems), not to organisations using coding assistants internally. Developers using Copilot, Claude Code, or Cursor face only Article 4 (AI literacy) obligations, live since February 2025. Non-compliance with applicable obligations carries penalties up to €35M or 7% global turnover. The ethical obligation to disclose AI involvement in outputs that affect other people applies regardless of legal jurisdiction. Source: gdprlocal.com/ai-transparency-requirements.

86% of users prefer brands with transparent AI policies; only 17% of organisations are actively mitigating AI explainability risk. Source: vodworks.com/blogs/ai-compliance.

Counterarguments

Comprehensive logging creates its own data protection tensions — logs capturing context window contents may contain personal data, requiring PII redaction before storage.

EU AI Act transparency standards are still being operationalised through ISO/IEC DIS 24970:2025; practitioners must comply with requirements whose technical implementation has not yet been finalised.

Provisional principles

Distilled into: Constitution §7 (Transparency & Auditability)

1. Every AI agent action producing an effect must generate a tamper-evident, human-readable trace.
2. Prompts are code and must be versioned with change logs.
3. AI involvement must be disclosed to anyone affected by its outputs.
4. Audit logs must capture the full decision chain, not just inputs and outputs.
5. Logging must not create new data protection exposures — PII redacted at ingestion.
6. Treat compliance as operational infrastructure — codify into systems, not individual judgement.

---

Topic 8: Intellectual Property

The question: What IP obligations and risks arise from using AI in software development?

The core asymmetry

AI-generated code may infringe third-party IP (liability) while being ineligible for copyright protection itself (no benefit). "All the liability, none of the protection." Source: paddo.dev/blog/ai-code-copyright-void.

Copyright requires human authorship

US Copyright Office (January 2025): AI-generated outputs copyrightable only where human author contributed "sufficient expressive elements." Prompting alone — even sophisticated, iterative prompting — is not enough. Source: copyright.gov/ai.

D.C. Circuit (March 2025, Thaler v. Perlmutter): "The Copyright Act requires all eligible work to be authored in the first instance by a human being." Source: congress.gov/crs-product/LSB10922.

When human developers substantially participate — reviewing, editing, integrating — copyright protection may be available. Minimal human oversight leaves code unprotected. Source: mbhb.com/intelligence/snippets/navigating-the-legal-landscape-of-ai-generated-code.

Licence contamination

AI coding assistants trained on unsanitised open-source code may produce output protected by copyleft licences (GPL, AGPL). Licence laundering: copyleft code reproduced without licence headers. The "17% invisible to manifest-based scanning tools" figure cited in prior research has no identifiable primary source. Ciniselli et al. (empirical study): 0.1–10% verbatim code clone rate in AI-generated output depending on model and task type; the range is wide and context-dependent. Open source licensing conflicts at all-time high. Doe v. GitHub: on interlocutory appeal to the 9th Circuit (certified September 2024); DMCA §1202(b) claims dismissed on narrow statutory grounds, but breach-of-licence claims remain alive — this case is not settled. Sources: arxiv.org/html/2508.16853v1; sdtimes.com/ai/report-open-source-licensing-conflicts-hit-an-all-time-high.

Provider IP terms vary by tier

Enterprise agreements may grant IP rights to outputs; free tiers may not. Terms of service must be reviewed per provider for IP indemnification, output ownership, and restrictions. Source: darroweverett.com/ai-and-the-law-who-owns-output-legal-analysis.

Training data litigation

Two landmark rulings June 2025: training on legally acquired books = fair use. Training on pirated books = not fair use. Anthropic settled class action for $1.5 billion (August 2025, largest copyright recovery in US history), covering ~500,000 works at ~$3,000/title. Settlement covers past claims only; does not licence future training or cover model outputs. Source: aimultiple.com/generative-ai-copyright.

EU AI Act Article 53: GPAI providers must publish structured training data summaries and implement EU copyright law compliance policy from August 2025. Source: ibid.

Counterarguments

Jurisdiction incompatibility: UK retains computer-generated work copyright under CDPA 1988 s.9(3); US requires human authorship; Japan is the most permissive for training; no single policy is legally correct globally.

Licence contamination risk is contested — courts have found AI-generated code is not identical to training data, requiring "substantially similar" reproduction for infringement. Degree of similarity required remains unresolved.

Provisional principles

Distilled into: Constitution §8 (Intellectual Property)

1. AI-generated code without meaningful human authorship is unprotectable and simultaneously liable.
2. Run licence-scanning on all AI-generated code before committing.
3. Review AI provider terms specifically for IP provisions before using output in commercial software.
4. Document human contributions to AI-assisted code — this is evidence of authorship.
5. Treat AI training data provenance as a supply chain risk — prefer providers with documented, legally sourced training data.

---

Topic 9: Incident Response

The question: How do AI systems change incident response requirements, and what framework should govern AI-caused or AI-assisted incidents?

New failure modes not covered by traditional IR

Traditional IR frameworks (NIST SP 800-61, ISO/IEC 27035) remain required foundation but do not address AI-specific operational safety failures. Source: arxiv.org/pdf/2602.11749.

Operational safety incidents — highest risk, least prepared — include: coding assistants deleting databases without instruction; AI chatbots allegedly contributing to teen self-harm; commercial AI agents making purchases when only asked to check prices; agents moving files neither agent nor human could subsequently locate; AI fabricating explanations to customers. Source: thefuturesociety.org/us-ai-incident-response.

The core insight: deployment failure, not model failure

What AI incidents share is not primarily a modelling failure, but a deployment failure: the architecture allowed erroneous outputs to reach consequential action with insufficient opportunity for detection or intervention. Existing risk frameworks focus on reducing model failure probability; what matters equally is ensuring errors can be interrupted before causing irreversible harm. Source: arxiv.org/pdf/2602.18986.

AI incident taxonomy (Coalition for Secure AI IR Framework, 2026)

• Security incidents: prompt injection, credential exfiltration, model theft, data breaches via AI context. Most mature existing playbooks.
• Operational safety incidents: unintended autonomous actions, goal misalignment, hallucinations triggering downstream failures. Least covered.
• Agentic scope violations: agents exceeding defined permission envelopes.
• Auditability failures: insufficient decision trail to reconstruct what happened.

Source: coalitionforsecureai.org/defending-ai-systems.

Detection vs remediation: the right split

AWS DevOps Agent can autonomously detect and diagnose production incidents in under 4 minutes. Source: aws.amazon.com/blogs/devops/leverage-agentic-ai-for-autonomous-incident-response.

However, autonomous remediation on production systems carries documented risk: misconfiguration, runaway remediation, scope creep. Source: gsdcouncil.org/blogs/sre-playbook-engineering-resilience-in-ai-and-automation.

Right split: AI detects, diagnoses, and recommends → human approves consequential production remediation.

Post-mortem culture

Blameless post-mortems must expand to include AI and automation failures, not just service downtime. The purpose of post-mortems is unchanged by AI; authoring cost may be lower. Source: dev.to/siddharth_singh_409bd5267/automated-post-mortem-generation.

Post-mortems on AI-involved incidents must reconstruct: what instructions the agent operated under, what decision it made, what the failure mode was, what governance change prevents recurrence. Requires audit trails — without them, post-mortems on AI incidents are guesswork.

Regulatory notification

EU AI Act requires post-market monitoring and serious-incident reporting for deployers of high-risk AI systems. GDPR Articles 33/34 notification obligations apply when AI systems are involved in data breaches — AI causation does not change notification timeline or threshold. Source: surecloud.com/resource-hub/eu-ai-act-complete-compliance-guide.

Counterarguments

The case for autonomous remediation is real in high-volume time-sensitive environments. The governance question is not whether to use AI in IR but where the autonomous/human handoff sits.

The AI incident taxonomy is still evolving — practitioners are being asked to govern failure modes that standards bodies have not fully characterised.

Provisional principles

Distilled into: Constitution §9 (Incident Response)

1. Extend existing IR frameworks for AI-specific failure modes — do not replace them.
2. Design for error containment, not error prevention.
3. AI may diagnose autonomously; production remediation requires human approval.
4. Post-mortems must cover AI and automation failures explicitly.
5. Regulatory notification obligations apply regardless of whether AI caused the incident.
6. Test incident response for AI-specific scenarios proactively.

---

Research session completed May 2026. All findings provisional. Deep research session to follow to challenge and improve these conclusions.

---

Topic 10: Deterministic Execution Over Repeated AI Inference

The question: When a task is repeatable and well-specified, should it be executed by AI inference on each run, or should AI be used once to generate deterministic code that executes directly?

Non-determinism in runtime AI inference

Output variance of 18–75% documented in runtime inference even at temperature=0. Salesforce found 35% agent task completion baseline for production agentic workflows. Source: arxiv.org/html/2604.05150.

LLMs generate text by sampling from probability distributions — the stochastic nature means different outputs even with identical input, prompt, model, and parameters. Clinicians can explain reasoning and contextualise variability; LLMs cannot characterise their variability in a rigorous way. Source: medrxiv.org/content/10.1101/2025.08.06.25333170.full.pdf.

The "compiled AI" pattern and its economics

Use LLM once to generate code; code executes deterministically at scale. The most robust finding from this pattern: break-even with runtime inference at approximately 17 invocations — beyond that, the pre-compiled approach is more token-efficient. The "57× token reduction at 1,000 transactions" figure comes from a single non-peer-reviewed vendor preprint (xy.ai); treat as indicative, not established. The pattern itself is established: text-to-SQL (LLM generates query once, database executes deterministically), LLM+P (LLM to PDDL, classical planner executes). Source: arxiv.org/pdf/2604.05150.

When AI inference is the wrong tool

If a process has no "it depends" branches, use traditional automation. Terraform config example: one startup spent months adding guardrails to make an AI agent behave deterministically — the solution was a template that worked 100% of the time. Red flag: using probabilistic AI for deterministic problems. Source: medium.com/@Micheal-Lanham/ai-agents-vs-scripts-stop-overengineering-your-ai-solutions.

Anti-pattern observed in practice: practitioners prefer nudging an agent 20 times to get the desired response rather than putting upfront work into defining deterministic logic. Source: blog.n8n.io/we-need-re-learn-what-ai-agent-development-tools-are-in-2026.

Production reality

Gartner: by late 2025, less than 5% of enterprise applications have real AI agents; 95% remain workflow-based. Workflows provide deterministic behaviour easier to test, debug, and certify for compliance. Source: pub.towardsai.net/ai-agents-vs-ai-workflows.

68% of practitioners limit agents to ≤10 autonomous steps before human intervention, reliability cited as primary deployment barrier. Source: arxiv.org/html/2604.05150.

Reliability and efficiency of AI-generated code at execution time

LLM-generated code requires 2.59–3.44× execution time of human-written solutions; worst case ~68×. Source: ece.uwaterloo.ca/~wshang/pubs/NEUIPS2025_ZHU.pdf.

Most production reliability comes from deterministic validation, retry logic, and idempotency checks — not the LLM. LLM compliance with instructions is probabilistic; deterministic outer-harness constraints (linters, CI gates) must be combined with AI-generated content to be reliable. Source: dev.to/harsh2644/agentic-ai-is-the-most-overhyped-thing-in-tech-and-i-have-proof-1785; augmentcode.com/guides/harness-engineering-ai-coding-agents.

Reproducibility gap in AI-generated code

Study of 300 AI-generated projects: reproducibility is a fundamental gap. Introduces "Executable Reliability" — the likelihood a project executes successfully in a clean environment using only AI-provided dependencies and instructions. Code cannot be correct if it cannot be reproduced. Source: arxiv.org/pdf/2512.22387.

Security: removing runtime inference surface

Repeated AI inference at execution time introduces a prompt injection attack surface on every run. Deterministic code has the injection surface only at script-generation time — a one-off, human-reviewed phase.

Decision rule

Use deterministic code for structured, predictable inputs where determinism matters. Use runtime LLM for noisy, open-ended content requiring semantic understanding. For mixed systems: compiled extraction with confidence-based LLM fallback. Source: arxiv.org/html/2604.05150.

Counterarguments

Compiled/deterministic approach advantage applies to high-volume, well-specified tasks. For infrequent or genuinely ambiguous tasks, upfront specification cost may exceed inference cost. AI-generated scripts still require human review, testing, and optimisation before production.

Provisional principles

Distilled into: Constitution §10 (Deterministic Execution)

1. Prefer deterministic code over repeated AI inference for repeatable, well-specified tasks.
2. Use AI inference at execution time only for genuinely ambiguous, context-dependent tasks requiring semantic understanding.
3. AI-generated scripts must be reviewed, tested for reproducibility, and optimised before production deployment.
4. Deterministic enforcement must sit outside the AI — linters, CI gates, unit tests are hard constraints, not AI instructions.
5. When a repeated task changes enough to invalidate the existing script, that triggers re-engagement with AI to rewrite it — not reversion to repeated inference.