176 lines
14 KiB
Markdown
176 lines
14 KiB
Markdown
# AI Governance — Open Questions & Research Challenges
|
||
|
||
**Purpose:** Agenda for the deep research session. Each item identifies a hypothesis, assumption, or finding from the initial research that warrants deeper investigation, challenge, or quantification. Ordered by governance impact — highest-stakes uncertainties first.
|
||
|
||
---
|
||
|
||
## 1. Sycophancy as root cause of hallucination — how strong is the causal link?
|
||
|
||
**Current position:** Sycophancy is the most tractable cause of hallucination. RLHF systematically induces approval-seeking behaviour that overrides accuracy.
|
||
|
||
**What to challenge:**
|
||
- The 59% regressive sycophancy rate and 78.5% persistence rate come from single studies (Fanous et al., SycEval, AAAI 2025). Do independent replications confirm these magnitudes?
|
||
- Is sycophancy genuinely a *cause* of hallucination, or do both share a common cause (RLHF misalignment) without a direct causal relationship?
|
||
- What is the relative contribution of sycophancy vs. training data gaps vs. decoding strategies to observed hallucination rates in software development contexts specifically?
|
||
- Are there prompting or configuration approaches with replicated evidence — not single-study findings — that reliably reduce sycophancy in production?
|
||
|
||
**Governance implication:** If sycophancy is one cause among several of roughly equal weight, the constitution's emphasis on designing against it first may be correctly placed but insufficiently supported.
|
||
|
||
---
|
||
|
||
## 2. AI as net sustainability negative — is this actually measurable?
|
||
|
||
**Current position:** 72% of AI investments destroy value through waste; AI is currently a net negative because ungoverned and not optimised for genuine value.
|
||
|
||
**What to challenge:**
|
||
- The 72% figure is from a single proprietary Larridin report — not peer-reviewed. What do independent measurements show?
|
||
- The METR study (19% slower) covers experienced open-source developers using early-2025 tools on open-source tasks. Does this generalise to professional software teams using current tools on commercial work?
|
||
- What rigorous measurement frameworks exist for AI ROI that are not vendor-published? DORA is the most credible — what does the full 2025 DORA dataset show beyond the summary?
|
||
- What would "net positive" look like in measurable terms? Is there a validated metric set (beyond token efficiency) that constitutes evidence of genuine value delivery?
|
||
|
||
**Governance implication:** The J-Curve principle and the time-bounded review requirement depend on being able to measure value. If measurement frameworks are genuinely inadequate, the principle may need to be restructured.
|
||
|
||
---
|
||
|
||
## 3. Human code ownership under AI assistance — where is the line?
|
||
|
||
**Current position:** Humans must be able to understand, audit, and manually override AI-generated code. AI assistance that creates comprehension dependency is an ethical failure.
|
||
|
||
**What to challenge:**
|
||
- The deskilling literature is largely from non-software contexts (manufacturing, aviation). Is there specific evidence of deskilling in software engineering from AI assistance?
|
||
- Is comprehension of every line the right standard, or is there a meaningful distinction between understanding architecture/intent vs. understanding every implementation detail?
|
||
- METR's finding (19% slower) may reflect tool immaturity rather than a structural principle. What does the evidence show as tools mature?
|
||
- What is the minimum viable human understanding of AI-generated code that is sufficient for responsible ownership?
|
||
|
||
**Governance implication:** If the comprehension standard is set too high, it may prohibit beneficial automation. If too low, it allows accountability laundering.
|
||
|
||
---
|
||
|
||
## 4. EU AI Act applicability to software developers — what actually applies?
|
||
|
||
**Current position:** EU AI Act obligations are treated as broadly applicable from August 2026.
|
||
|
||
**What to challenge:**
|
||
- Most AI-assisted software development does not involve "high-risk AI systems" as defined in Annex III of the Act. What risk tier does typical software/infra tooling (Copilot, Claude Code, Cursor) actually fall into?
|
||
- The Act primarily regulates providers and deployers of AI systems, not necessarily users of AI coding tools. What specific obligations fall on a developer using an AI coding assistant vs. deploying an AI-powered product?
|
||
- How does the research activity exemption (Article 2(6)) affect developers using AI in pre-production contexts?
|
||
- What does "significant modification" of a GPAI model mean in practice — does fine-tuning a model for your codebase trigger provider-level obligations?
|
||
|
||
**Governance implication:** If current tooling falls into minimal-risk categories, some constitution principles may be disproportionate for the actual regulatory requirement. Alternatively, obligations may be more specific and actionable than currently stated.
|
||
|
||
---
|
||
|
||
## 5. Data classification in AI context — are four tiers sufficient?
|
||
|
||
**Current position:** Four tiers (Public / Internal / Confidential / Restricted) mirroring ISO 27001, with AI-specific handling rules per tier.
|
||
|
||
**What to challenge:**
|
||
- The four-tier model was designed for file and database classification, not for AI context windows. Are there AI-specific classification schemes from the EDPB, CNIL, or ISO that go further?
|
||
- How should AI-generated content itself be classified? (A document produced by AI from Confidential inputs — what tier is the output?)
|
||
- How does the composite sensitivity problem (non-sensitive data combining to reveal sensitive attributes) get operationalised in a classification framework? Is the four-tier model capable of handling this?
|
||
- What open-source tooling exists for context window scanning and real-time classification? The constitution references "safe open-source options" but does not specify them.
|
||
|
||
**Governance implication:** A classification framework that cannot be operationalised with available tooling is ethics washing.
|
||
|
||
---
|
||
|
||
## 6. Open source licence contamination — how severe is the practical risk?
|
||
|
||
**Current position:** Licence contamination is a documented, growing risk; 17% of open source components enter codebases via AI generation invisible to standard scanning tools.
|
||
|
||
**What to challenge:**
|
||
- The 2024 ruling that dismissed claims found AI-generated code is not "substantially similar" to training code. Is the contamination risk overstated — is it more theoretical than practical for most generated output?
|
||
- What is the actual rate of verbatim or near-verbatim GPL/AGPL reproduction in AI coding assistant output? Are there studies with methodology that can be evaluated?
|
||
- What are the current open-source scanning tools that are effective for AI-generated code specifically, and what are their false positive/negative rates?
|
||
- Is the risk primarily in snippets and algorithms, or does it extend to overall code structure and design patterns?
|
||
|
||
**Governance implication:** Overstating this risk imposes unnecessary cost; understating it creates legal exposure. The constitution needs a calibrated, evidenced position.
|
||
|
||
---
|
||
|
||
## 7. Incident response for operational safety — what playbooks actually exist?
|
||
|
||
**Current position:** Operational safety failures (unintended autonomous actions, goal misalignment) are the highest-risk category and the least prepared for. Extend existing IR frameworks.
|
||
|
||
**What to challenge:**
|
||
- The Coalition for Secure AI AI Incident Response Framework (2026) is referenced but not evaluated in detail. What does it actually recommend for operational safety incidents specifically?
|
||
- What is the NIST AI Risk Management Framework (AI RMF 1.0) response framework — how does it handle incidents caused by agentic systems?
|
||
- Are there documented case studies of organisations that have successfully handled AI operational safety incidents? What made their response effective?
|
||
- For software/infra contexts specifically: what triggers should initiate an AI incident response, and at what point does an AI misbehaviour become a notifiable incident?
|
||
|
||
**Governance implication:** A principle that says "extend existing IR frameworks" without specifying what to add is incomplete.
|
||
|
||
---
|
||
|
||
## 8. Transparency disclosure requirements — what is the minimum viable implementation?
|
||
|
||
**Current position:** AI involvement must be disclosed; this is a legal obligation from August 2026 under the EU AI Act.
|
||
|
||
**What to challenge:**
|
||
- The EU AI Act transparency provisions in Articles 13/50 apply to specific system types (chatbots, deepfake generators, high-risk systems). Do they apply to internal AI-assisted development tooling where end users are not interacting with AI directly?
|
||
- What does "disclosure" actually require in the context of AI-assisted code review, automated deployment, or AI-generated documentation?
|
||
- Are there jurisdiction-specific requirements beyond the EU AI Act (UK, US state laws) that are more or less demanding?
|
||
- What is the current state of implementation: who is actually compliant with existing transparency requirements, and what enforcement actions have occurred?
|
||
|
||
**Governance implication:** A disclosure requirement that is vague about what to disclose, to whom, and in what form cannot be implemented.
|
||
|
||
---
|
||
|
||
## 9. Model selection criteria — what does sycophancy resistance actually look like?
|
||
|
||
**Current position:** Select models partly on sycophancy resistance; some models (Anthropic, large Llama) show more resistance.
|
||
|
||
**What to challenge:**
|
||
- What benchmarks exist specifically for sycophancy resistance, and how methodologically rigorous are they? (SycEval from AAAI 2025 is referenced — evaluate it.)
|
||
- Are sycophancy benchmarks stable across prompt variations, or do models that perform well on benchmarks still exhibit sycophancy in production contexts?
|
||
- How does model selection for sycophancy resistance trade off against other factors (cost, capability, data protection, provider terms)?
|
||
- Is the Anthropic/Llama advantage in sycophancy resistance confirmed by independent evaluations, or is it based primarily on Anthropic's own research?
|
||
|
||
**Governance implication:** If sycophancy benchmarks are not reliable, "select models on sycophancy resistance" is not actionable.
|
||
|
||
---
|
||
|
||
## Deep Research Session Objectives
|
||
|
||
1. Replicate or refute the key quantitative claims from the initial research against additional independent sources.
|
||
2. Identify specific, actionable tooling and implementation guidance for each principle — particularly data classification scanning, licence scanning, audit logging, and sycophancy evaluation.
|
||
3. Resolve the EU AI Act applicability question for the specific context of software development tooling.
|
||
4. Find and evaluate the Coalition for Secure AI IR Framework and NIST AI RMF in detail.
|
||
5. Assess whether the constitution contains any principles that are unsupported, disproportionate, or operationally unachievable given current tooling.
|
||
6. Identify any significant governance topics the initial research missed.
|
||
|
||
---
|
||
|
||
*Prepared after initial research session, May 2026. Upload alongside `ai-governance-research-session.md` to continue.*
|
||
|
||
---
|
||
|
||
## 10. Deterministic execution preference — where exactly is the decision boundary?
|
||
|
||
**Current position:** Prefer deterministic code over repeated AI inference for well-specified, repeatable tasks. Use the "compile once, execute many" pattern.
|
||
|
||
**What to challenge:**
|
||
- The 57× token reduction claim comes from the xy.ai "Compiled AI" paper, a startup with commercial interest in this pattern. Do independent studies confirm similar efficiency gains?
|
||
- What is the practical definition of "well-specified enough" to codify as a script? Is there a decision framework with clear criteria that has been validated in practice?
|
||
- The 2.5–68× execution time overhead for AI-generated code — is this a property of AI-generated code in general, or specifically of unreviewed/unoptimised AI output? Does human review close this gap?
|
||
- How does this principle interact with skills/workflows in the repo? A SKILL.md that instructs an AI agent to perform a task is itself a form of "codification" — is this deterministic enough, or does it still require a traditional script?
|
||
- Are there task categories in software dev and infra where the AI-inference-at-execution-time approach is clearly superior even for repeated tasks (e.g., code review comments, PR descriptions)?
|
||
|
||
**Governance implication:** The decision boundary between "use a script" and "use AI inference" needs to be specific enough to be actionable. A vague principle is not useful.
|
||
|
||
---
|
||
|
||
## 11. Agent instruction reliability — how much can AGENTS.md actually enforce?
|
||
|
||
**Current position:** AGENTS.md operationalises the constitution's agent-actionable principles. Hard prohibitions (Never rules) are the most reliably followed; required behaviours are aspirational and improve compliance without guaranteeing it.
|
||
|
||
**What to challenge:**
|
||
- AGENTIF benchmarking found the best model perfectly follows fewer than 30% of complex agentic instructions. Is this the right benchmark for our use case (simpler, more direct instructions vs. complex multi-constraint agentic tasks)?
|
||
- The 500-line limit finding comes from practitioner observation, not controlled research. Is there empirical evidence on the optimal length and structure for instruction files?
|
||
- Which specific instruction types in AGENTS.md are most likely to be violated? Are the hard prohibitions actually hard for current models, or does prompt injection still bypass them?
|
||
- How should AGENTS.md be tested and validated? What would a compliance eval for these instructions look like?
|
||
- The thin adapter pattern (AGENTS.md as single source, tool-specific files reference it) — does this actually work across Copilot, Claude Code, and Cursor, or do tool-specific constraints require substantive adaptation rather than thin references?
|
||
- Should AGENTS.md include few-shot examples (what to do vs. what not to do) to improve compliance? Research suggests agents learn implicitly from examples even within a single prompt.
|
||
|
||
**Governance implication:** If agent instruction compliance is fundamentally limited, the constitution's principle that "deterministic enforcement must sit outside the AI" is not just a best practice — it is the primary enforcement mechanism, and AGENTS.md is a supplementary signal at best. This changes how we think about operationalising the other principles.
|