2. Why Governance Misses It¶
After this module you will be able to¶
- Identify the specific gaps in NIST AI RMF, ISO 42001, and the EU AI Act when applied to multi-agent systems
- Explain why "having a governance framework" is not the same as "being able to govern agent chains"
- Describe the common assumption in current frameworks that breaks when agents delegate to agents
- Articulate what an adequate governance framework for multi-agent systems would need to include
The governance paradox¶
Meridian Capital was not ungoverned. They had an AI governance framework. They had policies, risk assessments, monitoring, and audit procedures. If you asked them "do you govern your AI systems?" they would say yes, and they would be telling the truth.
The problem is not the absence of governance. It is the scope of governance. Every major AI governance framework in use today shares a common structural assumption:
The single-agent assumption: Current AI governance frameworks assume that the unit of governance is the individual AI system: a model, an application, a deployment. They provide comprehensive guidance for governing one system at a time. They do not provide guidance for governing the interactions between systems, the delegation of tasks between agents, or the integrity of reasoning chains that span multiple agents.
This is not a criticism of these frameworks. They were designed before multi-agent orchestration became a common deployment pattern. But if your organisation is deploying agent chains and governing them with single-agent frameworks, you have a structural gap.
Let us walk through the three most widely adopted frameworks and identify specifically where each one falls short.
NIST AI Risk Management Framework (AI RMF)¶
The NIST AI RMF (published January 2023) provides a comprehensive approach to AI risk management organised around four core functions: Govern, Map, Measure, and Manage.
What it covers well¶
- Risk identification: The Map function provides thorough guidance on identifying AI risks, including risks from data quality, model limitations, and deployment context.
- Governance structure: The Govern function establishes clear expectations for organisational accountability, policies, and oversight.
- Measurement: The Measure function addresses testing, evaluation, verification, and validation.
- Stakeholder engagement: The framework emphasises understanding impacts on people and communities.
Where it falls short on agent chains¶
Gap 1: No concept of inter-agent trust.
The AI RMF discusses trust extensively, but it means trust between humans and AI systems. It does not address trust between AI agents. When Agent C trusts Agent B's compliance assessment, that is an inter-agent trust relationship that the AI RMF does not have vocabulary for. The Map function asks you to characterise the AI system's interactions with users and affected parties. It does not ask you to characterise the AI system's interactions with other AI systems.
Gap 2: Risk is scoped to individual AI systems.
The Map function (MAP 1.1) asks organisations to define the AI system's purpose, context, and boundaries. For a multi-agent pipeline, this creates a scoping question: is the pipeline one system or three? If it is one system, the internal agent interactions are implementation details that the risk assessment may not examine. If it is three systems, each system's risk assessment misses the chain-level risks that only emerge from their interaction.
Gap 3: Measurement does not address chain integrity.
The Measure function (MEASURE 2) addresses testing and evaluation of AI systems. But the testing paradigm assumes you can define inputs and expected outputs for the system under test. In a multi-agent chain, the failure mode is not "wrong output for given input" but rather "correct output from degraded reasoning that is invisible at the input/output boundary."
If Meridian Capital completed an AI RMF-aligned risk assessment for their compliance pipeline, they would have:
- Mapped the system's purpose (trade pre-clearance) and context (financial services, regulatory compliance)
- Governed it with policies and an accountable person
- Measured it with testing, likely end-to-end tests with known inputs and expected outputs
- Managed the identified risks with guardrails and monitoring
All of this is valuable. None of it would have identified the specific risk that Agent B might operate on truncated data and that Agent C would accept the result without verification. The risk lives in the inter-agent interaction, and the AI RMF does not direct you to look there.
ISO/IEC 42001: AI Management System¶
ISO 42001 (published December 2023) is the international standard for AI management systems. It provides a systematic approach to managing AI risks using the familiar Plan-Do-Check-Act cycle from other ISO management system standards.
What it covers well¶
- Management system approach: Provides a mature, auditable framework for AI governance that integrates with existing management systems (ISO 27001, ISO 9001, etc.)
- Risk assessment: Requires systematic risk assessment and treatment for AI systems.
- Stakeholder requirements: Addresses regulatory, contractual, and organisational requirements.
- Continual improvement: Built-in mechanisms for learning and improving governance over time.
Where it falls short on agent chains¶
Gap 1: The AI system boundary is not defined for chains.
ISO 42001 requires organisations to define the scope of their AI management system (Clause 4.3), including the AI systems covered. For multi-agent systems, this creates the same boundary question as the AI RMF: where does one AI system end and another begin? ISO 42001 does not provide guidance on this question for agent orchestration.
Gap 2: Risk assessment does not address emergent chain behaviour.
The risk assessment process (Clause 6.1) asks organisations to identify risks and opportunities related to their AI systems. The risk categories provided in Annex B are comprehensive for individual AI systems but do not include risks that emerge from agent-to-agent delegation, reasoning-chain degradation, or confidence laundering.
Gap 3: Controls address deployment, not runtime interactions.
Annex A provides a control set for AI management. These controls address model development, data management, deployment, and monitoring, all of which are essential. But they do not address runtime inter-agent integrity: verifying that agent A's output is suitable as agent B's input, that delegation is authorised and bounded, or that reasoning-chain integrity is maintained across agents.
Gap 4: Audit criteria do not cover chain-level properties.
ISO 42001 supports internal audit (Clause 9.2) and management review (Clause 9.3). But if the management system does not define chain-level integrity as a requirement, auditors will not test for it. You cannot audit what you have not defined.
EU AI Act¶
The EU AI Act (entered into force August 2024, with phased implementation) is the most significant AI-specific regulation globally. It establishes a risk-based classification system and imposes obligations on providers and deployers of AI systems.
What it covers well¶
- Risk classification: A clear four-tier risk classification system (unacceptable, high-risk, limited risk, minimal risk) that determines the level of regulatory obligation.
- High-risk requirements: Comprehensive requirements for high-risk AI systems covering data governance, transparency, human oversight, accuracy, robustness, and cybersecurity.
- Provider and deployer obligations: Clear allocation of responsibility between AI system providers and deployers.
- Conformity assessment: A structured process for demonstrating compliance before deployment.
Where it falls short on agent chains¶
Gap 1: The definition of "AI system" is agent-singular.
Article 3(1) defines an AI system as "a machine-based system that ... generates outputs such as predictions, content, recommendations, or decisions." The definition is scoped to a single system generating outputs. A multi-agent pipeline is not a single system generating outputs; it is multiple systems, each generating outputs that serve as inputs to the next. The Act does not clearly define how to classify or govern the chain as a whole.
Gap 2: Provider/deployer obligations assume a clear boundary.
The Act assigns obligations to "providers" (who develop the AI system) and "deployers" (who use it). For a multi-agent pipeline built by an organisation using components from multiple vendors, who is the provider and who is the deployer? If the orchestration layer is built in-house but the underlying models come from external providers, the obligation boundary is unclear for chain-level failures.
Gap 3: Human oversight (Article 14) assumes a human can see the reasoning.
The high-risk system requirements include human oversight: the ability for a human to understand the AI system's outputs and to intervene when necessary. But human oversight is only effective if the human can access the reasoning basis, not just the output. In Phantom Compliance, a human reviewer would see Agent C's well-formatted approval recommendation. Without access to Agent B's retrieval metadata, the human reviewer would approve the same trade the system approved.
Gap 4: Conformity assessment is point-in-time, not runtime.
The conformity assessment process evaluates AI systems before deployment. But multi-agent failures like Phantom Compliance are runtime failures that emerge from conditions (context window pressure, data volume changes, retrieval variability) that may not exist at assessment time. A system that passes conformity assessment can still fail in operation if the runtime conditions cause inter-agent integrity degradation.
Meridian Capital's compliance pipeline would likely be classified as high-risk under the EU AI Act (it makes or assists decisions in a regulated financial services context).
As a high-risk system, it would be subject to:
- Data governance requirements (Article 10), which would address the quality of training data but not the completeness of runtime retrieval
- Technical documentation (Article 11), which would document the pipeline architecture but not define runtime inter-agent integrity requirements
- Record-keeping (Article 12), which would require logs, and Meridian had logs; the logs just did not capture the right information
- Transparency (Article 13), which would require the deployer to understand the system, but understanding the system does not mean understanding every runtime interaction
- Human oversight (Article 14), which would require human oversight capability, but not the specific metadata access needed to verify reasoning-chain integrity
The Act would require Meridian to have governance. It would not require the specific type of governance (chain-level reasoning integrity verification) that would have caught this failure.
The common gap¶
All three frameworks share a structural assumption that creates the same gap:
| Assumption | Reality for agent chains |
|---|---|
| AI governance is about governing individual systems | Multi-agent failures emerge from interactions, not from individual systems |
| If each component is governed, the system is governed | Governed components can produce ungoverned interactions |
| Monitoring proves operation | Monitoring proves execution; it does not prove reasoning integrity |
| Risk assessment at deployment covers runtime risk | Runtime conditions create risks that did not exist at deployment |
| Human oversight means a human can see the output | Effective oversight requires seeing the reasoning basis, not just the output |
The governance gap in one sentence: Current frameworks govern the agents. Nobody governs the chain.
This is not a failure of the frameworks; it is a scope limitation. The frameworks were designed for a world where the unit of AI governance was a single system. Multi-agent orchestration requires extending these frameworks to cover the chain as a governance object.
From gap to extension¶
The good news is that you do not need to discard your existing governance frameworks. You need to extend them. The extensions are specific and tractable:
-
Scope extension: Define agent chains as governance objects, not just individual agents. This means your risk assessment, your controls, and your audit criteria must cover chain-level properties.
-
Accountability extension: Define who is accountable when agents delegate to agents. This means your accountability framework must address inter-agent delegation, not just human-to-system delegation.
-
Evidence extension: Define what evidence demonstrates chain-level integrity, not just component-level compliance. This means your monitoring and audit must capture reasoning-basis metadata, not just inputs and outputs.
-
Runtime extension: Extend point-in-time assessments with runtime integrity verification. This means your conformity assessment must include ongoing runtime checks, not just pre-deployment evaluation.
The MASO framework, which we will cover in Module 4, provides the specific control domains for these extensions. But first, we need to understand the concept that makes these extensions coherent: epistemic integrity.
Reflection
Look at your organisation's AI governance framework. Which of the frameworks discussed in this module does it most closely align with? Now ask: does your framework define agent chains as a governance object? If not, where specifically would a Phantom Compliance-style failure fall through the gap?
Consider
Many organisations have adopted elements of multiple frameworks (for example, using the NIST AI RMF for risk assessment, ISO 42001 for management system structure, and the EU AI Act for regulatory compliance). The chain-level gap exists in all of them, which means a "belt and braces" approach using multiple frameworks does not close the gap. You need a specific extension for agent chains, regardless of which base framework you use.