1. What Goes Wrong¶
After this module you will be able to¶
- Describe the Phantom Compliance scenario as a governance failure, not just a technical one
- Identify the accountability gaps that emerge when agents delegate to agents
- Explain what the board needs to know about multi-agent risk that it currently does not
- Map agent chain failures to existing enterprise risk categories
The governance view of Phantom Compliance¶
You have already seen the Phantom Compliance scenario from a technical perspective. Now consider it from the perspective of a Chief Risk Officer receiving a call at 17:30 on a Thursday.
The facts are these: your firm executed a trade in a restricted security. The compliance system approved it. Every log shows the system functioning correctly. The guardrails passed. The audit trail is clean. And yet, the firm is now facing a potential regulatory enforcement action.
The first question the board will ask is: who is accountable?
The second question will be: how did our governance framework allow this to happen?
The third question, the one that matters most for this track, is: what about our governance framework needs to change so that this category of failure is covered?
Why this is a governance failure¶
It is tempting to classify Phantom Compliance as a technical failure: Agent B received truncated data due to context window pressure. Fix the context window. Problem solved.
But the technical failure is a symptom. The governance failures are structural:
The governance failures in Phantom Compliance:
-
No accountability framework for agent-to-agent delegation. When Agent C relied on Agent B's compliance check, it performed an act of delegation. No governance policy defined who is accountable when that delegation produces a wrong result.
-
No requirement for agents to verify the completeness of their reasoning inputs. The governance framework required compliance checks but did not define what constitutes an adequate check. Specifically, it did not require verification that the data underlying the check was complete.
-
No distinction between "the system ran" and "the system worked." Monitoring proved the system ran. Nobody asked whether the system worked correctly, because the governance framework treated monitoring as equivalent to oversight.
-
No board-level visibility into reasoning-chain risk. The board received reports on model accuracy, system uptime, and incident counts. None of these metrics would have surfaced the Phantom Compliance failure before it occurred.
The accountability gap in agent chains¶
In traditional software, accountability follows the decision chain: a human makes a decision, or a system executes a rule that a human defined. When something goes wrong, you trace back to the person who made the decision or wrote the rule.
In a single-agent AI system, accountability is already more complex (the model may produce unexpected outputs), but there is still a clear chain: the model produced an output, a human (or a human-defined guardrail) reviewed it, and someone is responsible for the model's deployment and oversight.
Multi-agent systems break this chain in three ways:
1. Delegation without explicit authorisation¶
When Agent C accepts Agent B's compliance assessment, it is delegating the compliance decision to Agent B. But nobody explicitly authorised that delegation. It emerged from the system architecture. The compliance team approved the pipeline as a whole, but they did not specifically authorise Agent C to treat Agent B's output as a definitive compliance determination.
In your organisation, if a junior compliance analyst delegated a sanctions check to an intern and the intern missed a match, accountability would be clear: the analyst is responsible for the delegation, the intern is responsible for the work, and the compliance function head is responsible for the oversight framework.
When Agent C delegates to Agent B, who holds each of these accountability roles? If the answer is "we haven't defined that," you have a governance gap.
2. Invisible quality degradation¶
Agent B did not refuse to work. It did not produce an error. It produced a confident, well-formatted compliance assessment based on incomplete data. The quality of its reasoning degraded invisibly.
In governance terms, this is equivalent to a compliance officer who checks only half the sanctions list but reports a clean result. In a human process, we would call this negligence or inadequate procedure. In an agent chain, we have no governance category for it, because our governance frameworks assume that if the system runs without errors, it ran correctly.
3. Confidence laundering¶
Agent B's output carried no indication that its check was incomplete. Agent C received it as a definitive result. The partial check became, through the act of passing between agents, a complete check in the eyes of the decision-maker.
This is what the AIRS framework calls confidence laundering: uncertainty or incompleteness in one agent's work is washed out as it passes through the chain. Each downstream agent treats the upstream output as more authoritative than it actually is.
Reflection
Think about the AI systems in your organisation. When an AI agent produces an output that another system or person relies on, is there a defined standard for what that output must contain, including metadata about its completeness and reliability? Or does the downstream consumer simply receive the output and assume it is correct?
What the board needs to know¶
If you are presenting to a board or governance committee on multi-agent AI risk, they need to understand four things:
1. Agent chains create accountability gaps that existing policies do not cover¶
Most AI governance policies address individual models: how they are trained, tested, deployed, and monitored. They do not address what happens when models are chained together and one agent's output becomes another agent's input.
The board question: Do our AI governance policies address agent-to-agent delegation, or only individual agent deployment?
2. Clean audit trails do not prove correct operation¶
Phantom Compliance had a perfect audit trail. Every agent logged its inputs and outputs. Every guardrail passed. The logs would satisfy a conventional audit, and yet the system produced a regulatory violation.
The board question: Does our monitoring prove the system worked, or only that the system ran?
3. Existing risk metrics do not capture reasoning-chain risk¶
Standard AI risk metrics (model accuracy, false positive/negative rates, system availability) measure the performance of individual components. They do not measure the integrity of the reasoning chain that connects those components.
The board question: What metric would have detected the Phantom Compliance failure before the trade was executed?
4. Regulatory exposure is higher than current frameworks suggest¶
Regulators are increasingly focused on AI governance, and multi-agent systems create exposure that single-agent risk assessments do not capture. When an agent chain produces a regulatory violation, the regulator will ask not just "what went wrong" but "why didn't your governance framework prevent it?"
The board question: Are we governing the AI systems we actually have, or the AI systems we had two years ago?
Mapping to enterprise risk categories¶
For risk professionals, Phantom Compliance maps to several existing enterprise risk categories, but in each case it extends the category in ways that existing controls may not cover:
| Risk category | Traditional scope | Multi-agent extension |
|---|---|---|
| Operational risk | System failures, process errors | Agent chain integrity failures that produce correct-looking outputs from broken reasoning |
| Compliance risk | Regulatory violations | Violations caused by agent delegation where no single point of failure is identifiable |
| Model risk | Individual model performance | Chain-level risk where individual models perform correctly but the chain produces wrong results |
| Third-party risk | Vendor and outsourcing risk | Agent-to-agent delegation as an internal "outsourcing" that bypasses third-party risk controls |
| Reputational risk | Public-facing AI failures | "Invisible" failures that only surface during regulatory examination or audit |
The risk framing: Multi-agent AI systems do not create entirely new risk categories. They extend existing categories in ways that current controls and governance frameworks do not reach. The governance task is to extend your existing frameworks, not to build new ones from scratch.
The cost of getting this wrong¶
The consequences of unaddressed agent chain governance are not hypothetical:
- Regulatory enforcement: A firm that cannot demonstrate adequate governance over its AI systems faces enforcement action. For multi-agent systems, "adequate governance" increasingly means governance over the chain, not just the individual agents.
- Audit failure: Internal and external auditors are beginning to ask about AI system interactions, not just individual AI deployments. If your audit framework does not cover agent chains, you will have findings.
- Incident response gaps: When a Phantom Compliance-style incident occurs, the investigation will be slower and more costly if there is no governance framework for agent chains, because investigators will not know where to look or what to look for.
- Board liability: Board members have a duty of oversight. If the board was not informed that multi-agent systems create governance gaps, and an incident occurs, the question of whether the board was adequately informed becomes a liability question.
You are the CRO of Meridian Capital. It is three days after the Phantom Compliance incident. You are preparing a briefing for the board. The CEO has asked you to answer three questions:
- How did this happen despite our governance framework?
- Are there other systems with similar exposure?
- What needs to change?
For question 1, you can explain the technical failure, but the board will want to know why the governance framework did not prevent it. The honest answer is that the framework governed individual agents, not the chain.
For question 2, the honest answer is: you do not know, because you have not assessed your other AI systems for agent chain risk. This is a finding in itself.
For question 3, the answer is the subject of the rest of this track.
Reflection
If a Phantom Compliance-style incident occurred in your organisation tomorrow, how would the governance investigation proceed? Would investigators know to look at inter-agent data flows, or would they focus on individual agent performance? Would the board receive information about chain-level integrity, or only individual component metrics?
Consider
The speed of your investigation depends on whether your governance framework already recognises agent chains as a governance object. If it does, investigators have a playbook. If it does not, the investigation starts with "what are we even looking at?", which costs time, money, and regulatory patience.