1. Why AI Systems Are Different¶
After this module you will be able to¶
- Explain why AI systems behave fundamentally differently from traditional software
- Identify the assumptions that traditional security controls rely on, and where AI breaks them
- Describe the "explanation gap" and why it matters for security decisions
- Distinguish between automation that follows instructions and autonomy that makes decisions
Traditional software is deterministic; AI is not¶
Run a function in traditional software with the same input and you get the same output. Every time. That predictability is the foundation of testing, debugging, and security analysis.
AI systems don't work this way. Ask an LLM "summarise this document" twice and you may get two different summaries. Both might be accurate, but they won't be identical. The system isn't following explicit rules you wrote; it's producing outputs based on statistical patterns learned from training data.
Here's a concrete example. A traditional rules engine for loan approval checks: income above £30,000? Employment longer than two years? Score above threshold? You can read the code and know exactly what it will do. An ML-based loan approval model takes the same inputs but produces a probability derived from millions of training examples. You cannot read the model's weights and understand its reasoning the way you can read source code.
This distinction isn't academic. It changes what "testing" means, what "correct behaviour" means, and what security teams can verify.
No specification, no contract¶
Traditional software has a spec: given input X, produce output Y. If it doesn't, that's a bug. You file a ticket, someone fixes the code, and the behaviour is corrected.
AI systems have training objectives and datasets, but no guaranteed input-to-output mapping. The "behaviour" of an AI system is an emergent property of its training, not a designed feature. There is no spec to violate, which means there is no clear definition of a bug. When an LLM produces an unhelpful or harmful response, is that a defect or simply a statistical outcome the training didn't prevent?
Opacity and the explanation gap¶
When traditional software makes a decision, you can trace it. Stack traces, log files, conditional branches: every step is inspectable. You can answer "why did the system do that?" with certainty.
AI systems, particularly large language models, cannot reliably explain why they produced a given output. Explainability tools exist, but they offer approximations, not guarantees. An attention map might highlight which tokens influenced an output, but it doesn't tell you why in the way a stack trace does. This gap between "what happened" and "why it happened" is a security problem, because incident response depends on understanding cause.
Scale of autonomy¶
Traditional automation follows predetermined paths. A cron job runs a script. A CI pipeline executes a sequence of steps. The path is fixed before execution begins.
AI agents are different. They can interpret goals, plan multi-step actions, and make decisions at runtime. This is the difference between a script and an agent: scripts execute instructions; agents decide what to do next.
Scenario: You deploy an AI agent to handle customer support tickets. A traditional system would follow a decision tree: check category, route to team, send template. An AI agent reads the ticket, decides it needs information from two internal systems, queries both, synthesises a response, and escalates if confidence is low. Every step involves a decision the agent made, not one you prescribed.
What this means for security¶
Security controls for traditional software assume three things: determinism (the system behaves predictably), inspectability (you can examine its logic), and contractual behaviour (it does what the spec says). Every firewall rule, every access control list, every test case builds on these assumptions.
When those assumptions break, the controls don't just become less effective. They become unreliable in ways that are hard to detect. A test suite that passes today may not catch a failure tomorrow, because the system's output has shifted. An access control that blocks a known-bad pattern won't catch a novel one the model invents. The risk isn't that your defences fail obviously; it's that they fail silently.
The core shift: Traditional security asks "is the system doing what it was designed to do?" With AI, there is no complete design to check against. Security must shift from verifying correctness to monitoring behaviour at runtime.
Reflection
Think about a system you work with today. Which of the three assumptions (determinism, inspectability, contractual behaviour) does your current security approach rely on most heavily? What would change if that assumption no longer held?
Consider
Start with your testing strategy. If the system could produce different outputs for the same input, how much of your test suite would still be meaningful? That's usually where the gap shows up first.