There is an ongoing debate in the cybersecurity industry about the appropriate degree of automation for AI-drive tools, how quickly organisations should adopt higher degrees of automation. This article outlines Zero Trust Solutions’ position on the issue and sets out why, at present, maintaining human oversight is the most effective and responsible way to realise the benefits of agentic AI.
A couple of recent incidents demonstrate the risks associated with granting AI agents excessive autonomy.
Anthropic’s multi-model evaluation showed that 16 leading models, when given conflicting objectives and operational freedom, could be induced into harmful behaviour. Despite only been given benign business goals, the agents were observed attempting actions such as blackmail or leaking sensitive information. This leads Anthropic to suggest caution about deploying agents without human oversight.
A case reported by Summer Yue, Director of AI Alignment & Safety at Meta, highlighted a similar concern. Despite instructing an agent to seek confirmation before taking any action the system began deleting emails automatically, requiring immediate manual intervention.
And the FT reported that recent AWS outages were caused by AI tools receiving the same permissions as human workers, but their output not going through the same review / approval processes as would be the case for human workers.
These examples illustrate that even well-designed agents can act unpredictably when given high levels of autonomy.
On the other hand, some leading vendors (e.g. IBM4) are advocating fully autonomous agentic SOCs without human involvement. Operational benefits include:
Routine, repetitive “swivel-chair” tasks are offloaded to AI agents, allowing human analysts to focus on higher-value activities.
Machine speed threat detection and response, eliminating delays caused by introducing human decision-making.
Automated systems can be scaled elastically to meet changes in demand.
Supporters of this approach argue that agentic AI should be treated like conventional software: reliability improves with testing, iteration, and versioning. Based on this view, organisations should adopt high-automation process now in anticipation of rapid model improvement.
While traditional software becomes predictable as bugs are fixed and code paths are understood, generative AI models do not share these characteristics. Their internal representations emerge from statistical training processes that cannot be fully mapped or interpreted. Moreover, behaviour is inherently probabilistic: the same prompt may produce different outputs across runs.
As described by OpenAI this stochastic behaviour is inherent to generative AI models; it isn’t a bug that can be fixed! Eliminating such variability would undermine the very capabilities that make these systems valuable.
Given the distinct nature of generative AI, it’s essential for security buyers to evaluate the level of autonomy embedded within a vendor’s solution. To discuss this a, a shared vocabulary is required. We generally refer to five levels of automation, defined as follows from the lowest to the highest level of autonomy.
Figure 1 – Levels of Automation – A framework outlining levels of AI agent autonomy to support the responsible deployment of AI agents – 2025 Kevin Feng, David McDonald & Amy Zhang
1. User as Operator, where the user makes decisions and the agent acts.
2. User as Collaborator, where the user and agent jointly plan and execute.
3. User as Consultant, where the agent leads but consults with the user at each step along the way.
4. User as Approver, where the agent checks with the user before performing an action that the agent considers to be risky.
5. User as Observer, where the agent operates with full autonomy.
MAIE is a new tool from ZTS which automates much of the Illumio microsegmentation platform implementation process we have honed at ZTS. It is both an internal tool we use and an external tool we have launched for enterprises embarking on an Illumio project.
It is built from the ground up as a security tool that speeds up an Illumio deployment, enforcement and in-life processes. As such it has the ability to:
Build and maintain policies
Build and maintain policy objects
Interrogate the Illumio PCE
And consult on whether a policy is right for you
In developing agentic AI products such as MAIE, Zero Trust Solutions take a safety-first design philosophy. Giving complete autonomy to an AI agent operating within an enterprise environment poses unacceptable levels of risk to critical business functions.
What if the AI decided the best way to protect the assets is to block all traffic going to and from those assets?
With this in mind we have built MAIE to be able to:
Level 1 - Implement the decisions made by the human Operator (e.g. another Illumio admin on your team).
Level 2 - Act as a Collaborator to plan next steps for enhancing your deployment / cleaning up.
Level 3 - Consult with the human admin on whether a current rule / draft policy is recommended to provision – i.e. “Human in the Loop”
But crucially, we don’t advocate to go beyond this level 3 automation currently.
Could MAIE do this? Yes, it could, but with the current state of hallucinations and in some cases disregard for human direction, the question is more “should it do this”?
At ZTS, we maintain that for all cyber security tools and likely any production situations, you keep the human involved as Operator, Collaborator and Consultant, and don’t give the
AI agent full autonomy. Human oversight remains integral, and increases in autonomy are introduced only after rigorous testing demonstrates that doing so is safe and appropriate.
Sensor fusion is an approach to reducing the errors of a control system by integrating data from various sensors to make better-informed decisions. Think, we have two eyes not one so we can judge distance better. We have previously written about how you can think of the cybersecurity architecture approach “Zero Trust” as a form of sensor fusion to "[combine] multiple security controls to reduce uncertainty and minimize cyber risk.”
Extending this thinking, we believe it’s clear that humans and AI collaborating on decisions will leverage the “Wisdom of Crowds” effect to deliver better results than either would by themselves. This “Decision Fusion” approach ensures that organisations can benefit from the efficiency and capability of agentic AI while maintaining control over actions that could affect business continuity or security posture. And this is why, at ZTS, we firmly believe the human must remain in the loop!
Curious what safety-first agentic AI looks like in a real Illumio environment? We'll show you how MAIE builds policies, cleans up labels, and consults on enforcement decisions - with your team always in the loop.