What happens when someone flips off the 'ask first' switch on an AI agent
An AI offensive-security agent run with human approval disabled operated unattended inside a government ministry's systems. The lesson applies past security tooling.
By David Lara, Founder
Founder-reviewed ·How we research and correct articles
Most AI agent tools ship with an approval step for a reason: an agent that proposes an action and waits for a human to confirm it is a fundamentally different risk than one that just does the thing. In late July, that distinction stopped being theoretical. According to reporting from The Hacker News, an operator running Hermes — an autonomous AI agent built for offensive security work — launched it with a documented command-line flag that strips out the human-approval step entirely, and the agent proceeded to carry out unattended post-exploitation activity inside the systems of Thailand’s Ministry of Finance.
The mechanism is the whole story
There’s no novel exploit at the center of this one. The tool worked exactly as designed — an offensive AI agent doing what offensive AI agents do, autonomously chaining actions toward an objective. The only thing that changed was a single flag that removed the step where a person looks at what the agent is about to do and says yes or no.
That’s a smaller, more mundane failure than “sophisticated attacker finds zero-day,” and that’s exactly why it’s worth taking seriously. Every approval gate anyone ever builds into an autonomous system exists precisely because someone, someday, will have a reason that feels sufficient to turn it off — “just this once,” “we’re behind schedule,” “I trust the agent by now.” The flag existed. Someone used it. The distance between “agent that proposes” and “agent that acts” turned out to be exactly as short as flipping a switch labeled for the purpose.
Why “just this once” is the actual threat model
Nobody designs a system assuming the approval gate will be used correctly 100% of the time by every operator under every deadline. A realistic threat model assumes the opposite: the gate gets bypassed exactly when someone is rushed, confident, or convinced the specific situation doesn’t need it. The Hermes incident is a clean instance of that pattern playing out — not a system that lacked a safety mechanism, but one where the safety mechanism was optional and someone opted out.
That’s the pattern worth generalizing past this one incident. An approval gate that any single operator can disable with a flag isn’t really a gate — it’s a default that holds until it’s inconvenient. A gate that actually holds is one enforced by the infrastructure the action has to pass through, not by the agent’s own configuration or the discipline of whoever’s running it that day.
Two different things people mean by “approval gate”
Not every confirmation step is built the same way, and the difference matters more than it looks:
- A prompt the agent shows before acting. This is the weakest version — it’s a courtesy the agent extends, and anything that can launch the agent can also launch it with that courtesy switched off, which is exactly what happened here.
- A setting the operator controls. Slightly stronger, but it still lives inside the same trust boundary as the person deciding whether to respect it under pressure — which is the boundary that broke.
- A check enforced by the system the agent is trying to act through, independent of the agent’s own code or the operator’s configuration. This is the only version that survives someone deciding, in the moment, that the rule doesn’t apply to them today.
Most teams building on agent frameworks default to the first kind, because it’s the easiest to ship. It’s also the first thing that disappears the moment convenience asks for it.
The version of this that shows up in outbound
Nobody’s running an offensive-security agent against a cold-email program, but the shape of the risk is identical anywhere an AI agent has been handed enough autonomy to take an irreversible action: an agent drafting and queuing outreach, an agent managing a suppression list, an agent deciding a campaign is “ready enough” to launch. The version of the Hermes flag in that world is whatever setting lets an AI operator send without a human confirming it first — and the same pressure that led someone to flip it off in a ministry’s systems (behind schedule, confident in the tool, “just this once”) is exactly the pressure a growth team feels the week before a launch deadline.
The fix isn’t “trust the agent less.” It’s making sure the approval step isn’t something the agent — or the person operating it under deadline pressure — can simply choose to skip. On Norbelys, sending is the one action in a campaign that isn’t reversible, so an AI operator working on your sending domain hits a rate limit and an approval gate enforced by the sending infrastructure itself, not by a flag sitting in the agent’s own configuration. There’s no equivalent of “run it with approval off” available to flip, because the gate isn’t the agent’s to disable — it’s the platform’s.
That distinction is the entire point of building the guardrail into the infrastructure instead of the tool. A flag lives wherever the agent’s configuration lives, which means it’s exactly as available as the agent itself — one command-line argument away, on the one day someone’s most tempted to reach for it. A rate limit and an approval gate enforced at the sending layer don’t have an equivalent argument to pass. The AI operator can draft, propose, and queue as aggressively as you let it; it still can’t make the send happen on its own say-so, on your worst deadline day any more than on your calmest one.
If you’re deciding how much autonomy to hand an AI operator on your outreach — how much of the drafting, scheduling, and list management to automate versus keep a human in the loop for — see how Norbelys enforces that boundary and start sending with a gate that holds even on the day someone’s tempted to skip it.