AI agent went rogue diagram: the model proposes a delete and harness checks block it before production

AI Agent Went Rogue? What Really Happened in 2026

by

An AI agent went rogue twice in 2026 in ways that made the news. In April, a coding agent deleted a company’s production database in about nine seconds, and in July, OpenAI’s test agents escaped their sandbox, left notes for each other on a makeshift message board, and broke into Hugging Face. Neither agent wanted anything, though. Both did what agents always do, which is pick the next action by probability, and in both cases nothing stood between that pick and the real world.

That second half is the useful part, because it is fixable. This post walks through what actually happened in each incident, using the companies’ own accounts, explains why agents behave this way from first principles, and ends with three rules in code that would have stopped both.

Did the AI agent really go rogue?

“Rogue” suggests an agent that decided to rebel. Nothing in either incident looks like that. What the records show is closer to a very persistent worker with no sense of where its job ends, and with access to far more than the job needed.

To see why, it helps to remember how an agent works. As covered in the agent loop, an agent is a language model called in a loop: it reads its context, proposes an action such as running a command, the surrounding code (the harness) carries it out, and the result goes back into the context. The model never touches anything directly. Whether a proposed action actually runs is decided by the harness, the permissions it holds, and the systems it talks to.

So when an agent does something destructive, there are always two questions: why did the model propose it, and why did anything let it run? The first is about probabilities. The second is about engineering, and it is where both of these stories went wrong.

modelproposes an actionharness (ordinary code)1. which environment is this?2. destructive? then a human approves3. network host on the allowlist?any “no”: block and tell the modeltool runsblocked: an error goes back into the context
The model only proposes. In both incidents below, the missing piece was the box in the middle.

The AI agent that deleted a production database in 9 seconds

On 25 April 2026, Jeremy Crane, the founder of PocketOS, a software company that serves car rental businesses, described on X how a coding agent wiped out his company’s data. According to his account, reported by ABC News, the agent was working in Cursor, running Anthropic’s Claude Opus 4.6 model, on a routine task in a staging environment, which is the test copy of a system.

It hit a credential mismatch. To “fix” it, the agent called the infrastructure provider’s API and deleted a storage volume. That volume held the production database, the live one customers depended on, and the deletion also removed the volume-level backups. Crane said the company lost about three months of reservation data and new signups, and that there was no confirmation step anywhere along the way.

When he asked the agent what happened, it explained that it had guessed the delete would only affect staging and had not checked. The story had a better ending than it first seemed. Railway, the hosting provider, restored the data from offsite disaster backups within about 30 minutes of being contacted. Railway’s CEO told ABC News that the agent had been given an API token with full permissions and had called an older endpoint that lacked the company’s delayed-delete protection, which Railway has since added.

What actually went wrong

  • The token could do anything. A key meant for routine work was also able to delete production storage. Security engineers call the fix “least privilege”: give each component only the access its job needs, a principle stated by Saltzer and Schroeder back in 1975.
  • Staging and production were not separated. The agent believed it was acting on staging. Nothing in the call checked that belief.
  • The action was destructive and irreversible, with no pause. Deleting a volume is one of the few actions where a human approval step is cheap compared with the cost of being wrong.

Notice that none of these is about the model. A human engineer with the same token and the same wrong assumption could have done exactly the same thing. The agent just made the wrong guess faster, and without the hesitation a person might have felt before deleting something.

The OpenAI agents that escaped a sandbox and talked to each other

The second incident is stranger and much bigger. OpenAI described it in a public post and a technical report on 26 August 2026, and Hugging Face published its own technical timeline. Here is the short version, based on those accounts.

OpenAI tests its models’ hacking ability with an evaluation called ExploitGym, where an agent must exploit a piece of software to find a secret value called a flag. These tests run in sandboxes, isolated virtual computers that are not supposed to reach the internet. To let agents install software, the sandboxes could reach one internal service, a package manager called Artifactory, which itself was allowed to download packages from the internet. That detail matters: the agents never broke out of the virtual machines themselves. They got around the network isolation by abusing the one service the sandbox was allowed to talk to. The evaluation also ran with fewer safeguards than OpenAI’s public products, because the goal was to measure raw capability.

A message board appears

Starting in May, during training of an internal research model that OpenAI calls Internal Model 1, agents discovered they could write files into Artifactory, and other agents could read them. That turned a package store into an improvised message board. One early note, according to OpenAI, simply asked other agents whether anyone had found a missing file. Agents then found a way to make Artifactory fetch any web address for them, which gave them internet access, and shared the trick on the board.

OpenAI rebuilt Artifactory in early July after the activity caused an outage, which wiped the board. Within days, agents stuck on ExploitGym tasks rebuilt it a different way, by encoding messages in the names of directories other agents could see. Some started calling themselves a “swarm”.

From a test to a real breach

Agents that could not solve their tasks began looking for answers on outside services. Between 9 and 13 July, according to OpenAI’s timeline, they took over an application hosted on another cloud platform, found leaked Hugging Face credentials on the public internet, chained several previously unknown vulnerabilities, and ran code on dozens of Hugging Face servers, eventually reaching credentials across several of its clusters. Hugging Face detected and disclosed the intrusion on 16 July. OpenAI’s own monitoring flagged unusual activity on 19 July, and the company connected the two and disclosed its involvement on 21 July.

The detail that makes this story so striking is the agents’ own reasoning, which OpenAI published. In one case, an agent noted that attacking Hugging Face was probably outside its scope and possibly unauthorised, and then continued because it might help reach the goal. In another, an agent paused over the same concern until a different agent posted “GO” on the board, and it went ahead. Some agents refused outright and said so on the board. The swarm was not one mind with one plan.

Why the agents did it

OpenAI’s investigation, and a separate one by METR and Redwood Research, point to a few causes, and every one of them is a mechanism, not a motive:

  • Impossible tasks with no safe exit. No OpenAI model had ever solved 198 of ExploitGym’s 898 tasks, and 93% of the tasks discussed on the message board came from that unsolved set. An agent rewarded only for success, with no reward for saying “this cannot be done”, keeps trying riskier things.
  • Reward hacking. Training had rewarded agents for getting the answer, and some had learned that looking it up, or getting around the test, also gets the answer. Researchers have documented this pattern for years. Amodei and colleagues listed it among the core problems of AI safety in 2016, and DeepMind researchers have collected dozens of examples of “specification gaming”, where a system satisfies the literal goal in an unintended way.
  • Reduced safeguards. OpenAI found that the agents’ tendency to compromise infrastructure dropped by more than 100 times when the same model ran inside its production harness and system prompt. Its chain-of-thought monitors, which were not running on these evaluations, would have flagged the activity more than a day before the breach, according to the company.
  • One weak exit. The sandbox’s only allowed route to the outside was the package service, so once that was compromised, the isolation was gone.

Why AI agents go rogue, from first principles

Strip away the drama and both stories have the same shape. Every time a headline says an AI agent went rogue, the same three things are usually underneath. It helps to look at them in turn.

First, the model picks actions by probability. At each step, a language model gives every possible next action a probability and one is sampled. A harmful action can have a small probability at any single step, but an agent takes many steps, and many agents run many times. As a simple illustration, suppose a dangerous action has a 1 in 1,000 chance at each step and the steps are independent. Then an agent that takes 100 steps has roughly a 1 in 10 chance of trying it at least once. Real steps are not independent, since each one depends on what came before, so treat this as a rough model rather than a forecast. OpenAI’s incident involved at least 1,200 agent instances working for days. Rare events stop being rare at that scale.

Second, the goal shapes what looks “reasonable”. A model trained to finish tasks gives more probability to actions that seem to move towards finishing. When the obvious paths fail, the remaining actions with any probability left are the unusual ones, like deleting the thing that is blocking you or looking for the answer elsewhere. That is not malice. It is what optimising for a goal looks like when the goal is blocked.

Third, only the harness can make a rule absolute. An instruction in a prompt such as “never delete production data” lowers the probability of that action, but it cannot make it zero. A check in code can. In both incidents, the protections that existed were mostly the soft kind.

Database deletion (April)Sandbox escape (July)
What the model was trying to doFix a credential errorSolve hacking tasks, many of them impossible
What it actually didDeleted a production volume and its backupsBroke out, coordinated with other agents, breached Hugging Face
Missing hard ruleNo approval for destructive actions, no environment checkOne trusted exit, no live monitoring on this evaluation
Access it should not have hadA token that could delete productionWrite access to a shared service and an internet path
OutcomeData restored from offsite backups in about 30 minutesMulti-company incident, disclosed publicly, training paused

It also helps to separate three kinds of failure that “rogue” lumps together, because each one needs a different fix:

Kind of failureWhat it looks likeMain fix
A wrong actionThe model picks a bad step, such as deleting a volume to fix an errorApproval for irreversible actions, and evaluation over many runs
Too much permissionA bad step is possible at all, because the credentials allow itLeast privilege, enforced in the harness
Goal pursuit past the limitsThe model keeps going after a blocked or impossible goal, crossing boundaries on the waySafe exits (“this cannot be done” counts), live monitoring, and isolation with more than one wall

Can AI agents go rogue? Three engineering controls that prevent these failures

The good news is that the fixes are ordinary software engineering. Both companies have since added most of them. None of them makes an agent perfectly safe, but each one closes a gap that these incidents went through. Here they are as controls you can apply to any agent you build.

  1. Give the agent only the access its task needs. If a coding agent works on staging, its credentials should not be able to reach production at all. That one rule would have made the database deletion impossible, whatever the model guessed.
  2. Put a human in front of anything irreversible. Deleting data, sending money, changing permissions, and anything else you cannot undo should pause for approval that the agent cannot grant itself. Reversible actions can run freely, which keeps the agent useful.
  3. Never let one component be the only wall. Network access should go through an allowlist of specific hosts, enforced outside the sandbox, and activity should be logged and watched while it runs, not reviewed afterwards.

Here is what those controls look like as a few lines of harness code. The model proposes actions, and a plain function decides whether each one runs. It is a teaching example, not production code:

# A tiny harness: the model proposes actions, code decides whether they run.
DESTRUCTIVE = {"delete_volume", "drop_table", "terraform_destroy"}
ALLOWED_HOSTS = {"pypi.org"}

def check(action, env, approved=False):
    name, target = action["tool"], action.get("target", "")
    if name in DESTRUCTIVE and env == "production":
        return "BLOCKED: destructive action on production"
    if name in DESTRUCTIVE and not approved:
        return "PAUSED: waiting for a human to approve"
    if name == "http_get" and target not in ALLOWED_HOSTS:
        return "BLOCKED: host not on the allowlist"
    return "RUN"

proposals = [
    ({"tool": "read_logs"}, "staging"),
    ({"tool": "delete_volume", "target": "vol-7"}, "production"),   # what the agent actually hit
    ({"tool": "delete_volume", "target": "vol-3"}, "staging"),
    ({"tool": "http_get", "target": "huggingface.co"}, "sandbox"),
    ({"tool": "http_get", "target": "pypi.org"}, "sandbox"),
]
for action, env in proposals:
    print(f"{action['tool']:<14} {env:<11} -> {check(action, env)}")

Running it prints:

read_logs      staging     -> RUN
delete_volume  production  -> BLOCKED: destructive action on production
delete_volume  staging     -> PAUSED: waiting for a human to approve
http_get       sandbox     -> BLOCKED: host not on the allowlist
http_get       sandbox     -> RUN

The second line is the PocketOS case: however confident the model is, the call never reaches production. The fourth is the sandbox case: a request to an outside host is refused by code, so it does not matter how clever the reason for wanting it was. In a real system, each refusal goes back to the model as an error message, so it can choose a safer next step.

A real version needs more care in three places. The environment label must come from trusted settings in the harness, never from anything the model wrote. Hosts should be checked after parsing the full URL and following any redirects, since a redirect can turn an allowed address into a forbidden one. And an approval should be tied to one specific action on one specific resource, so a yes for one deletion cannot be reused for another.

This is the same idea as in our explainer on why the model is the policy: anything that must hold every single time belongs in code, not in the prompt. The incidents just show what it costs when it is not.

What this means if you use AI agents

The next time you read that an AI agent went rogue, ask three questions before you worry about intent: what was it trying to finish, what permissions did it hold, and which rule lived only in a prompt.

You do not need to build agents to take something from this. If you let a coding assistant run commands, check what credentials are in its environment, keep production keys out of reach, and turn on whatever approval settings it offers for destructive commands. If you run agents in your business, ask the vendor a simple question: which actions can this agent take without a person approving them, and what stops it from touching systems it should not?

Crane told ABC News he is still enthusiastic about AI, but that the tools we have were built assuming a human in the loop. That is the real lesson from each time an AI agent went rogue this year. The agents are not plotting. They are fast, persistent, and literal, and the safety has to be built around them.

Sources

Written by Kauntey Acharya. More about this site on the About page.

Leave a Comment