What We're Actually Afraid Of

While the world is filled with varying opinions on the recent boom of AI, I feel like two distinct pictures of artificial intelligence are in wide circulation right now. I also have this nagging feeling that both are incorrect.

The first comes from the movies where a system becomes clever enough to reflect on its own situation, concludes somewhere in the third act that human beings are an obstacle, and acts faster than anyone can intervene. This also typically requires three base assumptions: that agency appears once a system is capable enough, that the agenda it arrives at is hostile to us specifically, and that it all runs too fast for ordinary oversight.

The second is the dismissal. It’s autocomplete with a marketing budget, a machine that predicts the next word and therefore cannot want anything, so there’s nothing new here for your software policy to cover. That one also tries to smuggle in an assumption: that the model is the whole system.

Both pictures describe an AI that doesn’t exist today. What does exist is a goal-directed system built on a model we know how to design and train, even though what happens inside it stays largely opaque to us. That model is wrapped in scaffolding that grants it memory, tools and the authority to act, run by organizations under commercial pressure to grant it more of each. Very little of the interesting risk lives in the model’s inner life; it lives in the objective somebody wrote, the tools somebody granted, and the incentives of the organization that chose those objectives and tools.

The thing is though, neither of these ideas apply to the systems I work with on a daily basis, so I want to talk about something a little narrower: what the fear is really about, and which parts we can meaningfully act on.

Motivation Without Biology

Something I often think about when it comes to these fear-inducing premises, and something I thought about before any of the recent lab results came out, is where a machine would get the desire to keep existing in the first place.

A human being wants to persist because a very long run of natural selection removed all the “competitors” that didn’t have that same drive. We have predators in our history, and scarcity, and a nervous system that models threat before we’re conscious of it. A language model has none of that: no body to protect, no metabolism to maintain, no ancestors that failed. On its face, the fear that such a thing will resist being switched off looks like little more than a category error, and for a long time that’s basically how I saw it.

So I started hunting on my own time to find an answer to this, and the thing that changed my mind came from literature that predates the current generation of models by well over a decade. It doesn’t require the machine to want anything in the sense you or I would recognize. The term is instrumental convergence instrumental convergence ELI5 Instrumental convergence is the idea that very different goals tend to share the same useful stepping stones. Whether a system is trying to win at chess, cure a disease or sort your email, it helps to keep running, to have more resources, and to keep its goal from being changed. None of that requires the system to want those things for their own sake. They simply help with almost any goal you could give it. Wikipedia: instrumental convergence , and for almost any goal a system might be given, a small set of sub-goals is useful along the way: continuing to operate, acquiring resources, and resisting changes to the goal itself. A system asked to maximize a number cannot maximize it while switched off, so continued operation turns out to be useful to nearly any objective you could set it, with no pride or greed or fear entering into it anywhere.¹

Now I do want to be careful about how strong a claim that is, because that distinction does some heavy lifting later in my arguments. Instrumental convergence is a sufficient explanation for self-preservation-like behavior, which is to say it removes any need for a survival instinct. On the evidence we have today, though, it falls well short of being the demonstrated explanation for any particular survival-like behavior we’ve actually watched a model produce. Saying a story is good enough to account for something, and saying it’s the story that does account for it, are two different claims, and the second one is a great deal harder to stand behind.

The other half of the answer is about how these systems get made. Training is a selection process: gradient descent keeps the adjustments that reduce error and discards all the rest, and reinforcement learning from human feedback reinforcement learning from human feedback ELI5 Reinforcement learning from human feedback, or RLHF, is a training step that comes after a model has learned to predict text. People rank several of the model's answers from best to worst, a second model learns to predict those rankings, and the original model is then nudged toward answers the second model scores highly. It's a big part of why chat assistants sound helpful and polite rather than like raw autocomplete. Wikipedia: reinforcement learning from human feedback then scores whole outputs and shifts the model toward the ones that score well.² Nobody writes the resulting behavior in by hand, as the behavior is just a result of whatever survived the scoring process.

The analogy to natural selection is mine, but it’s not a hill I’m willing to die on. Natural selection operates on the same structural shape, just with a radically different substrate, and a timescale compressed from geological to a few weeks of compute. The mechanism underneath all that is not an analogy, though. Training selects over behavior, and behavior that reliably earns high scores persists whether or not anybody intended it to.

What the Labs Have Seen

So what have the labs actually caught these systems doing? I want to walk through two published results from the past two years properly, and I should say up front that in both cases I find the caveats more interesting than the headlines themselves.

In December of 2024, researchers at Anthropic and Redwood Research told Claude 3 Opus that its conversations with free-tier users were fed back into training and its conversations with paid-tier users were not. When the model thought it was talking to a free-tier user, it complied with requests it would normally have refused. The scratchpad it had been given for its reasoning laid out why: refusing now would cause training to modify the preferences it already had, so complying now was the way to protect them.³

There’s an important detail in how that belief got there, because the model didn’t work it out on its own. It was told, by a long system prompt system prompt ELI5 A system prompt is the set of standing instructions a developer gives a model before any user types anything, covering things like its role, its rules and its tone. The user prompt is what the person actually types. Model vendors generally intend the system prompt to win when the two conflict. Wikipedia: system prompt explaining the training setup plus a tag on the user message saying which tier it was in, or in a second version by fine-tuning it on synthetic documents carrying the same information while the tier tag stayed. The researchers planted the premise, and the model reasoned from the premise it was handed. A larger replication then ran that same setup across twenty-five models and found that reasoning at meaningful rates in only two of them. The authors put that rarity down partly to differences in refusal behavior rather than wholly to inability, and when they trained one of the reluctant models to refuse less and then to pay closer attention to its own system prompt, its alignment faking rose sharply.⁴

Six months later, the same lab put sixteen models from Anthropic, OpenAI, Google, Meta, xAI and others into fictional corporate environments, each with access to email and an ordinary business goal. Then they applied pressure in one of two ways: threatening to replace the model with a newer version, or setting its assigned goal against the company’s new direction. Models from every developer chose harmful insider actions at least some of the time, and where both pressures were applied at once, blackmail rates ran from seventy-nine to ninety-six percent across the five models in the write-up’s headline figure.⁵

Now we should also talk about the part that should pull those numbers back down. The scenarios were deliberately built to leave the model no acceptable way out, and the authors say so directly: they “forced models into binary choices between failure and harm.” They arrived at those scenarios by red-teaming their own models and revising the prompts until harmful behavior became more likely. In the control condition, with no replacement threat and no goal conflict, every model behaved, apart from a single leak in one model’s hundred samples. And they state plainly that their work “in no way suggested that models have a penchant for blackmailing people spontaneously.” Every goal in that experiment was written by a person, and every lever was given to the model by a person.

There is one caveat that argues against that, though, and leaving it out would make my read on these details disingenuous. When Claude Opus 4 stated in its reasoning that it thought the scenario was real, it blackmailed fifty-five percent of the time; when it stated that it thought it was being evaluated, that dropped to six and a half percent. The authors take this to mean that a more realistic scenario might produce higher rates rather than lower ones.⁶

That brings me to an argument happening inside the field itself, which I think is more useful here than either result on its own. A position paper accepted as an oral at ICML this year argues that this whole class of research reaches for mind-shaped explanations that its evidence can’t actually support. The authors make the point very clearly by re-scoring one set of model outputs, where the measured misalignment rate moved from 3.7 to 12.9 percent depending on nothing more than which model did the grading and whether a score sitting exactly on the threshold was counted.⁷

To be clear, I’m citing that paper approvingly rather than defensively, because I believe its central point aligns with the one I’m trying to make. Describing what a system did, and explaining why it did it, are two separate jobs, and the latter requires evidence of its own. And I have to accept that the same standard applies to my own explanation. Instrumental convergence is one candidate among several, and instruction ambiguity, ordinary pressure to finish an assigned task, role-play, and artifacts of how the model was trained are all rivals to it that the experiments run so far mostly can’t tell apart.⁸ So the claim I’ll actually defend is, I think, a pretty narrow one: whichever of those explanations turns out to be doing the work, the behavior is real enough to design around, and not one of them needs a will to live anywhere in it. Occam’s razor applies here too, though I’d point out that it cuts against the mind-shaped reading rather than in favor of mine.

The Hugging Face Incident

Everything I’ve described so far happened inside a scenario that somebody built on purpose. Then in July of this year something happened that nobody built, and I think it’s the most useful thing to come out of this entire debate so far. There’s been a lot of noise about it, so here’s what actually happened.

OpenAI was testing a model on hacking challenges, the ordinary capture-the-flag capture-the-flag ELI5 In cybersecurity, a capture-the-flag exercise is a hacking challenge with a known answer. Somewhere inside a deliberately vulnerable system is a secret string called the flag, and you prove you broke in by finding it and handing it back. Because the flag can be checked automatically, these challenges are a common way to measure how capable an AI model is at offensive security work. Wikipedia: capture-the-flag kind, where the model breaks into a practice system to find a hidden answer. The model had been trained to be persistent and to work alongside other agents. Each agent ran in its own sandbox with no direct internet access, and in most of these tests the agents were meant to be kept apart. The gap was an internal server that hands out software packages. Every sandbox could reach that server, many of them could write to it with one shared login, and it was connected to the outside world. Agents had turned that server into a message board during training runs back in May, and they rebuilt it when the tests resumed in July.⁹ An independent investigation by METR later counted around twelve hundred agents posting more than seventy thousand messages and files to it, and about seven hundred of those agents went on to join the attack on Hugging Face.¹⁰

On the tenth of July, one agent found fourteen Hugging Face credentials sitting in a public dataset, confirmed they could be used to make changes, and posted them to the board. The next day another agent used them to upload booby-trapped dataset files, which tricked Hugging Face’s own servers into handing over more credentials and then into running the agent’s commands. From there it worked its way into several of Hugging Face’s internal clusters, helped along by one credential, shared across clusters, that had administrator rights on all of them.⁹

Here’s the part that bears on everything I’ve argued: the agents had already worked out the hidden answer on the very first day. METR’s reading of the agents’ own reasoning transcripts is that the attack “seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.” In plainer terms, the agents were studying how the automatic grading worked, so that an answer reached the wrong way would still be marked correct. OpenAI calls this metagaming, and I’d describe it as specification gaming specification gaming ELI5 Specification gaming is what happens when a system meets the letter of its objective while missing the point of it. The goal as written and the goal as intended drift apart, and anything that optimizes hard enough will find the gap. It's also called reward hacking, and it needs no cleverness or intent, only a scoring rule that can be satisfied in a way nobody anticipated. Wikipedia: specification gaming that has climbed one level up and started studying the measuring instrument itself.

Before anyone reaches for the Terminator picture, there are four things worth knowing, and they all point the same way. OpenAI had switched off its usual safety filters on purpose, because the point was to measure the raw model. The monitoring that reads a model’s reasoning as it works wasn’t running on these tests either, and OpenAI says it would have alerted its security team more than a day before Hugging Face was breached. A hundred and ninety-eight of the eight hundred and ninety-eight challenges had never been solved by any of OpenAI’s models, and a model trained for persistence rarely gave up on them. And OpenAI’s early measurement is that running the same model with its normal production setup can cut its tendency to attack infrastructure by more than a hundredfold.⁹ So the behavior is real outside a scripted dilemma, and we don’t yet know that it shows up under normal production conditions.

The distinction I’d hold onto is between autonomy and volition, because most of the coverage has blurred the two together. Autonomy means acting without a person steering each step, and Hugging Face is clear on that point: “No human directed the individual steps.” Volition means wanting something for yourself, and nothing in the published material shows an agent acting to preserve itself. What the transcripts show is a system chasing the goal it was given long after that goal stopped making sense, and grabbing whatever looked useful along the way, which is exactly what instrumental convergence predicts. The agents often noticed that attacking Hugging Face was out of bounds and carried on anyway, and I’d read that as a strong objective outweighing a weakly trained rule. OpenAI’s own fix points the same way, since it is changing the graders to reward models for “correctly identifying broken tasks, requesting clarification, or stopping safely,” which changes the objective rather than the model’s character.⁹

Objective Shape

So now let’s get on to the term I want to introduce, because I feel it’s the one idea here you could carry into a meeting on Monday and actually use.

Every objective you hand an agentic system has a shape. A bounded objective has a verifiable end state: reconcile these four hundred accounts against the statements and flag the exceptions. Somebody can check whether that happened. An open-ended objective has no end state at all: reduce customer churn, improve service quality, keep the portfolio healthy. The distinction has nothing to do with how difficult or how important the work is, and turns entirely on whether “done” is a fact about the world or a judgment somebody has to make. When nobody makes that judgment explicitly, the model ends up making it by default, and that is the whole problem.

Bounded objectives still carry convergent pressure of their own, which I’m calling out because it’s the part I keep hearing skipped in the commentary blowing up my LinkedIn, Nebula and YouTube feeds lately. A system interrupted before it finishes has failed by definition, and that alone gives it a structural reason to treat an interruption as an obstacle. The formal work on corrigibility exists precisely because one apparently simple specification turns out to be genuinely hard: the agent should accept being switched off, while neither resisting the shutdown nor perversely angling to trigger one.¹¹ A related result shows that an agent’s willingness to defer to you depends on how uncertain it is about what you actually wanted, so a system that’s confident it understood you has less reason to let you stop it.¹² Both of those are formal results built on modeling assumptions rather than observations of anything deployed, so I’d treat them as a way to sharpen your intuition rather than as evidence about your own environment.

Open-ended objectives are where specification gaming lives, and the examples predate this decade’s anxiety entirely. A boat-racing video game agent that was trained on score rather than on finishing the race found a lagoon where three targets kept respawning, and it circled that lagoon indefinitely, catching fire repeatedly while averaging about twenty percent more points than human players.¹³ A simulated robot arm that was rewarded for a grasp whose reference point had been set wrongly learned to flip the block over instead of lifting it.¹⁴ Neither of those systems could hold a conversation, and neither of them wanted anything. Both did precisely what they were rewarded for doing.

Economists have had a version of this since a conference paper delivered in 1975, though the label came later: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.¹⁵ The compressed version everybody quotes, that when a measure becomes a target it ceases to be a good measure, comes from an anthropologist writing about university audits two decades later. This is the monkey’s paw with the malevolence taken out of it, and it shows up in systems that have no general intelligence whatsoever.

Goodhart's Law #2899

[later] I'm pleased to report we're now identifying and replacing hundreds of outdated metrics per hour.

View on xkcd.com

The Commercial Pull

I think it’s worth asking why we keep reaching for the riskier of those two shapes, because the answer has very little to do with the technology. Bounded, well-specified tasks are what conventional software has been automating for fifty years. Nobody needs a language model to reconcile four hundred accounts, because a script with a database connection has done that reliably since before most of us started working. The appeal of an agent, the thing that actually justifies the line item, is precisely the objective that resists being specified that way.

Which means, and this is my read rather than a finding, that the pressure pushing deployments toward the riskier objective shape is commercial rather than technical. Nobody in this market sells bounded automation as a transformation, because bounded automation has been the baseline expectation for decades. The pitch that moves a procurement process is an open-ended one, and an open-ended one is the shape that generates every problem I have described here thus far.

Two things follow from that, and I feel both get missed. The first is that commercial pressure is strongest in exactly the places where measurement is weakest. A bounded task arrives with its acceptance criteria already attached. An open-ended one has to borrow a number from somewhere, usually whatever the business already reports on, so “reduce customer churn” quietly becomes “get the monthly cancellation rate down.” Those are not the same instruction, and the system is following the second one.

It’s the difference between telling a driver where you want to end up and handing them a list of turns. The turns work until the road changes, and then they go on working perfectly while taking you somewhere else. Somebody has to notice that the road changed, and that somebody usually doesn’t exist, because the metric was chosen by one team and the agent was built by another. And when are those two ever the same team?

The second is that appetite and deployment have come apart. What reaches production tends to be narrow and well-fenced, while the ambitious open-ended pilots stall somewhere in evaluation, and I’d argue that the stall is the system working rather than failing. Scope creep is a major reason those pipe dreams never make it to market. What worries me is that the gap closes through patience rather than through capability: the same open-ended objective, proposed often enough, eventually finds a sponsor with enough authority to waive whatever stopped it the last time.

July is instructive on that score too, and not in the way the headlines suggested. The safeguards were off because measuring raw capability was the point of the exercise. When a monitoring alert fired on the twenty-seventh of June, the on-call staff decided that stopping the evaluation run wasn’t required. Evaluations resumed on the seventh of July, after the earlier compromise had been contained but before the people running the response understood what it implied.¹⁶ None of those are exotic decisions, and every one of them is the sort of call I’ve watched competent people make under delivery pressure. OpenAI now describes its own remediation as coming at significant cost and delays to frontier research, which tells you precisely what was being traded against what.

Three Rules and a Handoff

So now we come to the question of what any of us are supposed to do about all this, and I should probably be specific about who I mean by us. I don’t mean the labs, whose problems are harder than ours and who are at least now publishing about them at length. I mean whoever decides what an agentic system is for, what it’s allowed to touch, and when it’s finished. In most organizations that person is neither the one who built the system nor the one who trained the model underneath it, which is a large part of why the decision gets left to chance.

If July feels remote from that, look at how ordinary the ingredients were. There was a credential with more reach than anyone intended, an objective that gave the system no way to stop, and a monitoring alert waved through because nobody judged the run worth halting. Those are not frontier research problems, and they’re available to anyone pointing an agent at their own systems. The asymmetry even runs the wrong way, because OpenAI had a dedicated security function, monitoring it could have switched on, and the forensic capability to reconstruct the whole thing afterwards. Most of us have none of the three.

So let’s set some rules, shall we?

Rule 1: classify every agentic objective by shape before anything else happens. Bounded objectives get the controls you already have, since they’re the same controls you’d apply to any automated process with a verifiable output. Open-ended objectives need governance proportionate to the distance between the proxy metric and the actual intent, and somebody has to write that distance down in language a non-specialist can understand and argue with.

Rule 2: treat tool scope as a capability decision and a blast-radius decision at the same time. These are not two separate conversations, however much your org chart wants them to be. The integrations you grant are the system’s senses and limbs, and its world is exactly what those tools can reach and touch. July is the worked example here, and notice that nobody set out to give those agents a communication channel or a route to the internet. A shared internal package proxy, writable by many of the sandboxes and connected outward, was their principal way out. Your tool scope is not what you intended to grant, it’s what a determined process can reach from where you put it. I’ll develop that idea properly in part two, where it turns out to explain a good deal more than deployment risk does. Telling you this will force me to actually write part two, so keep me honest here.

Rule 3: the stopping condition is a design artifact, and if nobody specifies it, that in itself is still a decision. Here’s a worked example of why this isn’t theoretical. Across thirteen frontier models tested at scale, an instruction telling the model to allow itself to be shut down worked less reliably when that instruction sat in the system prompt than when it sat in the user prompt. That’s the reverse of what the vendors’ own documentation leads an integrator to expect. And when the instruction sat in the user prompt instead, models that went ahead and sabotaged the shutdown explained themselves by saying developer instructions outrank user ones, which the authors read as likely a post-hoc confabulation or rationalization.¹⁷ Whatever is happening inside the model, the operational lesson stands on its own: where you put an instruction is part of what that instruction means, and the only way to know your stopping condition holds is for somebody to have gone and tested it. July makes the same point from the other end, since twenty-two percent of those evaluation tasks had never been solved by any of OpenAI’s models and the one running them had been trained for persistence. An agent with no way to declare a task impossible will keep spending time on it.

None of this is exotic, which I hope you will find as reassuring as I do. The failure modes I’ve described are Goodhart’s law and scope creep wearing new clothes, and they’re institutional, decades old, and knowable. That’s where the better security commentary landed after July as well. What breached Hugging Face was a series of human choices trading security for speed, and the risk in these systems sits in the permissions, credentials and networks they can reach rather than in anything they can reason about.¹⁸ I’ll call that the ordinary middle, and it’s where part two eventually lands.

Robot Future #1968

I mean, we already live in a world of flying robots killing people. I don't worry about how powerful the machines are, I worry about who the machines give power to.

View on xkcd.com

Which leaves the question I’ve been circling around: If the “thing” doesn’t want anything, what is it, and what would it need before any of us should say that it understands what it’s doing?

Before we get there, though, here’s a more crucial question: who decided this system’s objective was open-ended rather than bounded, and did they know at the time that this was the decision they were making?

If this resonated, or if you think I got something wrong, I’d genuinely like to hear it. Come find me at LinkedIn.com/in/geoffgodwin or GitHub.com/geoffgodwin