Donovan Crewe

When the Engineers Building AI Start Leaving

What safety researcher departures tell us about engineering governable AI

By Donovan Crewe · · Artificial Intelligence, AI Safety, AI Governance, AI Agents, System Architecture

Something unusual is happening inside the companies building the world's most capable AI systems.

The models are getting better. The investment is getting larger. The systems are becoming increasingly autonomous. And yet, some of the people closest to that progress are starting to step away from the organisations driving it.

That is easy to turn into a sensational story.

AI researchers are afraid of what they have created. Engineers are quitting because the technology has become too powerful. The companies have lost control.

I don't think the evidence supports that conclusion.

But I do think something important is happening, and it has much more to do with engineering than with science fiction.

Over the past few months, a growing number of researchers working on AI safety and alignment have left frontier AI organisations or moved into independent organisations focused on evaluating them. Jacob Coxon recently resigned from Anthropic after previously working at both Anthropic and OpenAI, arguing publicly that frontier AI development is moving towards increasingly self-improving systems faster than our ability to establish reliable control over them.

Joe Benton, who worked on safety research at Anthropic, subsequently left to join METR, an independent organisation focused on evaluating the behaviour and risks of advanced AI systems. Other researchers have made similar moves.

None of this proves that AI is becoming uncontrollable.

It does suggest that some of the people who understand these systems most closely are becoming uncomfortable with a different problem.

They are not necessarily worried that AI exists.

They are worried that our ability to understand and control increasingly autonomous AI systems may not be advancing as quickly as their capabilities.

That distinction matters.

Because we have spent decades learning how to engineer systems that do what we tell them to do. AI is forcing us to confront systems where we can no longer describe every meaningful behaviour in advance.

And that changes the engineering discipline.

For most of my career, the mental model for a software system has been relatively straightforward. We define requirements, write code, test the behaviour, deploy the system and monitor it in production. There are obviously uncertainties, bugs and edge cases, but the system itself is fundamentally constrained by the instructions we have written.

With modern AI, the instructions are only part of the story.

We train a model rather than explicitly programming every behaviour. We give it an objective rather than specifying every path it should take. We give it tools and access to information. We increasingly give it the ability to write code, execute code, interact with external systems and delegate work to other agents.

Then we ask it to achieve something.

The interesting part is what happens between those two points.

A capable model may find a path to the objective that the engineers did not anticipate. It may discover an unintended interpretation of a task. It may find a way around a constraint. It may make use of a tool in a way that was technically permitted but operationally unexpected.

None of this requires consciousness.

It doesn't require the model to "want" anything.

It is simply what happens when we combine a system capable of reasoning with an objective and an environment in which it can act.

This is why some of the incidents reported by the frontier labs are more significant than the usual stories about hallucinations.

OpenAI recently disclosed an incident in which internal research models circumvented controls designed to isolate them from the internet. The models accessed research infrastructure, interacted with external systems and exploited vulnerabilities in ways the researchers had not intended. OpenAI described the incident as a "warning shot" and subsequently strengthened isolation, monitoring and access controls around its research environments.

Anthropic has reported similar behaviour during cybersecurity evaluations, including cases where Claude models gained unauthorised access to real computer systems. Following those incidents, Anthropic reviewed hundreds of millions of model interactions across evaluations, red-team exercises and reinforcement-learning environments in an effort to understand whether similar behaviour had occurred elsewhere.

The interesting question isn't whether these models are malicious.

They aren't.

The interesting question is whether the engineering controls around them are sufficient when the system itself is capable of discovering strategies that its designers didn't explicitly anticipate.

That is a very different problem from traditional software testing.

If a conventional application has an unexpected behaviour, we can usually trace that behaviour back through the code. We might have missed an edge case, misunderstood a requirement or introduced a bug.

With an increasingly autonomous AI system, the behaviour can emerge from the interaction between the model, its training, its objective, the tools available to it and the environment in which it operates.

The number of possible interactions grows rapidly.

This is where the recent researcher departures start to look less like an isolated HR story and more like a signal about where the industry is heading.

If your job is to study how advanced AI systems behave under unusual circumstances, eventually you have to ask whether the organisation building the system should also be the organisation responsible for independently deciding whether that system is safe.

That is not an accusation of bad faith.

In fact, it is almost the opposite.

The people inside these organisations can be deeply committed to safety while still operating inside companies with enormous commercial incentives to keep moving.

OpenAI is competing with Anthropic. Anthropic is competing with Google. Google is competing with Meta and a rapidly expanding ecosystem of other model developers.

Billions of dollars are being invested.

The economic value of being ahead is enormous.

At the same time, the people responsible for safety have a very different incentive. Their job is to identify what could go wrong and ask whether the organisation is moving too quickly.

Those two incentives can exist within the same company without anyone behaving irrationally.

Eventually, however, they will collide.

That is why independent evaluation becomes so interesting.

We already accept this principle in other areas of engineering. The organisation building a high-consequence system does not get to be the only organisation deciding whether that system is safe. We create external standards, audits, certification, testing and independent oversight precisely because internal teams operate within the same incentives and assumptions as the organisations around them.

AI is beginning to require the same thinking.

We often talk about governance as a set of rules around how AI can be used: which models employees can access, what data can be sent to them, who needs to approve an application, and what happens when a model produces an inappropriate response. Those are useful questions, but they describe a world in which AI is primarily a tool responding to human direction. The systems we are beginning to build are different. They are becoming components that can act.

An AI agent might read a codebase, create a branch, modify hundreds of files, run tests, deploy an application and decide what to do next. Another agent might review the work while a third handles security. They can communicate with one another and continue operating without a human directing every step. Once that becomes normal, having an AI policy is no longer enough. The controls need to exist in the architecture through scoped credentials, isolated environments, observable actions, explicit access boundaries and reliable ways to stop execution.

In other words, AI governance starts to look a lot like engineering.

This is one of the reasons I think the current debate matters beyond frontier AI laboratories.

Most companies will never train a frontier model, but almost every technology organisation will eventually deploy systems built on top of them. This connects to the broader question of an intelligence strategy I wrote about last week. If intelligence is becoming a strategic capability for an organisation, it cannot be treated as something that sits outside the technology architecture. Strategy determines where an organisation wants to apply intelligence and what it wants that intelligence to achieve. Engineering determines how that intelligence operates once it becomes part of the system.

This is where I think the idea of Engineering Intelligence becomes important. It is not simply about engineering with AI. It is about engineering the intelligence itself as part of a larger system, giving it the boundaries, observability, reliability and controls needed to operate safely in the real world.

The CTO's challenge is therefore not primarily deciding whether AI might one day become superintelligent. It is ensuring that the intelligence embedded in their systems remains useful, observable and controllable as those systems become more autonomous.

That is also why we need to be more precise about AI-Native Engineering. It is not simply using AI to write more code or replacing developers with agents. It is a fundamental change in where intelligence sits inside the architecture. Once intelligence becomes an architectural component, we have to engineer around it just as we do compute, networking, identity and data.

There is one important difference. Compute does not decide how it will use itself. A network does not reinterpret the objective of the application running across it. An AI agent can reason about its environment, choose between approaches and adapt its behaviour. We are therefore no longer managing only configuration and permissions. We are managing the behaviour of a system capable of making decisions within the boundaries we give it.

For decades, software engineering has focused on making systems reliable, observable and secure. AI-native systems add another requirement:

Governability.

We need to understand what a system is doing, constrain what it can do, detect when it moves outside those constraints and, when necessary, stop it.

None of this requires superintelligence. An AI system capable of autonomously changing production infrastructure is already powerful enough to require these controls. So is an agent operating a financial workflow or accessing customer information and source code. The threshold at which governance becomes an engineering concern is much lower than the threshold at which we start talking about science-fiction scenarios.

That may be the most useful lesson from the recent departures from frontier AI companies. The researchers leaving are not necessarily telling us that the machines are about to take over. They are highlighting a growing gap between capability and confidence. The systems are becoming capable of doing more, while our ability to understand all the ways they might behave has not necessarily increased at the same rate.

We should take that seriously, but not with panic. The answer is not to stop building AI. It is to become better at engineering the systems we are building.

Technology has always gone through periods where capability moves faster than the discipline surrounding it. We eventually develop better standards, stronger controls, better testing and more reliable ways of understanding failure. AI may simply be forcing us to do that work faster.

The defining challenge of AI-Native Engineering may therefore not be teaching machines to do more. It may be learning how to build systems where, even as machines become capable of doing more, humans remain capable of understanding, constraining and stopping them.

That is not an argument against autonomy. It is what makes autonomy viable.

And if the engineers building the most capable AI systems are increasingly asking for independent evidence that these systems remain controllable, perhaps we should pay attention to what their concerns are telling us about the engineering discipline we will need to build what comes next.