The Number Nobody Argued With

Jacob Coxon resigned from Anthropic on Tuesday 8 September and posted his reasons in public. Within a day the argument had sorted itself into two camps: he's brave, or he's grandstanding.

Both camps were looking at the wrong thing.

Coxon's own words were blunt enough. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote. "This is not a marketing stunt." A second safety researcher followed him out days later. But a resignation is still one person's opinion, and people have opinions.

What came next was a different kind of statement.

Evan Hubinger leads alignment science at Anthropic — the team whose whole job is working out whether we can supervise systems smarter than ourselves. He said publicly that Coxon was right. Then he put a number on it: greater than ten percent that AI kills every human being within the decade. He added that the company does not currently have a plan for aligning superintelligence.

That is the part worth sitting with. Not the resignation. The number, from the person paid to know, said out loud and left standing.

What a number like that normally does

Think about what ten percent means anywhere else.

If the safety lead at a chemical plant told you there was a one-in-ten chance of an event that killed everyone inside the fence line this decade, nobody would treat it as an interesting position. The plant would stop that afternoon. You don't debate a number like that and then act — you act, and you work out who was right later.

Here it became content. A news cycle, then a fight about the news cycle.

I can't tell you whether ten percent is the right number. Neither can most of the people who spent last week sounding certain about it in either direction. It is an estimate about something that has not happened, made by someone with every reason to be careful and some reason to be dramatic.

But you don't have to settle the number to notice how badly it got handled.

Six days to a team sport

By the middle of last week it had stopped being a question about machines at all. In Washington, one side was demanding a classified briefing on frontier model testing; the other was warning that doing anything at all would mean losing the race to China. The House had four working days left before campaign recess.

Once a factual question turns into a marker of which team you're on, it stops getting answered. That isn't a prediction. It's just what happens.

The useful answer came from the boring corner

On Saturday 12 September, Anthropic's CEO Dario Amodei published an essay arguing that within six to twelve months a swarm of agents could take over the internet with a persistent botnet — hundreds of billions of dollars in damage. Sam Altman and Elon Musk both publicly backed the warning, which is not a thing that happens.

Then the security researchers pushed back, and not the way you'd expect.

They didn't say the risk was small. Artem Dinaburg at Trail of Bits pointed out that the incidents we have actually had have generally been security incidents. Nidhi Aggarwal at HackerOne put it plainly — of course it can be very dangerous, but we can solve the problem — and traced the recent failures to a basic lack of oversight.

That is not a dismissal. It's a to-do list.

Two things at once

I build with these tools every day. The distance between what they do in my hands and "takes over the internet by spring" is enormous. They are remarkable and they are brittle, often within the same hour.

Holding both of those at the same time is the adult position. The people who can only hold one — either one — are the ones I'd read last.

And here is the unglamorous part, which is usually the tell that something is real.

You cannot do anything about ten percent on Monday morning. You can do something about an agent holding credentials it was never meant to have, a tool with write access to a system nobody has reviewed, an automation nobody can switch off because nobody remembers who built it.

That is the version of this problem with your name on it. It is also the version that stays true whether Hubinger's number turns out to be ten percent or zero.

Start there. The metaphysics will still be waiting next week.