So if the people building A.I. believe it could cause extinction, why keep building it?
Some think the Risk is Worth It
A.I. leaders acknowledge the risk of losing control and killing everyone.
In 2023, the chief executives of OpenAI, Anthropic and Google DeepMindagreed that A.I. extinction risk should rank alongside pandemics and nuclear war.
Some say you Can’t Study the Danger from a Distance
The Second Reason:
You can’t learn to make dangerous A.I. safe without building it first - like a spacecraft, you can study safety from afar, but can’t really test it without going to space.
OpenAI’s plan is “iterative deployment”: release each model, learn from its problems, and fix them in the next one.
The idea is like getting as close to the cliff edge as possible to see what the jump looks like.
Some feel it’s Winner-Takes-All
The third and perhaps most important reason is the race.
“we are close to creating a genie that can grant any wish”.
The trouble is everyone wants to hold the lamp - it would be hugely profitable, and each company doubts the judgment of rivals to use their wishes wisely.
So they race, reasoning that if they slow down, someone else will get there anyway, so it’s better to arrive first as the “responsible one”.
Some fear even a mutual agreement would be broken in secret.
So they press on...
A.I. Making Better A.I.
You might doubt runaway A.I. is plausible.
But when the companies themselves raise the alarm, we should listen...!
A.I. firms already report signs of “recursive self-improvement“, where each model helps to build a better successor.
In July, hundreds of A.I. employees signed an open letter calling for a slowdown.
But the dynamics of the race make that hard for any single company - or country - to do alone.
A Classic Arms Race
A.I. research has the hallmarks of an arms race.
OpenAI doesn’t want to lose to Anthropic, and the United States doesn’t want to lose to China.
History offers a template for how to manage a situation like this, with rules binding all players, and enforcement everyone can verify.
Nuclear weapons are the classic case.
Treaties and verification systems haven’t eliminated the risk of nuclear war, but they have slowed proliferation, and no nuclear weapon has been used in conflict for 80 years.
Rules for A.I.
In the US, where most cutting-edge A.I. research happens, the Trump administration shows little sign of slowing A.I. development.
In its first week it scrapped the old A.I. safety rules. Now it is trying to override state-level rules, arguing caution risks losing the race to China.
Some politicians are pushing back.
California recently passed laws supporting independent assessment of A.I. systems.
Unless something changes, staffers who quit over safety will simply be replaced, A.I. models will help build better A.I. models, and each generation will grow harder to monitor and control...
Is the situation hopeless?
I hold out three hopes.
First, that more people recognize A.I. escaping human control is a bigger risk than A.I.'s water use.
Second, that governments listen to their people. In the US, two thirds say A.I. is moving too fast.
Third, that we have a good plan ready before a crisis hits.
The best plan, in my view, looks something like this: delays, transparency. and verification, to slow the race and keep humans in control.
Insiders at the world’s top A.I. companies say our current safety plan isn’t good enough.
If they’re leaving their jobs over safety fears, we should listen to what they have to say...!
Until Next time…
Higher Ground News is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Don’t Forget to Like, Comment, Restack and Share shown at the bottom of this Article and Post
"AI" isn't the problem. It's the centralized, vertical, Big Data Center structure that is being used to roll it out. It doesn't have to be that way, but currently that's the only profile that is presented as an option.
The future of Human-Machine Intelligence is in POND ecosystems:
The arms-race diagnosis here feels right, and the nuclear comparison is the place I'd slow down. When you choose the bomb as your analogy, you inevitably inherit a bomb builder's playbook: nonproliferation, containment, and deterrence. That can freeze a standoff. It doesn't ask whether freezing the standoff is the safer world, or just a more efficient one. I took that question head-on here: https://www.thesingularityproject.ai/p/superintelligence-strategy-the-perfect
"AI" isn't the problem. It's the centralized, vertical, Big Data Center structure that is being used to roll it out. It doesn't have to be that way, but currently that's the only profile that is presented as an option.
The future of Human-Machine Intelligence is in POND ecosystems:
Personal & Private
On-device & On-premise
Nodal & Networked
Distributed & Decentralized
X: https://x.com/i/communities/1695429836993712389
Discord: https://discord.gg/kNvtcjPmJ7
Reddit: https://www.reddit.com/r/PONDAI/
The arms-race diagnosis here feels right, and the nuclear comparison is the place I'd slow down. When you choose the bomb as your analogy, you inevitably inherit a bomb builder's playbook: nonproliferation, containment, and deterrence. That can freeze a standoff. It doesn't ask whether freezing the standoff is the safer world, or just a more efficient one. I took that question head-on here: https://www.thesingularityproject.ai/p/superintelligence-strategy-the-perfect