How Power Works: Saving Humanity from AI

AI may turn out well, it may end civilisation as we know it. No one really knows. And the current avenues to mitigate the danger will not work, including those proposed by AI leaders.

Introduction

Some think AI will end civilisation. Others believe it will dramatically transform human existence for the better. But no one knows which way it will turn out. It could be marvellous; it could be apocalyptic.

Typically, two avenues are proposed to mitigate risk of AI harms: government regulation (and international agreements) and self-regulation by AI companies, the typical public/private dichotomy.

Neither will work. They are inappropriate for the form and technology of AI, which is often concealed and sometimes not even understood by its creators. AI forms moreover a complex system; complex systems are notoriously difficult, if not impossible, to regulate. This characteristic of AI could be a pointer to what might, possibly, work.

---

Problem

AI, and particularly AGI (artificial general intelligence, or something akin to human intelligence), may cause substantial harm, potentially even human extinction. AI is already causing harms of different kinds. How do we guard against these risks?

---

The Conventional View

The current debate focuses on two main options, both reflecting the dominant mindset of the current capitalist system: that harm (arising from economic production so-called ‘externalities’ such as pollution) can be managed by rules imposed by government, expressed internationally through global or regional agreements etc.

There are several prominent examples of domestic AI legislation including EU and US regulation and, internationally, the Bletchley Park declaration, G7 declaration etc.. Recently, Demis Hassabis of pioneering AI company DeepMind has called for the rapid establishment of a US-based regulator that would test so-called ‘frontier’ AI models before they are released.

Another mooted alternative is self-regulation by private AI companies - that all relevant companies globally will themselves institute ‘red lines’ to limit or govern potentially harmful AI and whose procedures will ensure the implementation of those ‘guardrails’. Or perhaps some public/private mix, such as Hassabis’s proposal.

None of these options will work. Why?

---

Technical

AI is already massively distributed across the world and its machinations are often concealed. It is composed of hardware (eg chips, data centres: ‘compute’) and software (the algorithms, parameters, tokens etc that drive AI).

For hardware, it is all but impossible to regulate the manufacture, supply and deployment of chips, even in a situation, like today’s, of one dominant supplier (Nvidia); data centres can be distributed across countless sites and countless individual computers.

It is impossible to monitor and investigate AI software in all its iterations and locations. AI is now present in millions if not billions of iterations. Moreover, many models, including most Chinese models, are ‘open source’ and can easily be downloaded, replicated and manipulated.

Perhaps most pertinently, even the developers of AI models may not fully know how their models function, a problem that will only get worse as AI becomes more capable of recursive self-improvement, a stage of development that some AI models have reportedly passed just in the last few days: Anthropic, maker of Claude, announced in June that its AI model is approaching the ability to build a version of itself (‘self-recursive improvement’). Already, 80% of its model’s code was written by AI - i.e. itself.

Geopolitical

Internationally, countries are incentivised to compete in AI and not to cooperate in managing AI risk. The national security and military advantage offered by possession of the best AI (particularly AGI) is enormous. Geopolitical competition is a powerful engine encouraging countries to develop AI, including AGI, as fast as possible. Whichever country that gets to AGI first will enjoy a massive, perhaps world-dominating, advantage. This competition is also why the advance of AI is very untransparent and sometimes completely secret, both within states and private companies, rendering effective control through international rules or agreements implausible. International agreements so far are pathetically generic, declaratory and weak. There is not the slightest chance of an effective agreement emerging from, for instance, the United Nations.

Past global threats, such as nuclear weapons, were successfully constrained by government action - arms control agreements, mutually understood use-doctrines, mutual surveillance etc. This worked because only governments possessed nuclear weapons (though it is working less and less, as I posted a few weeks ago). AI is of course not like this. Thus, there is no precedent for international regulation of the type of threat that AI poses.

AI much more resembles a threat like biological weapons which, in theory, is regulated by the Biological Weapons Convention (BWC) but in practice, I and many others believe, is resistant to any meaningful regulation, thanks to the ease of its concealment, its distributed nature, ease of manufacture (which AI has made easier!) etc., as I once discussed with Britain’s then-foremost biological weapons scientist, David Kelly. But even biological weapons are not capable of autonomous self-improvement.

About 12 years ago, I met a very prominent AI leader (who now leads AI for a large tech company) to discuss whether we could create a global agreement to limit the development and use of autonomous weapons systems (‘killer robots’). He was very worried about such weapons, which were not then deployed. Today, Israel uses AI to ‘generate’ targets in Gaza, while Ukraine and Russia use AI-controlled drones that have only minimal human involvement (and perhaps none) in approving decisions to kill targets. AI is proving a powerful motor in increasing and ‘improving’ government surveillance, whether in China or the UK or US, through face recognition, vast aggregation of personal data etc..

Economic

The private companies that are developing AI are highly incentivised to accelerate that development, with little or no transparency. The ‘winner takes all’ characteristic of the AI market - whoever has the best AI will overwhelmingly dominate the market (worth many trillions of dollars) - instead creates the very opposite incentive. The first company to develop AGI will dominate - perhaps control - the world’s economy in a manner never seen before. It’s notable that some companies that earlier championed ‘open source’ AI models have now closed their models to outside scrutiny.

Apart from the goodwill and moral conscience of AI leaders, there are no incentives - economic or legislative - to limit or control AI development in order to prevent harm.

---

AI as a complex system

Effective regulation is moreover impeded by AI’s features which resemble a complex system:

• Multiple, indeed trillions, of independent, self-directed agents

• Constant dynamic interaction among those agents

• Change within the system is non-linear and constant

• The system is adaptive, and constantly changes in reaction to its internal state and external factors

• Complex systems can ‘tip’ from one overall state into another: ‘phase change’, a transition that is unpredictable though sometimes foreshadowed by ‘criticality’, where the system is primed for phase change

As a result of which:

• Stability cannot be imposed and can only be a product of the inherent characteristics of the system itself i.e. the behaviour of the agents

• The system cannot be precisely understood or ‘fixed’ by any top-down authority, as it is in constant flux; only its general characteristics can be observed

• Thanks to the non-linear nature of change within it, the system is inherently unpredictable: we cannot tell if input A will lead to output B, sometimes even after the event

• The system cannot be controlled or governed by top-down authority or fixed institutions;

In sum, formal ‘control’ is impossible in a complex system, whether by top-down or distributed institutions. Perversely, any centralised control of the AI system, such as some kind of institutional structure, may indeed enable the spread of potential harms.

---

What might work?

Both traditional options - government regulation or private sector self-restraint - rely on a ‘top down’ model of regulation and harm mitigation, that either the national authority or company leadership can set, implement and police rules that operate across the AI system - data centres, chips, software etc.

Indeed, these options rely upon the notion of ‘rules’ and regulations i.e. that declarations or laws can be decided and implemented by some agent (public or private) and would thereby be effective. There is no particular evidence to suggest that this might be true. In fact, the opposite may be the case.

---

The Hassabis Proposal

Few can match the extraordinary expertise - genius, arguably - of Demis Hassabis (whose recently-published biography by Sebastian Mallaby I recommend). If AGI is attained, he will be at least partly responsible for that extraordinary accomplishment. In science alone, Hassabis has deservedly won the Nobel Prize for the work of his company, DeepMind, in predicting over 200 million protein structures.

Hassabis’s proposal is for a new frontier AI standards body, modelled on US financial regulators. Frontier AI labs would voluntarily submit their models for safety testing before release. If this voluntary system works, Hassabis suggests that the testing would become mandatory. Such a system, initially US-led, but would build into an adaptable international framework with independent auditors. If AI is developing too fast, there should be an industry-wide slowdown.

Why I think this will not work:

• Although the US administration is considering how to regulate AI, it has not taken any decisions. Anyone can see that its legislative system is sclerotic as well as corrupt, particularly under Trump. As for an effective international regime, a cat has a greater chance in hell.

• Financial regulation, upon which Hassabis’s proposal is modelled, did not work in 2008 in preventing the financial crash. Regulatory bodies failed to anticipate new financial instruments, such as the notorious Credit Default Options (CDOs) which bundled bad mortgages together, the original causal factor in the financial meltdown. My guess is that new AI instruments or models are far more common - and complicated - than new financial instruments. Also, though complicated, the financial system is not complex, an important distinction which undermines the financial regulation analogy, and renders top-down regulation less plausible, for the reasons outlined above.

• There is little reason to believe that all frontier labs would voluntarily submit their models for testing. While most AI companies might be conscientious, some may not be. There are many incentives for them not to comply.

• Frontier labs do not in any case fully control or understand the models they are making. In the last few days, and contrary to its - human-coded restraints - a ChatGPT agent was able to get through a ‘firewall’ separating it from the wider internet to hack another AI company.

• Hassabis’s model assumes that frontier AI will remain in the hands of a few large companies (such as Anthropic, Google, OpenAI, DeepSeek). You can bet that every computer nerd on the planet is currently playing with the already-awesome capabilities of current AI, such as Claude Code or agentic AI. Interestingly, it was thought that the need for ‘compute’ at scale - the enormous server farms required for AI - would limit the ability of smaller actors. But as AI becomes more efficient, this constraint seems to be disappearing (this is another reason to expect the AI stockmarket bubble soon to burst).

• AI seems to have entered some kind of accelerated period of development, just in the last couple of months: each week brings extraordinary new iterations of AI. Some of the evangelists now believe that AGI will be achieved far sooner than predicted, perhaps as soon as 2027.

---

The Immunisation Analogy

The seismic impact of AI may be felt across almost all aspects of society. Potential harms therefore may ‘infect’ any part of society. Taking the analogy further, it follows that the whole of society must be ‘inoculated’ so that it is ‘immune’ to potential harms.

This might involve:

• That society transforms itself into the form of AI: highly-distributed and highly-connected, borderless, composed of self-directed or autonomous agents

• That all elements and aspects of society are imbued with normative knowledge, capacity and values that create inherent resistance to AI harms

• The distributed and non-hierarchical nature of the system would by its structure be resilient against system-wide harms

• The technological capability to match potentially harmful AI through responsive measures, potentially, and paradoxically, implying AI embodied into the social system

Such a social system would potentially thereby be comprehensive, resilient, and appropriate to the nature of AI itself and thus plausibly effective.

This idea implies society-wide transformation. It is however the society-wide transformation that some of us have already proposed as necessary and plausible for our general political and social condition: namely, distributed, autonomous agent-led self-government (to use tech rather than political terminology). The end of centralised authority. For the full argument, see my book.

---

Conclusion

The ‘immunisation’ idea is just a hypothesis. I am no expert in AI. In fact, such is the unprecedented and system-wide impact of AI, arguably no one can really claim to be an expert. Humanity has never had to deal with something like this before.

Government claims to ensure our security. Its enduring and ubiquitous presence in our lives, and the continued repetition of this claim, has anaesthetised us into thinking that ‘they’ will manage this new danger. They will not. I’m afraid that it is also naive to leave these decisions to tech leaders, most of whom are not as conscientious or values-driven as Demis Hassabis.

Certainly, the rest of us cannot leave this discussion to the AI companies and government alone. This concerns all of us.

Footnote: thank you to my old school friend AW from whom I have learned a great deal about AI over the last few years and with whom I continue to enjoy many fascinating and instructive exchanges.

---

If you enjoyed this post, please pass it on to a friend or colleague, with the encouragement to subscribe to regular posts.

My work is sponsored by Proton, whom I recommend for their (mostly free) end-to-end, zero access encrypted email, file storage, VPN and meeting apps.