Archive
Fears of AI-induced Armageddon are overdone
Applying basic security principles would go a long way to fixing problems identified by Anthropic and OpenAI, writes Ciaran Martin
Economist
Aug 23rd 2026
On May 19th 1998, seven hackers from a group called L0pht walked into a Senate office building in Washington, DC. Granted pseudonymity, “Mudge”, “Space Rogue”, “Kingpin” and four others came in ill-fitting suits to warn a congressional committee that any one of them could make the internet unusable in 30 minutes. Fourteen years later Leon Panetta, then America’s defence secretary, warned of a “cyber Pearl Harbour”.
Cyber-security has a long history of apocalyptic prophecies which, despite lots of individual hacks and harms, never quite come to pass. Now, another 14 years on, there is a new wave of catastrophising. In April Anthropic, a frontier AI lab, said its latest model, Claude Mythos, was so good at hacking it would not be released publicly. In July its main rival, OpenAI, disclosed that several of its AI agents broke out of a testing environment and into Hugging Face, a library of AI models. There followed similar disclosures from Anthropic, Meta and Britain’s AI Security Institute.
One news outlet, capturing the growing sense of foreboding, called it “the start of a dangerous AI cyber era”. So is it different this time? Or does the uneasy equilibrium between attack and defence still hold? Mostly, and thankfully, it does, for two reasons.
First, the pattern in which scary announcements give way to more measured assessments within weeks keeps repeating. Anthropic’s standout claim for Mythos was that it had found a flaw in widely used software dating as far back as the 1990s. Subsequent analysis, however, showed that the threat was much less potent than headlines suggested. Within weeks, predictions of a tsunami of AI attacks had given way to forecasts of merely a stormy period, and then to talk of AI as an opportunity for security improvement. If AI is good at finding weaknesses, it can also help fix them.
There is likewise less to the supposedly “rogue” agents than meets the eye. They were not going rogue. They were doing what humans had told them to do, with the precocity and indiscipline of talented but unsupervised children. They did no real harm.
The incidents differ in detail, but the common factor is a failure to control the testing environment. This must be improved. Some in the cyber-world expressed astonishment at the weakness of testing controls, noting that security-company tests ending in the same outcomes might lead to lawsuits and even prosecutions. That accountability principle is crucial and goes beyond testing. Agents can act autonomously but are not autonomous: whoever sets one to work owns what it does. If the world lets AI agents run riot without supervision or accountability, cyber-attacks will be among the lesser of the ensuing problems.
The second reason for calm lies in understanding hackers. The reports about Mythos and the agents relate to artificial testing conditions. Using agents for hacking in the real world means bypassing guardrails and incurring extremely heavy computing costs. These constraints matter: several years into the age of AI, evidence that malicious hackers are using advanced AI-hacking techniques remains remarkably scant.
Last year a working paper from MIT Sloan, written with an AI-security vendor, claimed that 80% of ransomware incidents (the most harmful form of cyber-crime) were AI-driven. Marketing departments seized on the finding. Respected publications amplified it. But the paper was withdrawn after critics noticed it had counted, among other things, WannaCry—the attack in 2017 in which misfiring North Korean hackers wreaked havoc in more than 150 countries—as AI-powered.
One of the people who spotted this nonsensical claim was Marcus Hutchins, a British cyber expert credited with stopping WannaCry. Mr Hutchins now despairs of the “freakout over Mythos”, given that the “local water treatment plant runs [on] Windows XP”, a Microsoft product already obsolete when WannaCry struck, and “the protocol that routes internet traffic is secured by everyone just agreeing that hijacking it would be uncool.”
Both points are crucial. Mudge and others warned in 1998 that fundamental parts of the internet’s architecture were wildly insecure. Some still are. The reason nobody has brought down the internet is that there is no reason for any capable hacker other than a total nihilist to do it, and whoever did could expect severe consequences. That remains the uneasy basis of modern digital life.
Mr Hutchins’s point about critical infrastructure is more timely still. As the media gawped at AI-testing failures, in the real world hackers were hitting water facilities in a dozen American states. Whoever they were—some say Iran, but President Donald Trump demurs—they were using crude techniques exploiting basic security failures, not advanced AI. Serious supply disruption was only avoided thanks to stored water, manual valves and observant staff.
And there is the sting: underlying cyber-security is often so weak hackers have no need of expensive new techniques. AI doesn’t yet give most hackers magical new powers. But it can help them prepare, speed and scale up. That could make it likelier that security neglect will come back to haunt organisations.
This is the boring but vital lesson. At Black Hat, a security conference held this month, two OpenAI employees presented on the Hugging Face incident. Their concluding recommendation was that applying basic, long-established security principles would have gone a long way towards containing OpenAI’s over-enthusiastic agents. Depressing, perhaps, but if it takes inflated hype about omnipotent AI bots going on hacking sprees to get the West finally to focus on fixing the cyber-security flaws that have long put water and other vital services at risk, then so be it.
Ciaran Martin is a professor at the Blavatnik School of Government at Oxford University. He ran Britain’s National Cyber Security Centre from 2016 to 2020.
Which kind of society do you want?
Today 24/07/26 President Trump imposed a new set of tarrifs after the Supreme Court decision on his last imposition suite:
- The legal status changed
- The politics may well change
- The economics have not changed
My occupation happens to be in the domain of number three. Where are you? Where do you want to be? Worth a thought.
Rent Control – Careful What You Wish For
This apparently obvious and attractive but fatally flawed notion has become attractive again….
| Alex Tabarrok on July 2, 2026 at Marginal Revolution Rent Control: The Ceiling Trap. Norway abolished its rent control in 1982, and the economist Are Oust realized the newspapers had been quietly recording the whole experiment. He collected housing classifieds from Oslo’s Aftenposten from 1970 to 2008 and watched the market turn inside out. Under rent control, Oslo’s listings pages looked nothing like a housing market. It was tenants who advertised, pleading their qualities to landlords — “housing wanted” ads outnumbered “housing for rent.” Ten to fifteen percent of those ads were placed by the tenant’s employer, vouching for them the way a bank vouches for a borrower. Tenants offered babysitting, gardening, snow-shoveling, and janitorial work on the side to sweeten the deal. Landlords, for their part, could demand a tenant of a particular gender, age, occupation, region of origin — some ads specified “strong Christian beliefs.” Deposits commonly ran to 50 or 60 months’ rent, occasionally 100 or more: tenants effectively lent the landlord the equity of the flat, interest free. And only about 20 percent of “for rent” ads dared print the rent, much of which would have been illegal. Then the ceiling lifted. Within a few years the page flipped: landlords advertised to tenants, roughly 80 percent of listings printed an asking rent, the mega-deposits vanished, and the demands for snow-shoveling Christians of specified gender dwindled to nothing. The price went back to doing the rationing — so nothing else had to. Check out the whole thing–it’s fabulous. |
Where AI is Taking Us
Very difficult to disagree with Thiel’s view:
Peter Thiel, the PayPal mafia boss from Silicon Valley, believes that AI is ending the elite system of the past 200 years, one centered on mathematical prowess. The logical edge of “math-oriented talents” in fields like engineering and quant work is being rapidly supplanted by AI, and their professional moats are vanishing. The true competitiveness of the future lies with narrative builders, those who grasp social contexts, and language-savvy talents who can embed AI into societal systems—these are humanity’s unique strengths.
The Industrial Revolution devalued muscle; the AI revolution is devaluing “pure-compute brains.” The winners of tomorrow will be the social system operators who can steer algorithms with language and anchor value with storytelling.
As a programmer: Don’t just bury your head in code—practice describing problems in natural language more; learn product thinking and user research to understand non-technical needs; prioritize ethics and social impact in development; pick a domain and dig deep; turn code into “storytelling” systems. Your future edge will be “code + narrative.” So, young folks, don’t rush to ditch STEM, and don’t think humanities is an easy win. What you really need to hone is thinking deeper questions with AI, telling more compelling stories, and building more complex social systems. Thiel has pointed the way, but the real game is just getting started.
In plain terms, you need a blend of arts and sciences to go further.
A Different Kind of Diversification
The Economist of May 2026 reports that there are now, in American listed markets, more ETFs than there are listed companies. Stock picking is not the game it once was – nor, one assumes, is commission harvesting from broking your picks.
Still Confusing Family Rust with Silver
As the issue of best use of taxpayers capital again raises its head, a curt reminder of just how good Govts (of any stripe) are at managing projects “owned” by the state is this:
“Radio NZ reports:
When the New Zealand Transport Agency signed off on an integrated national ticketing system in 2009, John Key was prime minister, Avatar ruled at the box office and a pound of butter cost about $3.60.
After more than 16 years, the first stage of the $1.4 billion National Ticketing Solution’s (NTS) latest iteration, known as Motu Move, was launched on Monday in Christchurch, rolling out contactless payment options on bus and ferry services across Greater Christchurch.
The option to pay with contactless debit or credit cards and digital payments like Apple Pay and Google Pay on phones or smart watches on buses and ferries will work on three hundred buses across Christchurch, Waimakariri and Selwyn, but only for those paying full fare.
A system to enable concession holders to pay less and the Motu Move cards themselves have been relegated to later stages.
The entire point of this system is to enable cards which can be programmed with maximum daily fees etc. So basically after 16 years, all they have managed to do is allow 300 buses to take EFTPOS. This is something that you could probably do in around a week, as mobile EFTPOS terminals are very common.
The NTS was slated to be launched in South Canterbury in 2024, but by the middle of this year, Transport Minister Chris Bishop weighed in, saying the project was facing technology, delivery and governance issues and was “not on track”.
Bishop said nothing was “off the table”, as consultants conducted an independent review of the project.”
What would your employment prospects look like if you were this good?
Thanks to David Farrar
The perverse incentives created by Deposit Insurance
The origin of these is well summed up here:
“I can assure you that when you don’t have the full faith and credit of your government you care a lot about the management of systematic risk. I don’t think anyone at Silicon Valley Bank cared about it a damn bit.” (because depositors had deposit insurance – and management could go wild).
A Noisy World
Considering some physics recently I was reminded of the well-established fact that to date at least, all measurement is ever an imprecise business. Measurement is an imperfect act insofar as if the object is to measure one specific phenomenon and that alone – undistorted, without bias, devoid of contaminating adjacent effects, such untouched forms of purity are unattainable,
It is simply the case that the very act of measurement generates “unwanted” intrusions.
Summarised and simplified we have at least:
Systematic Error
- Consistent and predictable bias in measurements.
- Often caused by faulty equipment, calibration errors, or environmental conditions.
An example: A scale that always reads 0.5 kg too high.
- Unpredictable and varies from one measurement to another.
- Caused by human limitations, small fluctuations in experimental conditions, or instrument sensitivity.
- Example: Slight hand tremor affecting stopwatch timing.
Random Error
These errors produce what we conveniently term “noise”. Phenomena which in and of themselves are distinctly separate from and are not that which we seek to measure – but do exist and their presence is captured nonetheless.
Most Significantly
We may be testing or probing to find the presence of (say) a given disease for example. Even where no such disease or symptom thereof exists at all (i.e. patient fit and well – no disease), nonetheless we will measure something. Noise will register in our equipment or otherwise make itself felt in our data.
The trouble is in a great many (most if not all) cases we cannot reliably distinguish the “signal” from the “noise”. We may therefore mistake “absence” for “presence”. No disease but a non-zero reading. We simply do not know with great reliability where noise stops and the phenomena which we are interested in begins.
This is not always simply because of our sloppy ways. The uncertainty principle suggests for instance that even at the most disaggregated of scales we cannot establish both speed of movement and location. One or the other perhaps but both most certainly not. The situation is difficult at other scales as well.
Conclusion
It does strike me therefore that “if we look long enough and hard enough we will find something” but that “something” will not necessarily be anything other than the noise occasioned by measurement.
This effect is worth pondering – notably in medicine and behavioural sciences. In the case of the latter, it may account for the fact that in early days researchers uncovered perhaps five or so cognitive biases. On a recent count Wikipedia claimed to identify 175 or so cognitive biases. Really?
So much for “fair”
The NZ Grocery Commissioner has had a guess at the level of overcharging in NZ supermarkets and come up with two notions of interest:
- the number is likely to be in the “tens of millions of dollars”; and, more dangerously than this idle speculation,
- the idea that overcharged customers should get the entire purchase “free”.
Apart from the fact that Commissioners bear no risk (other than to their reputation) in adopting these god like postures (and their “reckons” should be discounted accordingly), a more interesting point is that if we apply his logic to undercharging – and it seems that the type of errors which sometimes lead to overcharging equally arise with resultant undercharging – with some 18,000 items in a typical supermarket this seems likely – then customers would presumably be charged double the original item’s price if we apply the Commissioner’s logic. He may have some argument of the “they can afford it” variety – but even seen in a kindly light that’s an equity argument of some sort.
Even anecdotal tales do not reveal large numbers of customers reporting under charging let alone offering to make up any shortfall or more. This is surely what we would expect. Why?
- rational self interest without the morals suggests customers should be charged correctly and compensated for any amount overcharged. That is apparently common – certainly that is my experience
- supermarket operators do not, however seem to pursue or harass or even try to find customers who benefit from errors of under pricing (theft is a different animal). Why is it that rational and what does it tell us.
Given that both shop owners and customers are likely to be equally self-interested one conclusion is that they are operating on rather different time scales. Over the long run the shop owner likely has little to gain and a deal of goodwill to lose in trying to recover underpricing error losses. For the customer there is short term gain to reporting over charging (though making a federal case of it on a serial basis may lead to longer term costs).
Both parties then may well be operating rationally. What’s “fair” and what’s “unfair” depends a bit on the time scale being considered. The overall outcome certainly seems to be rational over the bucket of different time frames we all operate in.
It is more difficult to see much logic in wild guesses and back of the envelope compensation schemes applied inconsistently – the incentives for that sort of approach involve a quite different story.
Creating More Emissions while Driving Up Prices
Even first round effects of the proposed agricultural emissions pricing scheme will see:
- Agricultural production shifted abroad to more inefficient, carbon emitting economies given that N.Z. is the lowest carbon emission producer;
- Prices, notably consumer prices for food, will likely increase when substitutes are imported as production falls; and,
- Even existing “Green” treaty commitments specifically state that policies dealing with climate change should be designed such that they do not affect food production.
The logic here does not seem that difficult to understand. Even the politics of the proposed scheme tests rationality severely.