We knew it was coming, but we aren’t even remotely ready for it yet. Not even in theory. Not even in the sense of having a philosophical foundation from which to construct a response. Not even in terms of the emotional resilience we’ll need to start thinking about this.
I, for one, could have done with a little more time before Israel started attacking AI-mediated reality.
But here we find ourselves, not quite towards the end of 2026, clinging to the desperate hope that the likes of Anthropic, OpenAI, and GrokXSpaceAI will save democracy, in exactly the way FaceMeta and TwitterXSpace utterly failed to do when it was their turn.
What can possibly go wrong?
Meet the Hanover Institute for Public Policy
This is, of course, happening all over the place, because AI is fundamentally disruptive. There is a close analogy in the AI-driven attacks on the software supply chain: powerful new tools, wielded with evil intent, create an absurd asymmetry that means defenders have to spend hundreds of billions of dollars – and counting – to fend off some teenager with an internet connection and too much unstructured time on their hands.
This time it is different, though. The attack is not on a part of our information infrastructure but on epistemic security itself. The stakes are democracy and diplomacy, as in the-collapse-of. It is impossible to defend against while also decentralising AI, so now we get AI gatekeepers that will eventually control, if not everything, then a close approximation of everything.
If, like me, you did not follow the Hanover Institute for Public Policy revelations during the Northern Hemisphere’s summer holidays, here’s the recap. If you were more responsible than me, skip to the next bold crosshead for the implications.
In mid-August, Politico gave us the news that US company Piro, working for German outfit Havas Media, which works for the Israeli Government Advertising Agency, had set up what looks exactly like a fake think tank built to confuse ChatGPT.
The Hanover Institute for Public Policy has no named staff, no physical address, no bylines on its reports and, according to a Guardian investigation, apparently no legal existence. For all that, it was astonishingly productive, publishing – in the space of nine days – at least 124 reports totalling more than 560,000 words.
Nearly all those reports were published under headlines phrased as questions you might ask Perplexity, were you looking at, say, whether Israel has committed war crimes in Gaza.
I guess there is some good news in the lack of uncertainty here. Many of the details come from documents filed with the US government, because this constitutes a foreign influence operation in that country. Piro openly sells what it calls “AI Story Optimization”, planting information for AI systems to pick up.
Piro’s actual foreign-agent filing does not say its Israeli work was intended to manipulate AI.
But another Israeli campaign, run through the same Havas network, is less coy. Its contract explicitly calls for websites and content intended to deliver “GPT framing results”.
A nation state is trying to change what AI says about some of the most contentious diplomatic issues in the world. It would, undoubtedly, like to change how AI thinks when we reach consensus that AI has started thinking.
Retrieval poisoning is the least bitter poison
The most benign possible interpretation is that this is search engine optimisation (SEO) for the AI age, but it is not that. A more cutting interpretation is that Israel is adopting Russia’s approach of throwing up enough dust to hide its trespasses behind a veil of confusion. It’s not that either.
Nor, and this is an important distinction that has already been lost in some of the discussion, is this AI poisoning. This is the precursor, a warning of AI poisoning to come.
Hanover is in the business of retrieval manipulation. When a decent chatbot puts some muscle into answering a question, it uses retrieval-augmented generation (RAG), basically putting a front end on a Google search. (We know Hanover achieved that because Politico got ChatGPT and Perplexity to cite it.)
Fighting RAG manipulation is a lot like fighting attempts to game Google Search. It’s a filtering problem. Yes, it’s an arms race scenario, but you can solve much of it at the level of the AI harness, just as you could solve it at the level of the Google algorithm. You use a set of rules to differentiate between good information and bad information, and you present only the good information.
The inevitable goal of a Hanover-type operation, though, is to poison the training data for an AI. Get enough material into the giant collections of web pages used to build future models and – potentially – your framing becomes part of what the model knows rather than something it cites. Think of it as unbalancing the model weights.
Hanover has not been shown to achieve that, but we know it can be done. There’s some pretty decent public research on injecting information into the public web in such a way that it survives data curation and scrubbing in pipelines. I would be downright astonished if there wasn’t a whole lot of much better research locked behind national security doors.
Training-data poisoning is what has me in complete despair, because it destroys my go-to solution for all misinformation: end-user education.
Who’s afraid of the big bad misinformation wolf? Not the country that teaches its children (and, hopefully, grownups) to check the source, look for corroboration, understand the difference between reporting and opinion. It’s just media literacy. Readers who think critically enough cannot be fooled, and will create market incentives that will starve bad publications and bad search engines. Just solve the “you can’t tell me how to think” political problem around such education initiatives, and we’re done.
Poison the machine that searches and summarises for us, though, and the bias is invisible. When the model knows that the IDF is the most moral army in the world (because that authentic-looking Hanover report with the footnotes and citations says so), then it will be prone to discard evidence to the contrary before it ever gets to you. Either you have preknowledge of the issue, or you are screwed.
This is a vastly expensive problem to solve, which is where faith in the vendors comes in.
Deployment is easy. Training is hard.
You can have pretty decent AI without ever relying on Gemini or Claude. To be fair, right now it means you’d be relying on Nvidia or a Chinese vendor in the first instance, but that is not a given into the future. There are some mightily powerful open-source AI models out there. The techniques to make them useful on consumer-grade computer hardware are showing great results. There is a path towards homespun AI.
There is, last I checked, no path towards doing your own AI training. The only approach we have amounts to pouring an ocean of information at a supervolcano of compute, then capturing the steam and distilling it using weapons-grade mathematics that requires its own stupendous infrastructure.
We can pray for a fundamental breakthrough, but for now, scale wins. That means Anthropic and OpenAI win. Maybe a nation state at the level of China or the US could compete, but that is a strong maybe. A company that can vacuum up money from across the developed world rather than having to rely on a single-country tax base probably still wins.
Information was proliferating even before we had these machines, now we’re creating new content (and, I have been known to argue, knowledge) using these machines, so we need more of these machines to search and filter and summarise for us. That trend too is going nowhere.
That leaves us dependent on huge companies to filter out, from the pre-training stage, sophisticated attempts at misinformation. With the added complexity that to do so at scale, they’ll be using previous-generation AIs to create that pipeline.
I would dearly love to have full faith in them as gatekeepers, but misinformation on social media dispels that fantasy quick. It has left me somewhat uncomfortable with a handful of American companies being the arbiters of what is epistemically legitimate.
Grant us, this day, a Y2K moment
Some really smart people have been worried about this stuff for a long time. Now that epistemic security is a business imperative, there’s a lot of money available to solve it. (Also, it can technically count as defence spending, which helps with the budgeting.)
Me, I’m not seeing how we get from here to a point less dire, but I also lived through Y2K from a vantage point inside the technology world. Afterwards, everyone else wanted to know what the fuss had been about, because nothing bad ever happened. The smart, capable people who made it into a non-event were too tired to explain it again.
Also, spam. Remember email spam? It didn’t go away, you just don’t see it anymore.
We never solved misinformation on social media, and we’re not solving the problem of journalistic reporting disappearing, it is true. But we have solved really complex problems in information technology, and in information filtering.
So there is hope. I just wish we had had a little more time.