Written For Readers Who Don't Read
Most of the web's readers are now machines. Three things we thought were settled are quietly being rewritten to suit them.
Ask a chatbot a question and watch what happens to the web behind it. It reads thirty or forty pages to build your answer, strips out what it needs, and hands you a tidy paragraph. You never see the pages, never click them. The site that “won”, whatever winning means now, gets a citation in light grey text and not one visitor.
That is most of the web’s reading now, and the firms running the pipes can watch it happen. On Cloudflare’s network, bots overtook people in requests for actual web pages this year, 57.5% to 42.5%. Its chief executive marked the crossover in June, about eighteen months ahead of his own forecast, and put the surge down to agents fetching pages on people’s behalf. AI is the fastest-growing slice by far, up roughly eight times quicker than human visits across last year. The web is being read more than ever. Just not by people.
Here is the uncomfortable part nobody quite wants to say out loud. The rules we built for the web, the ones about quality and access and honesty, were written for a human audience. The audience changed. The rules are being repealed to match, quietly, with nobody putting it to a vote. Three of them are already going. Here is which, and why.
Who polices the web, and why they bothered
For twenty-odd years the web had a traffic warden, and the warden was Google. Stuff keywords where nobody could see them, buy a thousand links, spin up doorway pages, and sooner or later something dropped on you from a great height. Most people assumed this was hygiene. Google keeping the web tidy on everyone’s behalf, out of the goodness of its heart.
It was nothing of the sort. Google policed the web because Google sold advertising against an index of it, and an index full of junk is worth less to advertisers. The cleaning was maintenance on a shopfront.
Andy Baio watched the same logic a decade ago, when Google let Books and its news archive rot the moment they stopped earning their keep, and warned against trusting a corporation to do a library’s job. He was being generous. The library was only ever a side effect of the ad business, kept alive while it paid and dropped when it didn’t.
I spent the better part of six years on the search quality and webspam side of that operation, so I can tell you the work was real and the engineers meant it. The reason it got funded was never in doubt.
Now look at the new readers. An answer engine doesn’t sell ads against a tidy index, because it doesn’t keep one for you to browse. It reads, weighs what it finds, and repeats the bit it wants. A weak page doesn’t get punished. It just doesn’t get picked up, which from the page’s point of view is worse, because at least a penalty came with an email. There is no warden any more. There is a doorman who never tells you why you didn’t get in.
And Google is now standing in both jobs at once, which is the genuinely funny part. It still runs the ad-funded index, the one a court just ruled an illegal monopoly in both search and the advertising sold against it. And it is building the answer engine that makes that index beside the point, then cramming ads into it as fast as the format allows: sponsored images blended into the picture results, ads inside the AI summaries, a whole new shopping pipe bolted on. A company defending its old business by building the thing that ends it, and selling ads on the murder weapon. You don’t need a diagram.
Who pays to get in
The old arrangement was a fair trade, a generous one even. Let our crawler in for free, Google said, and we’ll send readers back to you. Sites didn’t just tolerate the crawl, they fought to be crawled faster and indexed deeper, because the crawl was the on-ramp to an audience.
The AI crawl makes no such offer. It takes the same content, folds it into an answer, and sends back nothing. No click, no reader, no ad to sell. Cloudflare, which sits in front of about a fifth of the web, said the quiet part into a microphone in July 2025: the old deal is broken, so new sites on its network now block AI crawlers by default, and owners can charge per crawl through a marketplace that hands any bot unwilling to pay a polite “payment required” and nothing else. Thirty years of begging Google to crawl more, and the reflex now is to put up a tollgate.
In Britain, thirty-one publishers have gone past blocking. They’ve turned the old robots.txt file, the polite please-don’t the crawlers learned to ignore, into a binding contract: load the page, reuse an article without paying, and you’ve agreed to a £500 invoice a county court can enforce like any other debt. Nobody has actually collected from OpenAI yet, but the move tells you where this is heading. The toll has a price list now.
Why now, and not five years ago when the scraping started? Because the writing turned into the scarce thing. When a model is only as good as the text it learns from, and the open web is filling up with the model’s own exhaust, genuine human writing stops being raw material and becomes the prize. The owners worked out they were sitting on the input, and input has a price.
So watch what they actually do, because it tells you the real argument. The New York Times is suing OpenAI for training on its archive and, in the same breath, licensing that same archive to Amazon. OpenAI has signed the Guardian, the Atlantic, the Washington Post, News Corp and a queue of others. Nobody on that list is trying to send the machines away. They are haggling over the rate. The genie is out, the demand is structural, and the fight was never about whether this happens. It is about who gets paid and how much. The road gets built either way. The argument is only about the toll.
What counts as cheating now
There is one rule older than all the others: never show the crawler something different from what you show the person. That is cloaking, and cloaking gets you erased. Every SEO learns it on day one.
But read Google’s own definition, not the folklore. Cloaking is serving different content to users and search engines with intent to manipulate rankings and mislead people. What made it cheating was never that the two versions differed, but the intent to deceive sitting underneath. Showing the crawler a page about holidays and the human a page flogging knock-off pharmaceuticals is cloaking. Serving the same facts in a cleaner, machine-readable shape to something that only reads machine-readable shapes is not, and never was. Google’s own guidance says as much: adapt the presentation all you like, as long as the substance is the same.
So when the reader is an agent that wants structured data instead of your hero image and cookie banner, handing it structured data isn’t a trick. It is answering in the language it asked in. The taboo assumed the crawler was a stand-in for a human pair of eyes. Take that assumption away and most of the taboo goes with it.
This is where I have to be the ex-webspam bore for a moment, because the line still exists, and it just moved. Lie to the machine to shift a ranking, or feed it something you would never stand behind in front of a person, and it is still likely to be seen as intent to deceive, and in health, money and anything else where a wrong answer hurts someone, it is still dangerous. Deception is the line. Formatting your content for a reader that happens to be a machine was never on the wrong side of it.
What’s left for people
None of this is a forecast. It has already happened to the parts of the web the machines care about most, and it is working outward from there. The direction is not up for debate. The only live questions are the terms: who gets paid, who gets walled off, and whose business model wins the right to decide what a good answer looks like.
That last one is wide open, and worth watching closely, because the people building the new front door cannot agree on how to make money from it. Google is stuffing ads into the answers. Perplexity tried ads, killed them, and now swears a user has to believe they are getting the best answer rather than the best-paid one. Anthropic keeps Claude ad-free and says so loudly. OpenAI is testing ads while promising they will not bend the answer, which is precisely the promise Google made about search results, and we all remember how that aged. Whoever wins that argument inherits the warden’s old job, and gets to define “good” for a web most people now experience second-hand.
Which leaves a smaller, stranger job for the rest of us. The machine-read web does not care how clever your headline is or how slick your page feels. It cannot be charmed or flattered. It keeps what is useful to the answer and ignores the rest, which means the work that survives is the work under the decoration: the original reporting, and the judgement to know whether an answer is actually right before it reaches someone who will act on it. The web spent thirty years learning to perform for people. It now has to be useful to something with no thumbs to raise and no hands to clap.