Cloudflare Just Put a Meter on the Internet

Photo via Unsplash

Cloudflare gave AI companies a deadline of 15 September 2026. By that date, any company running a crawler has to separate the bot it uses for search indexing from the bot it uses to collect AI training data and to feed AI agents, and declare which is which. Crawlers that have not done so get blocked by default from any page carrying advertising. That is a week from now, and it applies across a network that carries more than a fifth of global web traffic.

The mechanism underneath it is simpler than the policy language. A site behind Cloudflare gets three options for each AI crawler rather than the two the web has offered since the 1990s. Allow it free. Block it. Or charge it a fee per fetch, collected at the network edge before the bot ever loads the page. Cloudflare processes the payment and passes the money on. That third option is pay-per-crawl, and it launched in July 2025 alongside the decision to block AI crawlers by default.

The ratio that made this inevitable

The old bargain of the open web was never written down but everyone understood it. A search engine crawls your pages, sends readers back, and both sides come out ahead. Cloudflare published the numbers on what that exchange had actually become. Google, which has the longest-standing traffic relationship with publishers, crawled about 14 pages for every click it sent back. OpenAI's crawler fetched roughly 1,700 pages per referral. Anthropic's fetched roughly 73,000.

Seventy-three thousand to one is not a degraded bargain. It is a different transaction wearing the old one's clothes. And the volume moved fast enough that it could not be ignored: AI-related crawlers were 22 percent of total crawler requests in spring 2025 and 52 percent by June 2026. Every large language model on the market learned from content collected on those terms. A handful of large publishers sued. Most did not have the resources for a court fight, and the value transferred from the people who produced content to the people who produced models without ever appearing on anyone's balance sheet.

What the arithmetic actually returns

This story gets told to Caribbean boards as a new revenue line. The arithmetic does not support that for almost anyone in the room. Take a site with a million monthly page views where AI crawlers are one to two percent of traffic, which is typical. At a tenth of a cent per page, that is about 20 dollars a month. At a full cent per page, about 200 dollars. Lunch, not payroll.

There is a second problem, and Cloudflare has conceded it. Pay-per-crawl pays for the fetch, not for the value. A publisher gets paid when a bot retrieves a page whether or not that page ever shaped an answer, and gets nothing when one crawled page becomes the basis of ten thousand responses. The company is developing the feature into Pay Per Use, which would charge when content creates value rather than when it is collected. That is the version worth negotiating against. The current version is measuring the wrong thing accurately.

A control point now exists where none did, and it is worth more than the cents flowing through it. The early adopters of the marketplace, Conde Nast, TIME, the Associated Press, Adweek and Fortune, are not there for the per-crawl money. They are there to establish that access is a negotiated thing.

What the Caribbean actually holds

The regional opportunity here is not publishing. A chamber of commerce metering its newsletter archive would earn about forty dollars a quarter. The opportunity is in long data series that cannot be reconstructed from anywhere else, held by institutions that have never thought of themselves as data owners.

The Caribbean Institute for Meteorology and Hydrology holds regional weather and water records going back decades. National disaster agencies and CDEMA hold post-event damage assessments at a level of granularity that satellite reconstruction cannot match, because somebody walked the site. Agriculture ministries hold yield records across crops and growing conditions that exist nowhere in the global training corpus. Tourism authorities hold arrival and origin patterns going back to the 1990s. Regional insurers hold hurricane loss histories that are, in effect, the only quantitative record of what Category 4 and 5 events do to this specific building stock.

What makes those valuable is not that they are big. By the standards of a training run they are tiny. It is that a model trained overwhelmingly on North Atlantic and continental data performs measurably worse on small island conditions, and this region holds the correction. My own doctoral work on climate modelling for small island states ran into that wall repeatedly: the physics is fine, the regional observations to constrain it are what is missing, and they are sitting in filing systems in Bridgetown and Kingston and St Augustine.

None of these organizations built those records to sell them. That is exactly why the inventory has to happen before the negotiation does.

Two decisions, and neither of them is about price

The first is whether you know what you hold. Almost no board I sit with can name its own data sets, say how far back each one runs, say whether it exists anywhere else, or say who inside the organization controls access. Four questions, written down, per data set. That inventory is the entire prerequisite, and no licensing conversation is possible without it. It is also useful for a dozen reasons that have nothing to do with AI.

The second is whether licensing is even the right answer. For a tourism authority or a private carrier, probably yes. For disaster and public health data, I think the argument often runs the other way. If a model that emergency managers and insurers will use anyway performs badly on Caribbean conditions because Caribbean observations were never in it, the region pays for that in forecasts and premiums, not in licensing fees. Deliberate open publication under clear terms may protect more people than a paywall does. What matters is that it becomes a decision rather than a default, because right now it is being settled by inaction and by whichever crawler got there first.

Who negotiates

Prices in a new market are set by whoever transacts first, and they harden fast. Firms and institutions that reach an agreement in the next year will define what regional data is worth for the decade after. Those that wait will accept whatever rate the market has already settled on, and by then the reference points will have been set by people with no exposure to this region at all.

That is a first-mover advantage with an expiry date on it, which is a phrase I use carefully because most claimed first-mover advantages do not have one. This one does, and the September deadline is the visible edge of it.

Start with the inventory. Four questions per data set, this quarter, before anyone talks to a vendor about a price.