Cloudflare Just Put a Meter on the Internet

Wenceslaus Hollar, Strasbourg: The Toll House, 1625–77. The Metropolitan Museum of Art, public domain.

For most Caribbean publishers, Cloudflare's new rules on AI crawlers (the automated programs that fetch web pages for AI companies) change little in revenue. For a handful of regional institutions holding long data records, they are a prompt to decide what those records are worth and who should have them. Cloudflare set 15 September 2026 as the date by which AI companies must run separate, declared crawlers for search indexing and for AI training and agents. Crawlers that have not split are blocked by default from any page carrying advertising, across a network that carries more than a fifth of global web traffic.

What pay-per-crawl is

Since the 1990s a website has had two choices for a bot: let it in or block it. In July 2025 Cloudflare started blocking AI crawlers by default for new customers and launched a third choice, pay-per-crawl. A site sets a price per page; the bot pays at the network edge before it loads anything, and Cloudflare passes the money on.

The July 2026 change added the declaration requirement. It matters because a publisher who wants to stay in search results no longer has to accept AI training as the price of being found.

Why crawl-to-referral ratios forced the change

Search crawling was a trade: the engine took your pages and sent readers back. Cloudflare's published figures show how far that trade has shifted. Google crawled about 14 pages for every visitor it referred. OpenAI's crawler fetched roughly 1,700 pages per referral, and Anthropic's roughly 73,000.

AI crawlers were 22 percent of crawler requests in spring 2025 and 52 percent by June 2026. A few large publishers sued AI companies over the use of their content; most publishers could not afford to.

What a typical site would earn

Take a site with a million page views a month where AI crawlers account for 1 to 2 percent of requests, or 10,000 to 20,000 fetches. The monthly revenue works out as follows:

Price per page	Monthly revenue
US$0.001	US$10 to US$20
US$0.01	US$100 to US$200

That is not a revenue line for a Caribbean newspaper or business association. The model also pays for the fetch, not for use: one page that shapes thousands of answers earns the same as one that is never used. Cloudflare has said it is developing a usage-based version, which would be the one worth negotiating over.

The early publishers in the marketplace, including Condé Nast, TIME, the Associated Press, Adweek and Fortune, are there mainly to establish that access is negotiated, not for the fees.

The Caribbean data global models lack

The regional value is in records that cannot be rebuilt from anywhere else, held by bodies that rarely think of themselves as data owners:

• The Caribbean Institute for Meteorology and Hydrology (CIMH): decades of regional weather and water records.
• National disaster agencies and CDEMA: post-event damage assessments collected on the ground, at a detail satellites cannot match.
• Agriculture ministries: yield records by crop and growing conditions.
• Tourism authorities: arrival and origin data going back to the 1990s.
• Regional insurers: hurricane loss histories for Caribbean building stock.

These sets are small by training standards. Their value is that models built mainly on North Atlantic and continental data do worse on small island conditions. In my own doctoral work on climate modelling for small island states, the physics was rarely the problem; the missing piece was regional observations, many of them sitting in filing systems in Bridgetown, Kingston and St Augustine.

Sell, share or publish: a decision rule

Before any price is discussed, each data holder needs an inventory. For every data set, write down four things: what it covers, how far back it runs, whether a copy exists anywhere else, and who controls access. Few boards I work with can answer those today, and the inventory is useful for audit and planning whether or not anything is ever licensed.

Then decide the route by the kind of data:

• Commercial data (tourism, private carriers, retail): licensing is usually the right route.
• Public safety data (disaster, weather, public health): open publication under clear terms may protect more people. If the forecasting and insurance models used in the region perform badly on Caribbean conditions, the region pays through worse forecasts and higher premiums.
• Personal data: neither, without a legal basis under national data protection law.

Prices in a new market tend to be set by the first deals. I expect agreements reached in the next year or two to become the reference for what regional data is worth. I cannot show that from Caribbean deals yet, because there have been almost none.

What to do this quarter

1. Heads of CIMH, national met services, disaster agencies and tourism authorities: complete the four-question inventory. One spreadsheet row per data set, owned by a named manager, finished before any vendor conversation.
2. Boards of these bodies: decide the route for each data set. License, publish openly or keep closed, recorded as a board decision with the reason.
3. Web managers of any site behind Cloudflare: review your AI crawler settings. In the Cloudflare dashboard, check which AI bots are allowed, blocked or charged, and set it deliberately.
4. Publishers and business associations: keep search crawlers open. Block or charge AI training crawlers if you wish, but confirm your pages still appear in search results a week later.
5. Legal counsel: draft standard licence terms before the first request arrives. Cover permitted use, attribution, whether training is allowed, and a review date.

Frequently Asked Questions

Does pay-per-crawl work if my site is not on Cloudflare?

No. Pay-per-crawl is a Cloudflare feature enforced at its network edge, so it applies only to sites that route traffic through Cloudflare. Other sites can still publish a robots.txt file asking AI crawlers not to collect pages, though compliance is voluntary, or block known crawler user agents and IP ranges at the server or firewall.

Will blocking AI crawlers hurt my Google ranking?

Not if you block the right bots. Google uses the Googlebot crawler for search and a separate control, Google-Extended, for use of content in its AI models, so you can refuse the second without affecting search. Check each company's published crawler names before blocking, because blocking a search crawler removes you from that engine's results.

Is robots.txt enough to stop AI training on my content?

It is a request, not an enforcement mechanism. Major AI companies say their crawlers respect robots.txt, but it does not cover content already collected, copies on other sites, or crawlers that ignore it. Network-level blocking, such as Cloudflare's, and contract terms for licensed data give stronger control.

Can a government agency sell its data to AI companies?

It depends on the agency's founding law and the country's rules on public information, procurement and data protection. Many agencies need ministerial or board approval to license data, and personal data needs a lawful basis under the national data protection act. Ask the Attorney General's office or the agency's legal counsel before discussing terms with any buyer.

What should a data licence cover?

At minimum: which data is licensed, what the buyer may do with it (including whether model training is allowed), attribution, how long the licence lasts, the fee, and what happens to copies when it ends. Add a clause on onward sharing, since data licensed to one firm can otherwise reach others. Keep the first licence short and non-exclusive, so an early low price does not lock you in.