A website can now charge a verified artificial intelligence crawler a fee or turn it away, and the session abstract places the settings behind that choice with the security team. I will test that premise on Wednesday, October 21, at 1:00 p.m. in The Frequency, a stage in The Hub on Level 1 of Moscone West, where I join Bryan Becker, a Cloudflare director of product, for a 40-minute podcast conversation at Cloudflare Connect.
Ron Davis, a Cloudflare analyst relations director, and I met in person at Info-Tech LIVE. I look forward to meeting two more directors on his team, Mike Wong and Caitlin Fischer, in San Francisco, and I hope to leave with a selfie with Matthew Prince.
Cloudflare started by absorbing distributed denial-of-service attacks so a website stayed online. The question it asks about each visitor today is whether a bot holds permission to read the page and what it owes for the read.
Cloudflare says automated traffic passed human traffic on its network in May 2026, more than a year ahead of its own forecast (Cloudflare, 2026). A crawler that wants through has a way to say who it is. Web Bot Auth, which builds on a standard from the Internet Engineering Task Force for signing web requests, has a crawler sign each request with a key it holds and publish the matching public key so the site can verify the sender.
A valid signature shows who sent the request and leaves everything else to site policy.
Five decisions ride on every signed request, and Cloudflare sells a control for each
- Identity. Cloudflare keeps a public directory of bots and signed agents. An agent qualifies by using Web Bot Auth and meeting a written policy on purpose and behavior.
- Permission. Since September 15 a site can keep search indexing and refuse the training copy, which I covered in my September 18 post.
- Format. Markdown for Agents converts a page to plain text when an assistant asks for it, as a free beta on paid plans, and I walked through the setup on August 29.
- Price. Pay Per Crawl lets a site set one price for its domain, a crawler that has not agreed to pay receives a 402 Payment Required response, and Cloudflare acts as the merchant of record. During Birthday Week Cloudflare added Pay Per Use and a Monetization Gateway beta that charges agents for access to websites and data.
- Authoring. EmDash, the open source content management system that Cloudflare took to version 1.0 on September 28, writes the page itself.
Every step has a documented reason, and a buyer should ask which one could move. Cloudflare released its Workers runtime as open source after a large customer made that a condition, and on October 9 it brought on the entire Deno team to make self-hosting a supported route, as I wrote in my Deno post. Cloudflare says an exit option helps it sign large customers, which is a reason to ask for yours in writing.
EmDash 1.0 narrows the gap I flagged in April
In April I wrote that EmDash's main security feature, plugins that run in isolated sandboxes, worked in full when hosted on Cloudflare's own infrastructure. I also wrote that the people who choose a content management system sit in marketing, while Cloudflare sells to developers and security teams. The April post has the detail.
Version 1.0 changes the first point. Web Hosting Today reports that off Cloudflare, EmDash starts Cloudflare's open source runtime as a child process and runs each plugin as an isolated service inside it, so the plugin sandbox works on a Node.js server too. The project is MIT licensed and built on Astro, the framework from The Astro Technology Company that Cloudflare acquired in January 2026. It counts more than 175 contributors, with Cloudflare employees a minority of the maintainers, according to the same report (Web Hosting Today, 2026).
Cloudflare hosts the default plugin registry, which runs on the AT Protocol so that each publisher signs its own release records, and Technobezz reports that Cloudflare has released the registry services as open source. Search Engine Journal argues that EmDash features still depend on Cloudflare's infrastructure, and it notes that EmDash Build, the alpha website builder, runs each project in its own Cloudflare sandbox. The code runs elsewhere while the default services stay with Cloudflare, so the portability question moves from the sandbox to the registry and the builder.
The Cloudflare blog has run on EmDash since August, and agents can work with the system through an application programming interface or a built-in Model Context Protocol server. A content management system is where a company decides what each page ships with. If EmDash can attach permission files and Markdown copies when an editor publishes, marketing sets policy in the place it already writes while security keeps identity and enforcement. I found no Cloudflare statement that EmDash does this, so it goes on my list for Wednesday.
One network serves a free resolver and high-impact federal data, so the release pace reaches both
Small sites use Cloudflare too. The company runs 1.1.1.1, a free Domain Name System resolver anyone can point a router at, and in September it described five changes to the 1.1.1.1 cache that freed about 100 terabytes of memory, which I covered on September 3.
At the other end, Cloudflare for Government reached Federal Risk and Authorization Management Program High certification effective August 6, nine months after the authorization process began on November 12, 2025, according to the program marketplace record that Web Hosting Today reviewed (Web Hosting Today, 2026). The same report says Cloudflare reached that tier on its existing global network instead of building a separate government cloud, and I covered the news on August 28.
Cloudflare made 46 announcements during Birthday Week alone (BornCity, 2026). Cloudflare adds each new layer to a network already in place, which makes the pace an advantage, and a buyer's change review has to keep up with a release rate the vendor sets.
A marketing team cannot argue about which assistant fetched a page without a log that shows it. On October 2 Cloudflare folded its logs and traces into one platform and opened Logpush log export to self-serve plans, where it had been Enterprise-only. Beyond the included allowance, logs cost $0.25 per gigabyte ingested when unified pricing starts December 1 (Cloudflare, 2026). Traces now export through OpenTelemetry, an open standard, to tools outside Cloudflare.
I need four answers on Wednesday to judge how much of this stack a customer can leave
These are the questions I take to The Frequency:
- Who should own the three bot settings in a company's Cloudflare dashboard, and what does Cloudflare do to bring the marketing lead into that decision?
- A signed crawler proves who it is. How does a site tell a live answer fetch from a training copy when one company runs both, and does that identity reach the logs a marketing team can export?
- Pay Per Use and the Monetization Gateway beta arrived together in Birthday Week. Who measures a use, and how does revenue that settles in stablecoins reach a finance team?
- Can EmDash write permission files and Markdown copies when an editor publishes, and does that work when the site runs off Cloudflare's network?
Becker, Bryan, and Shashi Bellamkonda. "Security vs. Reach: Who Controls Your AI Visibility?" Cloudflare Connect 2026, 21 Oct. 2026, Moscone West, San Francisco, https://www.cloudflare.com/connect/.
BornCity. "EmDash 1.0: Cloudflare gibt quelloffenes CMS für Produktion frei." BornCity, 2026, https://borncity.com/news/emdash-1-0-cloudflare-gibt-quelloffenes-cms-fuer-produktion-frei/.
Cloudflare. "Cloudflare Acquires Astro to Accelerate the Future of High-Performance Web Development." Nasdaq, Jan. 2026, https://www.nasdaq.com/press-release/cloudflare-acquires-astro-accelerate-future-high-performance-web-development-2026-01.
Cloudflare. "Everything We Launched during Birthday Week 2026." Cloudflare Blog, 2026, https://blog.cloudflare.com/birthday-week-2026-wrap-up/.
Cloudflare. "Have It Both Ways: Stay Discoverable in Search While Disallowing AI Training." Cloudflare Blog, 15 Sept. 2026, https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/.
Cloudflare. "Markdown for Agents." Cloudflare Docs, https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents.
Cloudflare. "Signed Agents." Cloudflare Docs, https://developers.cloudflare.com/bots/concepts/bot/signed-agents.
Cloudflare. "What Is Pay Per Crawl?" Cloudflare Docs, https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/.
"Cloudflare Reached FedRAMP High without Building a Separate Government Cloud." Web Hosting Today, 11 Aug. 2026, https://webhosting.today/2026/08/11/cloudflare-reached-fedramp-high-without-building-a-separate-government-cloud/.
"EmDash 1.0 Arrives as Free Open Source CMS with Decentralized Plugin Registry." Technobezz, 28 Sept. 2026, https://www.technobezz.com/news/emdash-1-0-free-open-source-cms-plugin-registry.
