Agent Web

When a user asks an LLM for information about your organization, the LLM sends an agent. That agent needs written permission to come in, and it needs the article as plain text, not a page full of menus. Cloudflare can make that plain-text copy and grade the setup. The permission file is robots.txt, published by you.

By Shashi Bellamkonda · August 29, 2026

GPTBot and ChatGPT-User are different visitors. GPTBot is OpenAI's training crawler. It copies pages to help build the next model. ChatGPT-User arrives because a person asked ChatGPT a question and ChatGPT went to fetch your page for that answer. Anthropic splits the same way. ClaudeBot trains. Claude-User fetches for a live chat. Claude-SearchBot builds a search index. If you want an assistant to use the site in a conversation, you allow the User and SearchBot names. You can keep the training crawler out.

robots.txt is a public text file at the root of your site. Search engines and AI companies read it before they request pages. The file states a preference. A crawler can ignore it unless you also block that crawler in Cloudflare's firewall or in AI Crawl Control.

Cloudflare will write part of that file for you. In the dashboard, open Security Settings, filter Bot traffic, and turn on "Set your preference to block training in robots.txt." Cloudflare then prepends this block to whatever robots.txt already returns a successful page (Cloudflare, 2026):

# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
# END Cloudflare Managed Content

The same managed block also blocks Amazonbot, Applebot-Extended, Bytespider, CCBot, and meta-externalagent. ChatGPT-User, Claude-User, Claude-SearchBot, and OAI-SearchBot are unnamed, so they fall under the wildcard Allow. Training stays out. A live chat fetch can still come in.

The managed line sets search to yes and training to no. It leaves ai-input blank. That third switch is the one that says an assistant may pull your page into a live answer. Cloudflare's scanner treats a blank switch as a miss (Cloudflare, 2026).

Read the live robots.txt before you rewrite it

Open a terminal and ask the public file to print itself:

curl -sS https://example.com/robots.txt

curl is a small program that fetches a web address and prints what came back. You are looking at the same text an assistant sees.

Check three lines. A Sitemap address that opens in a browser. Named visitor blocks. A Content-Signal or Content-signal line. Cloudflare's generator mixes the hyphen. The meaning is the same.

Content Signals are three answers you give every visitor. search: may they put you in a classic search index and show a short excerpt. ai-input: may they drop a passage into a live answer. ai-train: may they use the page to train or fine-tune a model (contentsignals.org, 2026). A blank ai-input field is an undeclared preference.

If assistants should read the site and the archive should stay out of training sets, make the wildcard stanza explicit. Keep Cloudflare's Disallow list for the training crawlers if you already use it.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

On a host you control, save that file at the domain root as robots.txt. On Blogger, paste it into Settings, Search preferences, Custom robots.txt. On Cloudflare, Bot Preference Sync, launched August 21 for every plan, copies the dashboard Search / Agent / Training toggles onto the front of the existing file (Cloudflare, 2026). After you save, run the same curl command. Confirm the public file matches what you wrote.

Ask Cloudflare to hand the visitor a clean text copy

Permission lets the visitor in. Format is the second change. A human page is HTML: menus, scripts, style sheets, cookie banners. An assistant wastes tokens on that chrome. Markdown is the same article as headings and paragraphs. Cloudflare's April 2026 scan of the top 200,000 domains found that 3.9 percent return Markdown when asked (Cloudflare, 2026).

The ask is a request header named Accept, set to text/markdown. In plain language, the visitor says "send me the text version." The site should answer with Content-Type text/markdown.

On Pro, Business, Enterprise, and SSL for SaaS plans, open the zone in dash.cloudflare.com, find Quick Actions, and toggle Markdown for Agents. The same switch also sits under AI Crawl Control. Cloudflare shipped it on February 12, 2026 as a beta with no extra fee on those plans (Cloudflare, 2026). Free plans do not get the toggle. Those sites need a real .md file on the origin or a small Worker that serves one.

With the toggle on, Cloudflare fetches your HTML, strips the chrome, and returns Markdown. Prove it on a live URL:

curl -sSI https://example.com/your-page/ \
  -H "Accept: text/markdown"

curl -sS https://example.com/your-page/ \
  -H "Accept: text/markdown" | head

The first command prints only the labels on the response. The second prints the start of the body. A pass looks like this: HTTP 200, content-type: text/markdown; charset=utf-8, vary: accept, and an x-markdown-tokens count that estimates how large the text is (Cloudflare, 2026). If you still see text/html, the toggle is off, the plan omits it, or an old copy sits in cache. Purge that URL and run the header command again.

As of February 2026 Cloudflare named Claude Code and OpenCode as tools that already send Accept: text/markdown (Cloudflare, 2026). Many chat assistants still ask for HTML. Publish a second address that always returns Markdown.

Give every page a /index.md twin

Cloudflare's docs site already does this. https://developers.cloudflare.com/workers/get-started/ accepts the header. https://developers.cloudflare.com/workers/get-started/index.md returns the same Markdown. You can copy that pattern with two Transform Rules in the Cloudflare dashboard (Cloudflare, 2026).

Rules, Transform Rules, URL Rewrite. Match a path that ends in /index.md. Rewrite the path by stripping that suffix so the request lands on the real page.

Rules, Transform Rules, Modify Request Header. On paths that contain index.md, set Accept to text/markdown. Markdown for Agents, or a file you already host, then returns the clean copy.

A Transform Rule is a small instruction at Cloudflare's edge. It changes the incoming address or a label on the request before your server sees it. You are teaching the edge: when someone asks for /your-page/index.md, fetch /your-page/ as Markdown.

curl -sSI https://example.com/your-page/index.md

You want HTTP 200 and text/markdown. A redirect loop back to the HTML page fails the check.

Publish /llms.txt as a short reading list

Jeremy Howard proposed this file on September 3, 2024. It lives at the site root. The required first line is a heading with the site name. Then one sentence in a blockquote, then a list of links to Markdown pages, each with a clause that says why that page exists (llmstxt.org, 2024).

In human terms, it is a table of contents written for a machine. You point the assistant at the few pages worth reading first, instead of making it wander the whole site.

# Example Co
> Product docs and implementation notes.

## Start here
- [Getting started](https://example.com/docs/start/index.md): install and first request
- [Auth](https://example.com/docs/auth/index.md): API keys and OAuth

Cloudflare splits its own library. /llms.txt points at /workers/llms.txt and /r2/llms.txt so one sitting does not swallow the whole set. Do the same once you have more than a few dozen pages. Skip directory indexes.

Blogger and many site tools cannot place a file at /llms.txt. Host the text in Cloudflare R2, which is object storage, or serve it from a Worker on the same hostname:

if (url.pathname === "/llms.txt") {
  return new Response(body, {
    headers: { "content-type": "text/markdown; charset=utf-8" }
  });
}

A Worker is a short program that runs on Cloudflare's network. This one says: if the path is /llms.txt, return this text. Then run curl https://example.com/llms.txt and confirm a 200 before you advertise the path.

Point at files that already exist

The Discoverability check also looks for Link headers, a label on the homepage that says "an alternate copy lives here." Rules, Transform Rules, Modify Response Header, on the homepage:

Link: </llms.txt>; rel="alternate"; type="text/markdown"

Add an api-catalog link only after that catalog file returns 200. An empty well-known path fails the next check.

Protocol cards (/.well-known/mcp.json, server cards, skills indexes, login discovery) describe tools an assistant can call. A publishing site with an empty card does not become more citable. Commerce checks for payment protocols sit on the report. They can wait if you are not selling through an agent (Cloudflare, 2026).

Score the hostname after the files exist

The public score lives at isitagentready.com. Type the hostname. Inside a Cloudflare account, open Overview for Agent Readiness, or go to Protect & Connect, Application Security, Investigate, run URL Scanner, then open the Agent Readiness tab (Cloudflare, 2026). The six buckets on that tab are Basic Web Presence, Discoverability, Content Accessibility, Bot Access Control, Protocol Discovery, and Commerce.

URL Scanner is Cloudflare fetching your page the way a bot would, then grading the response. You can trigger the same job from a terminal:

curl -X POST https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/urlscanner/v2/scan \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","options":{"agentReadiness":true}}'

Diagnostics groups the work as Quick wins (robots.txt, sitemap, crawler rules, Markdown), Technical groundwork (Content Signals, catalogs, Link headers, login notes), Advanced integration (login discovery, tool cards, signed bot identity), and Commerce (Cloudflare, 2026). Some failing rows offer a Set up in Cloudflare link that flips Markdown for Agents or managed robots.txt. Use that link after you have decided whether training crawlers stay out.

Answer Engine Optimization Visibility is a different tab. Early access opened August 6. You request it from Overview. Cloudflare asks Claude and GPT questions in your category and reports citation rate, mention rate, prominence, and share of voice (Cloudflare, 2026). Citation rate is how often your site is named as a source. Mention rate is how often the brand name appears even without a link. Prominence is how much of the answer is yours and how early it appears. Share of voice is your slice against named competitors.

Stephanie Cohen, Cloudflare's chief strategy officer, put the product in one line: "Being discoverable used to mean ranking on a page" (Cloudflare, 2026).

Open Visibility after curl already returns text/markdown. Measuring citations on HTML the model has to reconstruct tests the report, not the site.