Three failures compound silently on most enterprise websites, and the organizations running them have no idea any of them exist. The first is a robots.txt file written for a Googlebot-first world, referencing deprecated crawler strings that stopped working months ago. The second is a Cloudflare configuration blocking every AI crawler at the network edge before it ever reaches the server, let alone reads the robots.txt. The third is JavaScript rendering that produces a blank page for any bot that does not execute client-side code. Each failure alone is recoverable. All three together means a marketing or IT team that believes it has a deliberate AI visibility strategy has, in practice, no strategy at all.

I ran analytics tools at Network Solutions during the Omniture era, when the equivalent problem was misconfigured tracking tags. The data looked fine in the dashboard. The data was wrong at the source. The gap between what you believe is being measured and what is actually being captured is a persistent infrastructure problem, and it does not announce itself.

The AI crawler access problem is that same gap, running in reverse. Instead of failing to capture data about your visitors, you are failing to be captured by the systems your next customer is using to research their shortlist.

The instinct to block is a legacy reflex

The number that drove the blocking instinct is real. Cloudflare's own Radar data from June 2025 showed Anthropic's crawl-to-referral ratio at 73,000 crawls for every single visitor sent back. OpenAI's ratio was 1,700 to one. Google's was 14. Publishers looked at those numbers, recognized that the implicit crawl-for-traffic bargain that had governed the web for two decades was broken, and blocked. It was a rational response to a real asymmetry.

It was also largely ineffective.

A BuzzStream study of 4 million AI citations published in March 2026 found that among the top news sites blocking OpenAI's ChatGPT-User retrieval crawler, 70.6% still appeared in AI-generated answers. Among sites blocking Google's Google-Extended training crawler, 92.3% still appeared. Blocking training crawlers does not remove you from AI answers. It tends to make you a staler, less accurate version of yourself inside those answers, because the models continue drawing on whatever was captured before you blocked, while your competitors who stayed open become the fresh, citable source.

The conversion data makes the case more directly. Visitors arriving from AI search convert at 23 times the rate of traditional organic search visitors. The volume is still small relative to Google. The intent behind each visit is not. These are buyers who have already used an AI system to research, compare, and narrow their options before clicking through. The click is the end of a process, not the beginning. A brand that is invisible during that process does not recover at the click.

What is actually blocking you — and it is probably not what you configured

On July 1, 2025, Cloudflare announced what it called Content Independence Day: every new domain on its network would block all known AI crawlers by default, shifting the web from opt-out to opt-in for AI access. Because Cloudflare sits in front of roughly 20% of all websites globally, that single infrastructure decision blocked a significant portion of the public web from AI systems before a single robots.txt rule was ever read. The block happens at the network edge. The crawler never reaches the server.

The compounding problem: many sites on managed hosting platforms, Shopify, Wix, and others, inherited Cloudflare's default block without any administrator making an active decision. The block is invisible in the platform dashboard unless you know to look for it. If your site is on one of these platforms and has not been explicitly configured to allow AI crawlers, it is almost certainly blocked.

Blogger, which powers this site, is Google-hosted and does not run through Cloudflare's proxy layer. That is a structural advantage for AI visibility that most Blogger publishers do not know they have.

The second silent failure is JavaScript rendering. Disable JavaScript in your browser and reload your most important pages. Whatever disappears is invisible to every major AI crawler on the market. AI crawlers do not execute client-side code. They read the raw HTML the server returns. If your product pages, case studies, or key analysis pieces load their content through API calls or React components, those pages are blank to any LLM attempting to retrieve them, regardless of what your robots.txt says.

Each LLM handles this differently — and the differences matter

The crawler ecosystem is no longer a single-entry problem. Each major AI platform now operates multiple bots with distinct purposes, and the robots.txt decision for each one is independent. Getting this wrong in one direction removes you from live AI answers. Getting it wrong in the other puts your content into training pipelines you did not intend to authorize.

One maintenance issue that is causing silent failures right now: Anthropic previously operated under the user-agent strings Claude-Web and Anthropic-AI, both now deprecated. If your robots.txt still references those strings, your directives have no effect on any current Anthropic crawler. The bots are simply walking past rules written for identifiers they no longer use.

The Google governance trap nobody warned you about

Google's crawler structure creates a specific problem that does not exist with any other platform. Blocking Google-Extended has no impact on your search rankings — Googlebot and Google-Extended are entirely separate systems. But blocking Google-Extended removes your content from the pipeline feeding AI Overviews and Gemini grounding. There is currently no way to appear in Google Search without being eligible for AI Overviews. If you want to block AI Overviews, you would have to block Googlebot, which removes you from search entirely. That trade does not exist in a form anyone would accept.

The additional wrinkle: Google-Extended does not appear in standard server logs the way Googlebot does. It operates as what Google calls a standalone product token. If you are auditing crawler access through your logs alone, Google-Extended is invisible — which means many sites that believe they have successfully blocked AI training access have simply blocked a bot they cannot see arriving anyway.

Gated content is a different decision from blocking

The blocking argument and the content protection argument are often conflated, and they should not be. For content that carries genuine proprietary value — subscription-tier research, client-facing deliverables, premium databases — the answer is directory-level disallow rules applied to specific paths, not a site-wide block. Allow retrieval crawlers on your public analysis. Disallow training crawlers on the directories that house protected content. These are separate robots.txt entries and they can coexist in the same file.

For most B2B thought leadership publishers, however, the gating problem is not intentional. It is structural. Content that exists in open HTML is being rendered invisible by JavaScript execution requirements, PDF formats that crawlers cannot parse cleanly, or dynamic page elements that only resolve after client-side code runs. The gate was never deliberate. It just looks that way to every LLM that tried to read the page.

The minimum viable open configuration for a public-facing enterprise content site looks like this:

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# Protect sensitive paths only
User-agent: GPTBot
Disallow: /members/
Disallow: /premium/

User-agent: ClaudeBot
Disallow: /members/
Disallow: /premium/

Sitemap: https://yoursite.com/sitemap.xml

Access is necessary but not sufficient

Getting the crawler access right removes the invisible gate. It does not guarantee accurate representation.

AI systems are probabilistic. When the signal your site sends is weak or ambiguous — inconsistent heading hierarchy, content that only partially loads, important analysis buried inside a PDF that a crawler retrieved as a near-empty shell — the model fills the gap with inference. That inference may be accurate. It may not be. The correction mechanism does not exist the way a search ranking correction does. You cannot file a reconsideration request with ChatGPT.

The structural practices that reduce inference error are the same ones that have always produced well-indexed content: semantic HTML, explicit heading hierarchy, fact-dense prose in crawlable text rather than images, a current and accurate sitemap, and schema markup that identifies the author, the publication date, and the organization behind the content. None of this is new. The reason it matters now is that the stakes of getting it wrong are higher. A miscited AI answer about your product category, surfaced to a buyer who never clicks through to verify, is a harder problem to recover from than a missed keyword ranking.

Two free tools make the diagnosis concrete. The Adobe AI Content Visibility Checker, a free Chrome extension, compares what a human visitor sees against what an AI agent can actually read, returning a citation readability score with specific content highlighted as hidden from large language models. The second, What's Up With That from Marshall Kirkpatrick, applies more than 35 analytical frameworks to any page or draft, identifying whether your content is making a genuinely distinct argument or restating category consensus. AI models cite the former and average out the latter. I covered both in detail on Misunderstood Marketing.

shashi.co is Google-hosted, not behind Cloudflare's default block, and written in clean HTML with explicit structure. That is not an accident. It is the same instinct I developed running analytics at Network Solutions: the measurement infrastructure has to be right before the strategy built on top of it means anything. The equivalent in 2026 is making sure the retrieval infrastructure — the robots.txt, the rendering layer, the structural HTML — is actually doing what you believe it is doing.

The score that tells you where the standard is heading

I ran shashi.co through isitagentready.com, a tool Cloudflare released in April 2026 that scores sites against emerging agent-readiness standards: robots.txt configuration, sitemap presence, Markdown negotiation for agents, API catalog publication, OAuth discovery, MCP server cards, and agent skills indexes.

A score of 25 sounds like a failing grade. Read against what shashi.co actually is — a content publication with no APIs, no OAuth-protected resources, and no transactional layer — and the six checks for API Catalog, OAuth discovery, MCP server cards, and Agent Skills index are simply not applicable. A newspaper does not fail a check for not having a payment terminal. The two genuine gaps are Markdown negotiation and Content Signals.

Markdown negotiation is a Cloudflare-proposed standard that lets agents request a page as text/markdown rather than HTML, receiving a cleaner version with less rendering overhead. Blogger cannot implement this natively. It is on the horizon for content platforms, not yet actionable today.

Content Signals is more immediately relevant. It is an IETF draft that extends robots.txt with granular intent declarations:

Content-Signal: ai-train=no, search=yes, ai-input=yes

That single line says: do not use this content to train future models, but surface it in AI search answers and use it as live context when a user asks a question. The current allow/disallow binary in robots.txt cannot make that distinction. Content Signals, when it reaches ratification and crawler adoption, will. No major crawler honors it yet. The sites that implement it early will have the most granular control when they do.

About Shashi Bellamkonda

Loading bio…

LinkedIn Profile

Get new posts by email:

Subscribe

Powered by