A security rule that blocks automated scraping can also stop an AI assistant from reading a company's product pages. Marketing may discover the problem after asking why those pages no longer appear as sources in AI search.

Security engineers have good reasons to restrict automated traffic. Scrapers consume infrastructure, and agents that act inside customer accounts introduce authorization risk. Marketing, for its part, wants public content available to prospective buyers who use AI assistants.

Both teams need a voice in the policy. Content management system owners belong in the room too, because publishing controls and platform integrations help determine what agents can reach.

Cloudflare's Bot Preference Sync and cryptographic bot verification can implement parts of that policy. Their value depends on the decisions behind them: which content stays discoverable and which actions need their own authorization.

Autonomous agents are already testing access boundaries

AI agents now work beyond search indexing and page summaries. Their ability to use customer accounts has led financial institutions to restrict access.

In October 2026, Trending Topics, citing a Bloomberg Television interview, reported restrictions at JPMorgan Chase, Citigroup, Bank of America, and Charles Schwab, though each institution's policy differs. The article attributes to Schwab Chief Executive Officer Rick Wurster a figure of major outside agents stopped "in the tens and twenties each week." That number counts blocked agent attempts, not attempted money transfers.

Wurster said Schwab will not allow those agents to move money. The article also reports that outside AI models will be able to check balances and run analyses through Schwab's own Model Context Protocol servers.

Retailers have split on the same question. Semafor reported in September 2026 that Amazon banned the Meta Muse assistant while Shopify welcomed it. Its reporter also hit a bot blocker while trying to rebook a Delta flight through Muse. That is one restriction on one task, and it says nothing about Delta's policy for other third-party agents.

These cases show why a policy has to separate reading public information from acting inside a customer account. A company can welcome an assistant that researches its products and still require a distinct, approved integration for transactions.

Waiting until unauthorized automation reaches a checkout or a customer portal invites a rushed block, and a rushed block catches legitimate access too.

Structured agent tools need their own access policy

Websites can hand agents structured tools instead of making them interpret page layouts and simulate clicks.

The Model Context Protocol connects AI applications to external tools and data sources. WebMCP covers browser-based interaction: a website registers tools that an agent can call inside a user's session. The two are related, but their deployment models and authorization requirements differ.

Shopify's documentation describes WebMCP as a proposed web standard. Its tools run on Liquid storefronts and on Hydrogen storefronts that load them, and agent support is limited to Chromium-based browsers today. The tools can search catalogs, update carts, and support checkout actions in the shopper's active tab.

Tool exposure therefore becomes a governance decision. Marketing and commerce teams should name the customer journeys they want agents to support. Security and engineering should define the permissions and approval steps for each action.

Browser-session tools inherit the session's context. Remote integrations need their own authorization design, which may include OAuth and scoped permissions. Either way, the server still has to enforce access controls.

Prompt injection deserves a review of its own. Shopify warns that merchant and third-party text in tool responses can carry malicious instructions and should be treated as data.

Security and marketing judge success by different results

Security teams watch automated traffic for abuse and unauthorized access. Marketing teams want to know whether prospective customers can find and inspect public content through AI assistants. The two measures meet at the access rule.

OpenAI gives a concrete example. Its documentation says OAI-SearchBot supports ChatGPT search results, while GPTBot collects content that may be used for model training. The settings are independent, so a site owner can allow the search bot and disallow the training bot.

OpenAI also says a site that opts out of OAI-SearchBot will not appear in ChatGPT search answers, although it can still appear as a navigational link. That is a specific access consequence for one operator. Marketing should treat crawler access as a prerequisite to check, and citations as a separate result to monitor.

Crawler purpose should drive the rule

A useful policy sorts traffic by purpose and accounts for operators that combine several purposes in one crawler.

• Search and retrieval crawlers support indexing and answer retrieval. Examples include OAI-SearchBot, PerplexityBot, and Claude-SearchBot, run by OpenAI, Perplexity, and Anthropic. Restricting one affects page availability for that operator's service, so read the operator's documentation before predicting the result.
• User-triggered agents fetch content when a person asks. Examples include ChatGPT-User, Perplexity-User, and Claude-User. Blocking a fetch can stop an assistant from reading the requested page. A request alone does not show purchase intent or prove that a citation followed.
• Training crawlers collect content that may enter model-training datasets. Operators with separate search and training controls give you more precise choices than a blanket AI block.

Traditional search crawlers belong in the policy as well, including those that serve more than one purpose. A policy built around AI-branded user-agent names alone will miss that dependency.

LumenGEO's June 2026 log shows how varied the traffic on one website can be. Across a 14-day first-party log of 3,392 AI crawler fetches from 13 bots, the firm classified 44.3% as user-triggered agent traffic, 15.1% as retrieval, 29.6% as training, and 11.0% as search (LumenGEO, 2026). Agent and retrieval traffic together made up 59.4%.

Your own logs will look different. Read them before deciding which traffic to restrict.

A robots.txt file states a preference and the edge enforces it

A robots.txt file tells cooperating operators what you prefer. A noncompliant scraper can still request the page.

Cloudflare introduced Bot Preference Sync in August 2026 to update robots.txt from the AI bot settings a site owner has configured. The post describes the goal as keeping what a site states in robots.txt in step with what Cloudflare enforces at the edge, and it says the feature prepends a managed block so existing directives stay in place.

The setting you pick matters. Cloudflare's September update separates Disallow AI Training from Block. Disallow AI Training publishes a no-training preference, lets designated Accountable mixed-use crawlers keep reaching your content for search, and blocks other training crawlers. Block also stops those mixed-use crawlers, which Cloudflare says includes Applebot, Bingbot, and Googlebot, so search access goes with them.

Operator support varies too. Cloudflare said Microsoft is building domain-level support for a no-training preference in robots.txt, targeted for early 2027. Until it ships, selecting Disallow AI Training does not send that preference to Bing through robots.txt.

Test what each control does on your site. A setting with "training" in its name can still reach search services.

A signed request proves who is asking, not what they may do

A user-agent string is a claim of identity, and any requester can copy it.

Web Bot Auth adds signed requests. Cloudflare's implementation supports Ed25519 keys published in a well-known key directory and follows the HTTP Message Signatures framework, alongside two Internet Engineering Task Force drafts. Operators register their bots and signing keys with Cloudflare.

A valid signature shows which registered operator sent a request. You still decide what that operator may do.

An unsigned request is not malicious by default, either. Pair verification with behavior monitoring and application-level permissions.

Your content management system should carry these controls

Content management system vendors should build AI crawler and agent policy into their publishing and integration features, and enterprise buyers should add those capabilities to platform evaluations.

The publishing workflow is a natural place to record whether a content section is meant for public discovery and what usage restrictions apply. Those records should match your network controls.

A CMS should let administrators manage crawler preferences without colliding with a content delivery network's managed robots.txt. It should keep existing directives intact during publishing changes and make policy changes auditable.

Confidential content still needs authentication and access controls. A crawler directive replaces neither.

Agent-facing tools add another requirement. Before a platform or plugin exposes a new tool, administrators should be able to review its permissions and see whether it reads information or changes customer state. Permission to publish content should not carry authority to expose transactional functions.

Reporting belongs in the evaluation. Marketing needs to connect published sections to crawler access problems, and security needs visibility into blocked requests. Citation monitoring is its own job, because a successful fetch does not show that an answer engine used the content.

Five decisions for the alignment meeting

Before anyone changes crawler rules, bring together the Chief Information Security Officer's team, web marketing leads, and the CMS owner.

1. Set the public-content boundary. Identify the product pages and documentation meant for discovery. Define separate access requirements for customer portals and sensitive content.
2. Agree on model-training preferences. Involve legal counsel where content rights or licensing terms apply. Check each provider's controls and mark where technical enforcement is necessary.
3. Test existing rules against approved crawlers. Review firewall challenges and rate limits on representative public pages. A successful HTTP response confirms access to that page and says nothing about indexing or citation.
4. Assign monitoring responsibilities. Security tracks access failures and suspected abuse. Marketing tracks AI referrals and citations as their own measures, with a process for investigating changes.
5. Approve structured tools through change control. Require security review before a CMS, commerce platform, or plugin exposes agent-accessible actions. Document the authorization requirements and a way to switch the integration off.

The meeting should produce a documented access policy with named owners. Before the next publishing or firewall change, those owners should be able to say which public pages agents can read and which actions need their own permission.