Most advice on AI agent readiness starts with a file: drop an llms.txt in the root, publish an MCP manifest, adopt the protocol of the month. That’s backwards. Agents mostly fail on boring things: a price that only appears after JavaScript runs, a “button” that is really a styled div, a bot-protection rule that blocks the very assistant a customer sent to book with you. Protocols matter, but they are the top floor. This guide works from the foundation up, explains how AI agents read websites, and ends with a checklist sorted into do now, watch, and ignore for now.
What AI Agent Readiness Actually Means
An agent-ready website is one where software acting for a person can find accurate information and finish a legitimate task without guessing, routing around obstacles, or tripping your security. “Agent” covers three quite different visitors, and each needs something different:
- Fetchers and crawlers. Automated clients that pull your HTML and leave. OpenAI documents OAI-SearchBot (ChatGPT search) and ChatGPT-User (fetches a user triggers); Google documents Google-Agent, which it describes as used by “agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”
- Browser agents. Gemini in Chrome’s auto browse, Perplexity’s Comet and ChatGPT’s agent open your page in a real browser and click, type and scroll on someone’s behalf.
- Tool-calling agents. These skip the page and call an interface you publish: an API, MCP server, WebMCP tool or commerce endpoint.
The layers stack. A WebMCP tool is useless if your firewall blocks the browser it runs in, and a perfect bot policy can’t help if the answer is buried in client-side JavaScript.
How AI Agents Read Websites
The Chrome team’s web.dev guide to agent-friendly websites describes three inputs browser agents combine: screenshots read by a vision model, the raw HTML, and the accessibility tree, which it calls “a high-fidelity map that ignores the visual noise of CSS.” Screenshots are slow and expensive, so agents lean on structure first. Google’s AI optimization guide for Search describes browser agents the same way: analysing screenshots, inspecting the DOM and interpreting the accessibility tree.
Fetchers are cruder. Vercel’s December 2024 analysis of AI crawler traffic found that none of the major AI crawlers it tested executed JavaScript; OpenAI’s and Anthropic’s crawlers downloaded script files without running them, while Gemini (on Google’s infrastructure) and Applebot were the exceptions. Behaviour may have changed since, but assume a fetcher sees only your server-rendered HTML.
The consequence for agent readiness: an agent judges your page by its source and accessibility tree long before your design. Accessibility work and agent work are largely the same job.
Step 1: Make Every Important Fact Fetchable Without JavaScript
List the facts an agent is likely to be sent for: prices, stock, hours, specs, returns policy, contact details. Then check each exists in the raw HTML:
curl -s https://example.com/pricing | grep -i "per month"If the price shows in your browser but grep returns nothing, a non-rendering fetcher can’t see it either. The fixes are standard JavaScript SEO: server-side rendering or static generation, facts in text rather than images, nothing essential behind click-loaded tabs, and no interstitial replacing the page for unknown clients.
SEO Rocket’s site audit crawls with a real crawler (Screaming Frog under the hood, JavaScript rendering included), so status codes, indexability, titles and thin pages across thousands of URLs surface in one pass.
Step 2: Build Pages a Browser Agent Can Operate
Browser agents fail where screen-reader users fail. The web.dev guide recommends:
- Use real controls. Prefer
<button>and<a>over clickable divs; where you can’t, give the element aroleandtabindex. - Tie labels to inputs. A
<label for>that matches the input’sidlets the agent connect “Postcode” to the right field. - Remove ghost elements. Transparent overlays sitting on top of controls can cause visual analysis to discard the control underneath.
- Keep layout stable. An Add to cart button that moves between page types confuses screenshot-driven agents.
- Make targets visible. Interactive elements should be visible, larger than 8 square pixels, and use
cursor: pointer, which the guide calls a strong actionability signal. - Confirm the irreversible. For purchases and bookings, require an explicit confirmation step a human approves.
<label for="postcode">Postcode</label>
<input id="postcode" name="postcode" autocomplete="postal-code">
<button type="submit">Check delivery</button>Forms are where agents stall: custom date pickers, checkouts that reset on error and a CAPTCHA on every step. Native inputs and honest labels are the cheapest fix here.
Step 3: Set a Deliberate Bot Policy (robots.txt Is Only Half of It)
robots.txt governs crawlers, and each crawler is a separate decision. OpenAI says opting out of OAI-SearchBot keeps you out of ChatGPT search answers, while disallowing GPTBot signals your content shouldn’t be used for training. Blocking one doesn’t require blocking the other. Our robots.txt guide covers the syntax.
The catch: user-triggered agents largely sit outside robots.txt. Google states that because the fetch was requested by a user, its user-triggered fetchers “generally ignore robots.txt rules,” and OpenAI says robots.txt rules “may not apply” to ChatGPT-User. If you’re on Cloudflare, its Content Signals Policy adds a Content-Signal line to robots.txt with yes/no values for search, ai-input and ai-train. Cloudflare is explicit that these are preferences, not enforcement.
So the real gatekeeper for agents is your CDN or firewall. A common agent readiness failure is a blanket “block AI bots” toggle or an aggressive challenge page stopping a customer’s agent mid-task. Review blocked requests for documented agents such as ChatGPT-User and Google-Agent, and verify before allowing: user-agent strings can be spoofed, which is why Google publishes its user-triggered agent IP ranges (user-triggered-agents.json) for verification.
Step 4: Let Verified Agents Prove Who They Are
IP lists don’t scale, so identity is moving to cryptography. Web Bot Auth has an agent sign its requests using HTTP Message Signatures (RFC 9421); a Signature-Agent header points to a key directory published at a well-known URL, and the server checks the signature against it. The IETF’s webbotauth working group draft is on the Standards Track but still an Internet-Draft, so details can change.
It is already being verified in production. Cloudflare checks signatures at its edge, and per its docs the separate “signed agent” category was retired on 1 July 2026: those agents are now verified bots labelled “intermediary” (operated by many end users), as distinct from “direct” (one operator’s own bot). Google says it is experimenting with Web Bot Auth using the https://agent.bot.goog identity. You don’t implement signing yourself: pick a CDN or WAF that verifies it, then allow verified intermediaries where you want agents. The details are in our Web Bot Auth explainer.
Step 5: Offer Machine-Readable Shortcuts, and Know Which Count
Once agents can reach and read your pages, shortcuts reduce the guessing:
- Structured data. Google’s guide says structured data “isn’t required for generative AI search” and there’s no special markup to add, though it still recommends it for rich results. For agents, Product, Offer, Organization and LocalBusiness markup that matches the visible page gives unambiguous facts. See structured data for AI agents.
- Sitemaps and stable URLs. Agents follow links and sitemaps.
- Markdown on request. Some agents send
Accept: text/markdown. Cloudflare’s Markdown for Agents converts HTML at the edge for Pro, Business and Enterprise zones. Cloudflare also reported that in February 2026 only three of the seven agents it tested (Claude Code, OpenCode and Cursor) sent that header by default. - llms.txt. Jeremy Howard’s 2024 proposal for a Markdown index of your key pages. Google’s guide says you don’t need AI text files to appear in Search, and Search itself doesn’t use them. Lighthouse’s agentic browsing audits do check for one. Decision rule: worth an hour on documentation and SaaS sites that coding agents read; not a ranking lever. More in our llms.txt guide.
Step 6: Expose Explicit Tools Where the Task Justifies It
Tools let an agent call a function instead of driving your UI. Three options, at very different maturity:
- WebMCP. A Draft Community Group Report from the W3C Web Machine Learning Community Group, explicitly “not a W3C Standard nor on the W3C Standards Track,” edited by people from Microsoft and Google. Pages register tools through
document.modelContext; Chrome documents an imperative JavaScript API and a declarative one that annotates HTML forms, with an origin trial starting in Chrome 149. Start with what WebMCP is. - MCP. The Model Context Protocol connects AI clients such as Claude to remote tools and data over JSON-RPC; the current specification revision is dated 2026-07-28. SEO Rocket runs one: a free, read-only server at
https://app.seorocket.ai/mcpwith four tools for searching our SEO guides, finding free tools, defining terms and mapping any site from its XML sitemaps. Narrow, read-only, plainly described tools are the pattern to copy. - Commerce protocols. Google’s guide notes that protocols like the Universal Commerce Protocol “are emerging that will allow Search agents to do more.” Stores should start with our agentic commerce guide.
Keep this in proportion: when Cloudflare scanned the 200,000 most visited domains for its April 2026 Agent Readiness launch, MCP Server Cards and API Catalogs together appeared on fewer than 15 sites.
How to Test Your Site’s Agent Readiness
Four checks, cheapest first:
- Lighthouse agentic browsing. An experimental Lighthouse category (Chrome 150 or later) with deterministic audits: WebMCP tool registration, an agent-centric subset of accessibility audits, layout shift, and llms.txt presence. It reports a pass ratio, not a 0 to 100 score.
- Cloudflare’s scanner. isitagentready.com checks discoverability, Markdown support, bot access control (Content Signals, AI bot rules, Web Bot Auth) and capabilities such as MCP Server Cards, API Catalogs and WebMCP, with commerce protocols reported as informational. It scores standards adoption, so don’t chase protocols you have no use for.
- A real task test. Run five tasks a customer would delegate (“book a consultation next Tuesday”) in a real browser agent. Where it stalls is your priority list.
- Logs. Confirm agents actually arrive and get 200s, not 403s. Our guides to log file analysis and AI agent analytics cover the method. SEO Rocket’s AI visibility tracking covers the other side: whether assistants mention you at all.
The AI Agent Readiness Checklist
Do now
- Key facts (price, stock, hours, policies) present in server HTML; curl test passes.
- Real buttons and links, labelled inputs, no overlays on controls, stable layout.
- Confirmation step before any purchase, booking or account change.
- robots.txt reviewed crawler by crawler: search, training and user fetches decided separately.
- Firewall and bot rules audited for documented agents, with verification by IP range or signature.
- Structured data that matches the visible page; current XML sitemaps.
Watch
- WebMCP, while it is in origin trial and spec drafts.
- Web Bot Auth, as more CDNs and agents adopt it.
- An MCP server, if you have data or tools people would query from Claude or similar clients.
- UCP and ACP, if you sell products online.
Ignore for now
- llms.txt as a Google ranking lever.
- A perfect score on any readiness scanner.
- Agent-only copy that says something different from what humans see.
AI Agent Readiness FAQ
Is agent readiness the same as GEO?
No. GEO is about being cited in AI answers; agent readiness is about being usable when an agent arrives to act. They share a foundation, but you can be cited often and still lose the booking because your form defeats the agent.
Does llms.txt make a site agent-ready?
Not on its own. It’s an optional index some tools read. Google says Search doesn’t use it, and it does nothing for an agent blocked by your firewall or stuck in a custom form.
Do I need WebMCP right now?
Only if you run high-value, repeatable tasks such as booking or quoting and have a developer to track a moving draft. The foundation serves every agent today.
How long does the foundation take?
For a typical brochure or small store site, the fetchability, form and bot-rule fixes are usually days to a couple of weeks of work. Heavily client-rendered sites take longer, because the fix is architectural.
Start With the Foundation
Agent readiness is old disciplines graded by a stricter reader: technical SEO, accessibility and sensible security. Get facts into the HTML, make forms operable, decide who gets through the firewall, then add tools. That is the same order the playbook behind SEO Rocket, proven across 1,000,000+ ranking pages, has always used: crawlable first, clever later.
