On 8 November 2026 we crawled the 1,000 most popular websites on the Tranco list and checked each one for an llms.txt file. llms.txt adoption came out at 210 of 1,000 sites, or 21.0%. The surprise is where the files are. The very biggest sites trail the pack: only 8 of the 51 sites in the top 100 ranks serve one. Further down, in ranks 251 to 500, it is 36 of 117. Newer agent standards are rarer still. Seven sites publish an MCP discovery file, and five homepages carry WebMCP markers.
This is one of four studies from the same crawl. The others cover AI crawler blocking in robots.txt, homepage schema and XML sitemaps. Every number below comes from that one snapshot; outside claims are linked to their source.
The Headline: 210 of 1,000 Top Sites Serve an llms.txt
An llms.txt is a Markdown file at the root of a domain. It gives language models a short, curated map of a site: a title, a summary and links to the pages that matter. If the idea is new to you, our explainer on what llms.txt is covers the format. In this study, a site counted as having one if /llms.txt returned a text file with real content rather than an HTML error page or an empty body.
By that test, 210 of the 1,000 sites had one, or 21.0%. For comparison, 905 of the same 1,000 sites returned a robots.txt. llms.txt, a 2024 proposal, has real momentum, but it is still an opt-in extra for about one top site in five.
Adoption Is Lower at the Very Top
You might expect the biggest brands to lead on new web standards. They don’t. The chart below splits the 1,000 sites into four bands by Tranco rank.
- Ranks 1–100: 8 of 51 sites (15.7%)
- Ranks 101–250: 16 of 70 sites (22.9%)
- Ranks 251–500: 36 of 117 sites (30.8%)
- Ranks 501–2,293: 150 of 762 sites (19.7%)
Read the top bands with care, because the counts are small. Only 51 of the top 100 ranked domains gave us a crawlable website; the rest are CDNs, APIs, ad hosts, duplicates of a brand already counted, or sites that refused our crawler. Three or four sites move that bar by several points. Even so, adoption peaks in ranks 251–500 (30.8%) and is lowest in the top 100 (15.7%).
Our reading of the list of adopters below suggests why. The top 100 is heavy with consumer platforms, search engines and social networks, which have little need to give an AI model a reading list. The adopters we can name just below the top 100 are mostly developer-tool, SaaS and infrastructure companies. Their documentation is exactly what llms.txt was designed for.
What the Files Actually Look Like
Having an llms.txt is one thing. Having one that follows the format is another. The format asks for a Markdown H1 title followed by sections of links. Of the 210 files we found:
- 204 (97.1%) start a line with a Markdown H1 (
# Title). - 188 (89.5%) contain at least one Markdown link to a full URL, which leaves 22 files without one.
- The median size is 9,459 bytes, about 9.5 KB, or a few pages of plain text.
Most adopters ship real indexes, not placeholders. The usual miss is a file with no links: it describes the company but points a model nowhere.
These checks matter more now because Google’s own tools have started testing them. Lighthouse added an llms-txt audit in a new “agentic browsing” category. Its source code fails a file that has no H1, no Markdown link or fewer than 50 characters. Our checks are similar but not identical (we only counted links to full URLs), so treat them as a rough guide, not a Lighthouse pass rate.
llms-full.txt and the <link> Tag Barely Register
Two companion ideas have not caught on. The first is llms-full.txt, a single file that holds a site’s full documentation as Markdown so a model can load it all at once. Only 41 of the 1,000 sites (4.1%) serve one. A full-content file makes sense for documentation, not for a news homepage or a shop.
The second is pointing to llms.txt from the HTML itself, with a <link> element in the page head. Only 5 of the 1,000 homepages (0.5%) do this. Discovery happens by convention instead: tools check the root path, the same way crawlers request /robots.txt unprompted. A working file at the root matters far more than a discovery tag.
Who Has One: The Sites Behind the Number
The highest-ranked adopters in our sample, with their Tranco ranks, include cloudflare.com (2), github.com (34), fastly.net (41), adobe.com (60), wordpress.org (67), digicert.com (75), opera.com (78) and wordpress.com (100). Just below the top 100 are sentry.io (103), samsung.com (120), dropbox.com (133), ubuntu.com (167), shopify.com (170), paypal.com (173), stripe.com (208), roku.com (226) and weather.com (230).
The list leans toward infrastructure, developer platforms, payments and software, companies with large API docs and developer audiences. weather.com and samsung.com show the file has spread further, but the core is still technical documentation. If your site is a local service business or a small online shop, the “top sites have one” argument applies to you less than the headline suggests.
Beyond llms.txt: MCP and WebMCP Are Still in Single Digits
We also checked for two newer ways a site can talk to AI agents. The chart below puts them next to llms.txt, and the scale difference is stark.
MCP discovery files. A file at /.well-known/mcp.json can advertise a site’s Model Context Protocol server to AI clients. After we checked each response by hand and removed five false positives (error messages, unrelated JSON and HTML pages), 7 of 1,000 sites publish one: cloudflare.com, sentry.io, forter.com, twilio.com, usercentrics.eu, notion.com and kraken.com. No discovery path is standardised yet. As we covered in our guide to MCP servers for content sites, the Server Card proposal is still a draft, so these seven are early adopters of a moving target. We run a public MCP server ourselves (see the SEO Rocket MCP server), and we would still call this experimental.
WebMCP. WebMCP lets a web page expose tools that a browser-based agent can call. WebMCP adoption is just as thin: markers on 5 homepages. forter.com uses the declarative form, with a toolname attribute on a form. usercentrics.eu, onrender.com, hostinger.com and wpastra.com reference the JavaScript API. A string match is not proof of working tools. Some of these may be feature checks or vendor scripts rather than registered tools.
A few names appear more than once. forter.com shows up in all three lists: llms.txt, an MCP file and WebMCP. cloudflare.com and sentry.io have both llms.txt and an MCP file. Right now, agent-ready websites are a small group of developer-facing companies trying everything, not a broad shift. Our NLWeb vs llms.txt vs WebMCP comparison explains which standard fits which kind of site.
Does Any AI Engine Actually Use llms.txt?
This question drives most llms.txt adoption decisions, and the answer is less exciting than 21% suggests. No major AI engine has confirmed that it uses llms.txt to rank or cite sources.
- Google says it plainly. Its guide to optimising for generative AI features puts llms.txt under “what you don’t need to do”. It says you don’t need “new machine readable files, AI text files, markup, or Markdown to appear in Google Search,” while adding that it’s “completely fine” to keep maintaining llms.txt files. The AI features documentation says the same about AI Overviews and AI Mode.
- OpenAI’s crawler documentation describes control through robots.txt rules for each user agent. It does not present llms.txt as a signal its bots read, even though OpenAI publishes an llms.txt for its own docs.
- Anthropic and Perplexity: we found no statement from either that their crawlers or answer engines read other sites’ llms.txt files for ranking or citations.
That fits the adoption pattern. An llms.txt for your docs helps developers who paste it into AI coding tools and agents that choose to read it. Lighthouse auditing the file shows Chrome takes agent use seriously; it is not a search ranking signal. Our piece on llms.txt vs robots.txt covers the difference between a file that controls access and one that only suggests.
What This Means for Your Site
Three practical takeaways from the data:
- If you have docs, an API or a product people build with, publish one. That is the profile of the adopters we can name, and it is where the file is actually used. Keep the H1, link to real pages with absolute URLs, and keep it to an index rather than a dump. The median adopter’s file is under 10 KB. SEO Rocket’s free llms.txt generator builds a correctly formatted one from your site.
- If you are a local business or small shop, it is optional. It costs little and can’t hurt, but 79% of top sites don’t have one, and Google says it is not needed for its AI features. Your time is better spent on crawlable pages, clear entities and making sure AI crawlers aren’t blocked by mistake.
- Measure citations, not files. A file only matters if AI answers start citing you, and that can be tracked. SEO Rocket’s AI Visibility tracking checks how often ChatGPT, Google AI Overviews, Gemini and Perplexity mention your brand, so you can see whether any change, llms.txt included, moved anything. Our guide to AI visibility tracking covers the method.
Hold off on MCP discovery files and WebMCP unless you build agent integrations on purpose. At 7 and 5 sites out of 1,000, waiting costs you nothing. For a fuller checklist, see how to make a website agent-ready.
Methodology
Sample. We started from the Tranco top-1M list (list ID Q2VV4), a research ranking that averages several popularity lists. We crawled ranks 1 to 2,300 on 8 November 2026 and kept the first 1,000 unique websites. A site counted if its homepage returned HTML. We de-duplicated by final origin after redirects, so country domains that redirect to one brand count once. Many top-ranked domains are CDNs, APIs or ad hosts with no website, hence 2,300 ranks for 1,000 sites. The last site included is rank 2,293.
What we fetched. For each site we made one request per URL, with a crawler identifying itself as SEORocketStudy/1.0 and a 10-second timeout. We fetched the homepage HTML, /robots.txt, /llms.txt, /llms-full.txt and /.well-known/mcp.json.
How we classified. An llms.txt or llms-full.txt counted only if it returned text content rather than an HTML page and had more than 20 characters after trimming. “Has an H1” means a line starting with # . “Has links” means at least one Markdown link to an http(s) URL. Every MCP discovery file was checked by hand, and 5 false positives were excluded. WebMCP markers are string matches in the homepage HTML: navigator.modelContext in scripts, or a toolname attribute on a form. The rank-band percentages use the number of sites in each band as the base. All other percentages use the 1,000 sites unless we state a different base. The raw dataset is available on request: contact us for the dataset.
Limitations
- One snapshot. This shows 8 November 2026, not a trend.
- Tranco is not “the web”. It is a popularity proxy weighted toward big platforms and infrastructure brands. Infrastructure and developer brands are over-represented, so adoption across the wider web is likely lower.
- Some sites block unknown crawlers or serve bots different content. A few homepages failed for that reason and were skipped. A site that blocked our crawler on
/llms.txtbut not its homepage would be counted as not having one. - WebMCP detection is homepage-only and string-based. Tools on inner pages would be missed. We matched the original
navigator.modelContextentry point, and the current WebMCP draft has moved the API todocument.modelContext. Our count of 5 could therefore miss newer sites and include sites that only check whether the feature exists. - Small top bands. The top-100 band has 51 sites and the 101–250 band has 70. Read their differences as direction, not precision.
- Having a file is not the same as it being used. We measured whether sites publish llms.txt, not whether any AI system reads it.
FAQ
What percentage of websites have an llms.txt file?
In our crawl of 1,000 top sites from the Tranco list on 8 November 2026, 210 (21.0%) served an llms.txt. Adoption was lowest in the top 100 ranks (8 of 51 sites) and highest in ranks 251–500 (36 of 117).
Does llms.txt help you rank in Google or AI Overviews?
No. Google’s AI optimisation guide lists llms.txt among things you don’t need to appear in its generative AI features, while saying it’s fine to keep one. No other major AI engine has confirmed it uses the file for ranking or citations.
How many sites use MCP discovery files or WebMCP?
Very few. Seven of 1,000 sites publish a verified /.well-known/mcp.json discovery file, and five homepages carry WebMCP markers. Both standards are still drafts, and adoption so far comes almost entirely from developer-focused companies.
