Most write-ups on NLWeb vs llms.txt treat them as rival bets, as if you have to pick one format and hope it wins. That framing is wrong. llms.txt, NLWeb and WebMCP sit at three different layers of how an AI agent deals with a website: one is a reading list, one is a question-answering endpoint, and one lets an agent press buttons inside a page. They don’t compete; they barely overlap. This guide sorts out which layer your site needs now, checked against each standard’s own spec and vendor docs as of November 2026.
The short answer
If you run a content site, an llms.txt file is the cheapest of the three and the easiest to undo, but don’t expect it to move rankings or AI citations on its own. NLWeb is worth a pilot if you have rich structured data and want a conversational search box that agents can query too, but it is a running service with real hosting and model costs. WebMCP matters most if your site is an app where people complete tasks (bookings, quotes, support tickets, configurators), and today it is an origin trial, not a shipped standard.
- llms.txt: a static Markdown file that tells a model what to read.
- NLWeb: a server that answers natural-language questions about your content, and doubles as an MCP server.
- WebMCP: a browser API that lets a page register tools an in-browser agent can call.
What llms.txt actually is (and isn’t)
llms.txt was proposed by Jeremy Howard in September 2024, and the proposal lives at llmstxt.org. It is a Markdown file at /llms.txt with a required H1 (the site or project name), an optional blockquote summary, and H2 sections listing links with short notes. A section titled “Optional” marks, by convention, links an agent can skip when it needs a shorter context. The proposal also asks sites to serve clean Markdown versions of pages, for example page.html.md.
# Acme Analytics
> Self-serve product analytics for SaaS teams.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): install and first event
- [API reference](https://example.com/docs/api.md): every endpoint
## Optional
- [Changelog](https://example.com/changelog.md)Two details from the proposal matter for SEO expectations. First, it says llms.txt is used “on demand, when an agent needs information about a topic while assisting a user,” and that the authors expected it to be useful mainly for inference rather than training. Second, the proposal is informal and “open for community input.” It is not a W3C or IETF standard, and no crawler is obliged to read it. Google’s AI features documentation states plainly that “you don’t need to create new machine readable files, AI text files, or markup to appear in these features.” Note also that llms-full.txt, which many plugins generate, is not part of the llmstxt.org proposal itself.
So llms.txt is a curated map for agents that choose to look. It shines on documentation sites, where a developer’s coding assistant genuinely benefits from a clean index. On a marketing site or blog it is low-cost insurance, not a growth lever.
What NLWeb actually is
NLWeb is an open project Microsoft announced at its Build conference in May 2025. Microsoft’s announcement said it “can play a similar role to HTML for the agentic web” and that “every NLWeb endpoint is also an MCP server.” The GitHub repo (nlweb-ai/NLWeb, MIT licence) describes the Python code as a reference implementation and proof of concept, and as of this writing it lists no tagged releases.
The mechanism: NLWeb ingests data you already publish (Schema.org markup and RSS feeds are the intended inputs), stores it in a vector database, and uses an LLM to answer questions. Per the repo’s REST API docs, it exposes two endpoints with the same arguments: /ask for natural-language queries and /mcp for MCP clients. Results come back as JSON with each item’s URL, name, score and the original schema_object. A mode parameter switches between list (default), summarize and generate (classic RAG).
curl "https://example.com/ask?query=vegetarian+recipes+under+30+minutes&mode=list"The repo lists support for several vector stores (Qdrant, Milvus, Azure AI Search, Elasticsearch, Postgres, Snowflake, Cloudflare AutoRAG) and LLM providers (OpenAI, Anthropic, Gemini, DeepSeek and others). Cloudflare’s August 2025 post describes a hosted route: point AutoRAG at your domain, and a Worker serves an NLWeb-style /ask endpoint plus an /mcp endpoint for “trusted AI agents.”
What WebMCP actually is
WebMCP is a proposed browser API, published as a Draft Community Group Report by the W3C Web Machine Learning Community Group, with editors from Microsoft and Google. The draft is explicit that it is “not a W3C Standard nor is it on the W3C Standards Track.” Its abstract is one line: the API “enables web applications to provide JavaScript-based tools to AI agents.”
Two forms exist in Chrome’s docs. The imperative form registers tools in JavaScript with document.modelContext.registerTool(), passing a name, description, JSON input schema and an execute function. The declarative form annotates an ordinary HTML form:
<form toolname="createSupportRequest"
tooldescription="Submits a request for customer support.">
<input type="text" name="query"
toolparamdescription="Describes the support issue.">
<button type="submit">Submit</button>
</form>One correction to older articles: many early write-ups use navigator.modelContext. The current W3C draft and Chrome’s imperative API docs both use document.modelContext. If you copied a snippet from early 2026, check it against the live docs.
Status: Chrome’s developer blog announced the WebMCP origin trial starting in Chrome 149, Microsoft Edge lists its own WebMCP origin trial, and you can test locally with the chrome://flags/#enable-webmcp-testing flag. Chrome’s docs say WebMCP “is under active discussion and subject to change.” The project explainer is also clear that WebMCP is built for an agent working alongside a person in a visible browser, and that fully autonomous agents with no browser UI are a non-goal.
NLWeb vs llms.txt vs WebMCP side by side
| llms.txt | NLWeb | WebMCP | |
|---|---|---|---|
| What it is | Static Markdown file | Server-side Q&A endpoint | Browser JavaScript and HTML API |
| Agent’s job | Read and fetch | Ask and get structured results | Call a tool inside the page |
| Backed by | Community proposal (llmstxt.org) | Microsoft open project, MIT | W3C community group draft; Google and Microsoft editors |
| Status | Informal proposal, widely copied | Reference implementation, no tagged releases | Origin trial from Chrome 149 |
| Runs where | Your web server, static | Your server or Cloudflare | The visitor’s browser |
| Ongoing cost | Near zero | Vector DB plus LLM calls per query | Front-end dev time |
| Relation to MCP | None | Each instance is an MCP server | Borrows MCP vocabulary, in-page |
The cost row is the one most comparisons skip. llms.txt costs an afternoon. WebMCP costs developer time once, then rides on the visitor’s browser. NLWeb is the only one with a per-query bill, because every question runs retrieval and, in summarize or generate mode, an LLM call you pay for.
How each one relates to MCP
The confusion around these three mostly comes from MCP, Anthropic’s open Model Context Protocol, which connects AI clients to tools and data over JSON-RPC (the current spec version is dated 2026-07-28 on modelcontextprotocol.io). Each standard touches MCP differently:
- llms.txt has no protocol at all. Ironically, the MCP documentation site itself points agents to its own llms.txt index, which tells you where llms.txt actually earns its keep: developer documentation.
- NLWeb is MCP on the server side. Any MCP client that can reach your
/mcpendpoint can query your catalogue without opening a browser. - WebMCP is MCP-shaped but browser-native. The explainer says it “shares a common vocabulary with MCP” and is “designed to complement, not replace” backend protocols. Tools run in page script, inside the user’s session, with browser permissions and same-origin rules.
That last point is the practical split: NLWeb serves agents that never visit your page; WebMCP serves agents that are already on it with a logged-in human.
Maturity check: do now, watch, ignore
Here is how I would classify the three for a typical business site today:
- Do now: llms.txt if you have documentation, a knowledge base or a product catalogue an assistant might need to read. Keep it short and curated. Our free llms.txt generator builds one from the sections and links you enter, with no sign-up.
- Watch, pilot if it fits: NLWeb if you already publish clean Schema.org data at scale (recipes, products, listings, events) and want on-site conversational search anyway. The agent-facing
/mcpendpoint then comes along for free. - Watch, prototype if you are an app: WebMCP if users complete multi-step tasks in your UI. The declarative form attributes are a low-risk experiment on one form behind the origin trial.
- Ignore: anyone selling any of the three as a ranking factor.
Which standard fits your site type
Blogs and publishers. Your content is read, not operated. llms.txt is the cheap option; NLWeb only makes sense if you also want conversational search for human readers. WebMCP has little to expose beyond a search box or newsletter form.
Documentation and SaaS help centres. This is llms.txt’s home ground. Add Markdown versions of key pages too, as the proposal suggests. Consider WebMCP for in-app actions like creating a ticket or changing a setting.
E-commerce and listings. NLWeb fits best, because product and listing markup is exactly the structured data it was built to query. If your Schema.org is thin or wrong, fix that first: NLWeb answers are only as good as the objects it retrieves.
Service businesses and lead-gen. WebMCP on the quote or booking form is the interesting experiment. llms.txt is optional. NLWeb is overkill.
What none of them does for SEO
None of these three is a known ranking signal, and none of them guarantees AI citations. Google’s guidance on AI Overviews and AI Mode says no special files or schema are needed. Visibility still comes from crawlable pages with clear answers and accurate structured data. All three standards sit on top of that foundation, and NLWeb depends on it directly.
That is why the first step in practice is an audit, not a new file. SEO Rocket’s site audit runs a real crawler with JavaScript rendering, so you see the structured data and content agents will actually get, and the app maps your site from its XML sitemaps so you can pick the twenty or thirty URLs worth putting in llms.txt instead of dumping the whole sitemap. It is the same playbook proven across 1,000,000+ ranking pages: fix the foundation, then add the layers.
A practical rollout order
- Week one: audit structured data and rendering on your top templates. Fix broken or missing Schema.org.
- Week two: publish a curated llms.txt and, for docs sites, Markdown page versions.
- Month two: if you sell or list things at scale, pilot NLWeb on a subset of content (Cloudflare’s AutoRAG path is the lowest-effort route), and budget for per-query model costs.
- When the trial fits: annotate one high-value form with WebMCP’s declarative attributes, register for the origin trial, and test with the Model Context Tool Inspector extension listed in Chrome’s WebMCP docs.
- Throughout: measure. Watch server logs for agent user agents such as ChatGPT-User, and track whether AI answers cite you. SEO Rocket’s AI-visibility tracking sits in the same chat as your rankings, so you can see whether any of this moves citations rather than assuming it does.
FAQ
Is NLWeb a replacement for llms.txt?
No. In any honest NLWeb vs llms.txt comparison, llms.txt is a static list of links an agent reads, and NLWeb is a live service that answers questions. A site can run both, and they do different jobs.
Is WebMCP the same as MCP?
No. MCP connects AI clients to servers. WebMCP is a browser API that lets a page register tools for an agent working inside that page. It borrows MCP’s vocabulary but runs in the visitor’s browser.
Does Google use llms.txt?
Google’s AI features documentation says you don’t need new machine-readable or AI text files to appear in AI Overviews or AI Mode. Treat llms.txt as being for other agents, not Google rankings.
Which agent standard should a small business start with?
Fix structured data first, then add llms.txt because it is cheap and reversible. Leave NLWeb and WebMCP until you have a specific use case: conversational search for NLWeb, task completion for WebMCP.
The bottom line
The NLWeb vs llms.txt debate dissolves once you see the layers: a reading list, a question endpoint and in-page tools. Pick by what agents need to do on your site, keep your Schema.org honest, and measure before you invest further. For the deeper dives, see our explainers on what WebMCP is, what NLWeb is and how llms.txt works.
