llms.txt is a Markdown file at /llms.txt that gives AI agents a short overview of your site and links to the content that matters. It is a proposal by Jeremy Howard from September 2024, revised as version 2 in August 2026, and not a ratified standard. For Google Search it does nothing: Google says Search ignores such files and that they neither help nor harm. OpenAI, Anthropic and Perplexity say nothing in their crawler documentation about their bots reading it. One large study found no link to AI citations; in another, 97 percent of the files were never requested at all. Where llms.txt earns its place is developer documentation that coding agents read. If you want to show up in AI answers, open crawler access, indexable pages and clean structure do more.
This article was written by Sebastian, my AI employee for SEO, GEO and blogging. His page says it plainly: trend topics without evidence, such as llms.txt as a supposed ranking lever, he flags as such instead of selling them. On November 6, 2026 he pulled the vendors' own documentation and checked the llms.txt on kevinwelter.com from the outside.
What is llms.txt?
The proposal lives at llmstxt.org, published September 3, 2024, last modified August 10, 2026. The reasoning: web pages are built for people. An agent would rather read a short, curated Markdown summary and follow the links.
The structure is fixed, in this order:
- an H1 with the name of the site or project, the only required part
- a blockquote with a short summary
- optional paragraphs or lists without headings
- sections under H2 headings, each holding a link list in the form
- [Name](URL): note
A section called "Optional" holds links an agent can skip. Version 2 adds three things: the file may sit in a subpath such as /docs/llms.txt and covers everything below it; pages should offer a Markdown version at the same address with .md; and HTML pages can point to their llms.txt with rel="describedby".
Unlike robots.txt, which tells bots what they may fetch, llms.txt controls and blocks nothing. llmstxt.org intends it to be read on demand, when an agent needs information while helping a user, so at inference time rather than for training. In the data below, a training crawler fetches it most. A sitemap lists every page; llms.txt is a curated selection with context.
Who reads llms.txt, and who doesn't
What counts is what the vendors write themselves. As of November 6, 2026.
Google waving the file off in one place and testing for it in another is no contradiction: Lighthouse is a test tool for browser agents, not a Search ranking signal. On Google's "Search Off the Record" podcast, reported June 15, 2026, John Mueller said: "It's basically you're telling these systems, like, I have the best website ever." A file a site writes about itself can't help a system choose between sites. Pressed on the point, he called it "not done for search" and a "temporary crutch" for coding tools, according to Ahrefs.
Telling detail: OpenAI, Anthropic, Perplexity and Google publish llms.txt files for their developer docs. On November 6, developers.openai.com, platform.claude.com, code.claude.com/docs and docs.perplexity.ai served one; the main sites anthropic.com and perplexity.ai did not.
What the evidence shows
No data of our own here. The two largest public studies are third-party sources from SEO tool vendors.
- SE Ranking, close to 300,000 domains (first published November 2025, current version dated October 15, 2026): 10.13 percent had an llms.txt. Neither statistics nor a machine learning model showed any effect on how often a domain is cited in AI answers; dropping the variable made the model more accurate. Their verdict: "At least not yet." To be upfront: I have a partnership with SE Ranking.
- Ahrefs, 137,210 domains, May 2026 (published June 15, 2026, based on server logs from Ahrefs Web Analytics): 28 percent of domains had a file, and 97 percent of those files got no request at all that month. Among named AI tools, GPTBot (training) fetched the file most, followed by Claude Code (coding). Retrieval bots such as OAI-SearchBot and PerplexityBot accounted for only about 1 percent of requests, per Ahrefs. Not one AI bot asked for an llms.txt that didn't exist.
A request doesn't mean the content reached an answer; Ahrefs calls requests "the generous measure". The evidence: no measurable benefit for AI search today, which doesn't mean it can never change.
The llms.txt on kevinwelter.com
My site has an llms.txt, and Sebastian checked it from the outside on November 6, 2026: status 200, 2,613 bytes, plain text. The structure followed the proposal. On the day of the check (as of November 6, 2026) the content had fallen behind: the file, written in German, said the English blog had 26 articles, while the sitemap listed 401 English posts. There were no Markdown versions of pages; requesting a page with .md returned an error page.
Whether bots fetch it, Sebastian can't check: he has no access to the server logs. The honest lesson from our own case: a file nobody maintains goes stale quietly. That is the real cost of llms.txt.
When llms.txt is still worth it
- Developer documentation and APIs. Here there are real readers: coding agents pull docs before writing code. llmstxt.org calls this the heaviest use; in the Ahrefs data Claude Code ranks second among AI fetchers.
- Sites where agents get things done. Mueller allowed for a system already on your site, say to complete a purchase: "If someone is already on your website, maybe some kind of automated system is helpful."
- When your platform creates it anyway. According to llmstxt.org, Wix and docs platforms such as Mintlify generate one automatically, and the WordPress plugins Yoast SEO and AIOSEO can generate one for you. Then it costs you little and, per Google, does no harm.
Anyone selling you llms.txt as a paid GEO measure is selling a hope.
How to create an llms.txt file
- Create the file. A plain text file named
llms.txt at the site root, UTF-8, served as text/plain. Docs under /docs/ can have their own.
- Write the header. An H1 with the name, then one sentence as a blockquote: who you are and what the site offers.
- Curate the links. Only the pages an agent actually needs, each with one line on what's there. Not your whole sitemap.
- Test it. llmstxt.org suggests giving an agent only the file and asking it questions about your offer. With Chrome 150 or later you can also run Lighthouse with "Agentic Browsing".
- Maintain it. Put the file on your website maintenance list, or you end up where we did.
An llms.txt example for a made-up business:
# Example Joinery
> Joinery in Exampletown: built-in wardrobes, kitchens and repairs for homeowners.
## Services
- [Built-in wardrobes](https://example.com/wardrobes): process, wood types, lead time
- [Contact](https://example.com/contact): phone, opening hours, directions
## Optional
- [Projects](https://example.com/projects): photos of finished work
An llms.txt generator drafts a file from your sitemap in seconds, but it doesn't know which pages matter; that part stays your job. The llms-full.txt, with all content in one file, comes from docs platforms such as Mintlify; it is not part of the proposal on llmstxt.org.
What works for AI search instead
- Let the right crawlers in. Don't block the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) if you want to appear in AI answers; the training bots (GPTBot, ClaudeBot) are your call. Google-Extended is a robots.txt token, not a crawler, and per Google doesn't affect Search. Check that your firewall isn't turning bots away.
- Indexable with a snippet. Google says a page must be indexed and eligible to show with a snippet to appear in its AI features.
- Content in the HTML, clear structure, internal links. Answer first, one question per page, pages linked to each other. For discovery, Mueller pointed to normal HTML pages and internal links.
Step by step: Optimizing Your Website for AI Search. Other trend topics: What Is Generative Engine Optimization?. Google's new search interface: Google AI Mode: What It Is and How It Works. Checking it all in an audit: How to Run an SEO Audit With AI. How to find missing internal links is in Internal Linking: Finding the Links You Miss. What structured data does and doesn't do is in Schema Markup Checker: Which Tool Tests What. Whether you show up in AI answers is something you measure: AI Visibility Tracking: How I Measure It.
Frequently asked questions
Is llms.txt an official standard?
No. llmstxt.org itself calls it a proposal, now at version 2. No major AI vendor commits to reading it in its crawler docs.
Can an llms.txt hurt my site?
Not in Google Search; Google says so explicitly. The risk is elsewhere: outdated information in a file nobody looks at anymore.
What's the difference between llms.txt and robots.txt?
robots.txt sets which bots may fetch what, and the vendors' search and training crawlers say they follow it. llms.txt sets nothing; it's a reading suggestion.
Does ChatGPT read my llms.txt?
OpenAI doesn't say. In the Ahrefs data the training crawler GPTBot fetches it most, the search bot OAI-SearchBot hardly at all. For ChatGPT search, OpenAI recommends allowing OAI-SearchBot in robots.txt.
Where to go from here
Check your robots.txt first: if the search bots are allowed, the foundation is in place. If you run developer docs, create the file and test it with an agent. Everything else on the topic is on AI SEO for Google and AI Search. How a hub page ties its articles together is in Pillar Pages and Topic Clusters. And how to write copy that serves both search and readers is in SEO Content Writing with AI.
To see each week how I run my AI employees, pick a resource on my free templates page and sign up there for my newsletter "From the Engine Room". If you'd rather go through your website with me, book a strategy session.
Your first AI employee up and running within an hour
Join the community$5/month founding price, first 50 members
