Crawlers & control files
llms.txt
A proposed root-level file meant to point AI models at the pages you want read.
llms.txt is a proposed plain-text file, placed at the root of a site, that offers AI models a curated map of the content you want them to read — typically a tidy list of your most important pages in Markdown. It is meant to help models find and prioritise your best material, in the spirit of an AI-era sitemap.
Adoption so far is skewed toward publishers: no major AI engine — OpenAI, Anthropic or Google — has confirmed it reads the file, and Google has said it is not required for Search. Treat llms.txt as low-cost housekeeping rather than a visibility lever. It does no harm and signals that you are thinking about AI readers, but do not expect engines to act on it today.
It is not a substitute for the fundamentals: a model still has to find your pages citable once it reads them, which comes down to extractable passages, entity clarity and corroboration, not the existence of the file.
Key takeaways
- llms.txt is a proposed Markdown file (Jeremy Howard, Answer.AI, September 2024) that offers LLMs a curated map of your site's key content.
- No major AI engine has confirmed reading it, and Google has said it does not use or need llms.txt for Search.
- It is cheap to publish and harmless, but don't treat it as a citation strategy, real AEO work is schema, entities, and content.
A proposal, not a standard anyone reads yet
llms.txt was proposed by Jeremy Howard of Answer.AI in September 2024 and documented at llmstxt.org. The idea is a single Markdown file at `/llms.txt` that gives an LLM a clean, curated index of your most important pages, aimed at inference time rather than at search ranking. It's a reasonable idea on paper: hand the model a tidy map instead of making it parse your whole navigation.
Here's the honest status. No major AI engine has publicly confirmed that it reads llms.txt, and Google's search team has said plainly that it does not use the file and has compared it to the old keywords meta tag. Server-log studies have shown the big assistant crawlers generally don't even request the file. So the "built it, they'll cite you" promise some agencies attach to llms.txt isn't supported by evidence as of late 2026.
That doesn't make it useless. It costs almost nothing to publish, it won't hurt you, and some developer tools and agents do consume it. Just file it under "low-effort, unproven" rather than treating it as the thing that gets you into AI answers. Your citations come from machine-readable content and letting the right AI crawlers in, not from a file the engines may never read.
- # Your Company
- > One-line description of what you do
- ## Docs
- - Product overview: what it is
- - Pricing: plans and costs
llms.txt vs robots.txt
| llms.txt | robots.txt | |
|---|---|---|
| Status | Proposed, unadopted | Established standard |
| Read by major AI engines? | Not confirmed | Yes (named crawlers) |
| Purpose | Curate content for LLMs | Control crawler access |
Example
A client asked us to prioritize an llms.txt file because a competitor "had one." We shipped it in an hour since it's trivial, then spent the real budget on entity optimization and schema, the work that actually moves citations. Six weeks later their AI mentions climbed, and none of it traced to the llms.txt file. The file was a checkbox; the content work was the lever.
Frequently asked questions
Does llms.txt improve my AI visibility?
There's no evidence it does as of late 2026. No major AI engine confirms reading it, and Google says it doesn't use it. Publish it if you like, but don't expect citations from it.
Who created llms.txt?
Jeremy Howard of Answer.AI proposed it in September 2024, with the spec at llmstxt.org.
Is llms.txt the same as robots.txt?
No. robots.txt controls crawler access and is widely honored. llms.txt is a proposed content-curation file that the engines have not committed to reading.
Related terms
Machine-Readable Content
Content labelled and structured so software can read its meaning, not just render it.
AI Crawler
A bot that collects pages to train or feed AI, as opposed to building a search index.
AI Bot Blocking
Using robots.txt and server rules to decide which AI crawlers may fetch your site.
Citation Readiness
A page's overall fitness to be quoted and confidently attributed by an AI answer.
Generative Engine Optimization
Earning a spot as a named source inside AI answers, rather than a rank in the link list.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit