Crawlers & control files
llms.txt
A proposed file that points AI models to the content you want them to read.
llms.txt is a proposed plain-text file, placed at the root of a site, that offers AI models a curated map of the content you want them to read — typically a tidy list of your most important pages in Markdown. It is meant to help models find and prioritise your best material, in the spirit of an AI-era sitemap.
Adoption so far is skewed toward publishers: no major AI engine — OpenAI, Anthropic or Google — has confirmed it reads the file, and Google has said it is not required for Search. Treat llms.txt as low-cost housekeeping rather than a visibility lever. It does no harm and signals that you are thinking about AI readers, but do not expect engines to act on it today.
It is not a substitute for the fundamentals: a model still has to find your pages citable once it reads them, which comes down to extractable passages, entity clarity and corroboration, not the existence of the file.
Frequently asked
Does llms.txt actually work?
Support is early and uneven across engines, so treat it as cheap housekeeping rather than a reliable ranking factor. It can help models find your key pages but does not replace extractable content and entity clarity.
Related terms
Machine-Readable Content
Content encoded so software can parse its meaning, not just display its text.
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Citation Readiness
How ready a page is to be quoted and attributed inside an AI answer.
Generative Engine Optimization
Optimizing to be cited inside AI-generated answers, not ranked in blue links.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit