Retrieval & ranking
Retrieval Corpus
The full body of documents a retrieval system searches to answer a query.
The retrieval corpus is the entire body of documents a retrieval system can search when answering a query — the pool from which chunks are fetched. For an open web engine the corpus is its crawl of the public web; for a closed system it might be one company's documentation. If your content is not in the corpus, it cannot be retrieved or cited, full stop.
Getting into the corpus is the first gate of AI visibility, and it is where crawlers and robots.txt matter: a page the retrieval crawler cannot fetch never enters the corpus. Being in the corpus is necessary but not sufficient — once in, you still have to be retrieved and then selected.
The practical checks are straightforward: make sure the retrieval crawlers can reach your key pages and that those pages are indexable.
Related terms
Chunk Retrieval
Fetching the top few most relevant text passages from a vector index for a query.
AI Crawler
A bot that fetches pages to train or feed AI, not to build a search index.
AI Bot Blocking
Using robots.txt and server rules to stop AI crawlers from accessing your site.
Retrieval-Augmented Generation
Fetch relevant documents first, then have an LLM write an answer grounded in them.
Selection Rate
How often an AI cites your page once it's already in the retrieved set.
Want to know where you stand in AI answers?
We’ll show you which engines cite you, and which don’t.
Get a free AI citation audit