What is AI crawler?
An AI crawler is a bot an AI company uses to fetch pages. Public examples include OpenAI’s GPTBot, Anthropic’s ClaudeBot, and PerplexityBot. Sites allow or limit them in robots.txt. A fetch by one of these bots is not the same event as a citation in a consumer answer.
Why it matters for AI visibility
Crawlers are how some AI products discover and refresh pages. They matter for AI visibility because a page that cannot be fetched cannot be newly retrieved by that product. They are not all doing the same job. Some crawlers are associated with model training, and live search products may use a different bot from the training bot. Blocking GPTBot, for example, is a statement about that crawler’s fetches. It does not erase your brand from answers that rely on other sources, earlier training, or a search crawler you still allow. Read each company’s current crawler documentation before you write a rule, and say in your own notes what you intended to allow. Then judge success by whether answers cite the page, not by whether a bot requested it once. A crawl is access. A citation is selection.
Example
A site blocks GPTBot and allows the crawler it believes a live answer product uses. Server logs later show fetches from the allowed bot and none from GPTBot. A citation check still has to be done in the product itself. The log only shows fetching, not whether an answer used the page.
Related terms
llms.txt is an optional overview file a crawler might read. It does not grant or deny access. Training data versus live retrieval explains why blocking a training crawler and blocking a search crawler have different effects. Grounding only happens if some retriever can still fetch a source.