Crawler directory

Every crawler DataStated can name: the provider, the user-agent token, what the crawl is for, and where the provider publishes the addresses a request can be checked against. This page is generated from the registry the classifier runs on, so it is the same list your dashboard uses.

The registry names 50 agents from 17 providers and points at 18 published address lists. Each documentation link and each list was read from the provider's own site on September 7, 2026, and the token, purpose and list address were taken from that page. Where a provider publishes nothing, the entry says so, and hits from that crawler show as "could not be checked", never as confirmed. The address lists are re-read every six hours.

A token matches as a whole word, case-insensitively, so Googlebot-Image/1.0 is Googlebot-Image and never Googlebot, and NotGPTBot is nothing. A more specific token always sits ahead of the one it extends, and the first match wins. Anything automated that no entry names is kept as Other: well-known HTTP libraries and headless browsers under their own name, anything else under the bot word in its user agent.

Purposes

• AI answers (ai_answer): a user asked an assistant something and it opened the page to answer.
• Search indexing (search_index): building a search or answer index.
• AI training (ai_training): collecting a corpus to train models.
• Other (other): link previews, ads checks, tools, anything else.

The directory

Verification is published addresses when the provider publishes a list and a hit's address is checked against it, stated range when the provider gives its range in prose and that range is checked, reverse DNS when the provider offers only that (not looked up, so hits cannot be checked), and none when nothing is published.

Questions? Email us at hello@datastated.com.