AI crawlers

Which search engines and AI assistants read your site, counted from requests your own server reports. Bots do not run JavaScript, so the browser tag never sees one; a small package on your server does, keeps only the automated requests, and sends them to DataStated to be named, sorted and checked.

Takes about five minutes

What it measures

Server-side fetches by AI assistants, search engines and other bots. Each hit is classified by provider (OpenAI, Anthropic, Google, Perplexity and the rest of the crawler directory) and by purpose: AI answers, an assistant opened the page to answer someone; Search indexing; AI training; and Other, which covers link previews, tools and unnamed bots. An automated request the directory does not name is kept as Other under the tool's own name (curl, python-requests, HeadlessChrome) or the bot word from its user agent.

Where the provider publishes its address ranges, the hit's source address is checked against them: verified when it was inside them, not verified when it was not, which usually means something pretending to be that crawler. Where the provider publishes no list, or the hit carried no address, the hit is could not be checked, and it is never folded into either side. The lists are re-read every six hours, on the worker's pass; a failed refresh keeps the last good list rather than marking every hit unverified. A provider that only offers reverse DNS (Bing's preview and ads fetchers) is not looked up, so those hits cannot be checked either.

Every number is a count of accepted hits, not people. Requests with ordinary browser user agents are dropped by the reporter and ingestion checks. User-agent recognition is not proof of automation; published network ranges provide a separate verification signal where available.

Install the reporter

The published SDK is @datastated/crawl@0.1.0. Reliability changes in the working tree are not in that release. Do not assume a package install includes them; the replacement release is still pending.

Zero dependencies, Node 18 and up, edge runtimes and Cloudflare Workers. It needs an owner API key with the Record events permission (events:write); make one on your account page and keep it on the server.

npm install @datastated/crawl@0.1.0

Next.js

import { createReporter, nextCrawlMiddleware } from "@datastated/crawl";

const reporter = createReporter({ site: "acme.com", key: process.env.DATASTATED_KEY! });

export const middleware = nextCrawlMiddleware(reporter);
export const config = { matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"] };

Already have a middleware? Pass it as the second argument, nextCrawlMiddleware(reporter, myMiddleware), and it runs after the hit is recorded.

Express

import express from "express";
import { createReporter, expressCrawl } from "@datastated/crawl";

const app = express();
const reporter = createReporter({ site: "acme.com", key: process.env.DATASTATED_KEY });
app.use(expressCrawl(reporter));

The 0.1.0 middleware trusts forwarded client-IP headers. Use it for address verification only behind infrastructure that strips visitor-supplied forwarding headers and supplies a trusted client address. Do not enable blanket proxy trust on an internet-facing app. If that boundary is unavailable, omit the address and treat the crawler as unverified. Express reports after the response finishes, so the status code is known.

Hono

import { Hono } from "hono";
import { createReporter, honoCrawl } from "@datastated/crawl";

const app = new Hono();
const reporter = createReporter({ site: "acme.com", key: process.env.DATASTATED_KEY! });
app.use("*", honoCrawl(reporter));

Cloudflare Workers

import { createReporter, workersCrawl } from "@datastated/crawl";



export default {
  fetch(request, env, ctx) {
    const reporter = createReporter({ site: "acme.com", key: env.DATASTATED_KEY });
    return workersCrawl(reporter, (request) => fetch(request))(request, env, ctx);
  },
};

On Workers the send is handed to waitUntil so it finishes after the response; everywhere else the reporter's own timer sends it.

Anything else

import { report } from "@datastated/crawl";

await report(fetch, { path: "/pricing", ua: userAgent, ip, status: 200 },
  { site: "acme.com", key: process.env.DATASTATED_KEY });

report() posts one hit or an array and resolves to the API's answer, or null for a person or a failed request. The batching reporter also takes url (a self-hosted instance), batchSize (100, at most 500), flushMs (5000), fetch and onError. A batch that fails with a network error or a 5xx is posted once more, then dropped and reported through onError; a 4xx is final on the first answer. Configuration errors can throw when the reporter is created. Delivery is best-effort: 0.1.0 has no request timeout or bounded queue, and a lost response can cause a retry to count twice. Do not treat its counts as an exact delivery ledger. The pending SDK release must be verified before documenting those limits as fixed.

What leaves your server

Only requests whose user agent is automated: a bot word, a named crawler or an HTTP library. For each one, the path, the user agent, the client address and the status code when known. The address is used to verify the crawler and stored only as a salted hash, the same daily salt the tag's events use; the address itself never lands. Requests from ordinary browsers never leave your server.

The allowance

Crawl hits have their own allowance of 100,000 accepted hits per site per month (UTC months), separate from your plan's events. They never count toward events and nothing is ever billed for a crawl. Past the line the extra hits in a call are dropped; the response says so in an x-datastated-crawl-limit header and a warning in the body, every successful answer carries x-datastated-crawl-remaining, and the reporter passes the notice to onError.

The AI crawlers panel

The AI crawlers panel appears on the dashboard once your server has reported a hit; a site with only the tag gets no panel. Five tabs rank the same hits: Purpose, Providers, Crawlers, Pages and Files. Crawler rows show hits and the three verification counts, From the provider's addresses, Not from them and Could not be checked; page rows show how many distinct crawlers fetched the path. The Files tab counts fetches of /llms.txt (and /llms-full.txt), /robots.txt, and any sitemap (sitemap*.xml or .txt, gzipped or not, at any depth). The caption under the panel states the verified share in words, over checked hits only.

API, MCP and CLI

The window response carries a crawls section; hits is 0 on a site without the reporter. GET /api/sites/{site}/crawls answers that section alone for a range, as a bare payload with no envelope: range, hits, verification (checked, verified, unverified, unchecked, verifiedShare), categories, providers, agents, paths (most fetched first, up to 50), files, days and a note. POST /api/sites/{site}/crawls is what the reporter calls: {"hits": [{ts, path, ua, ip?, status?, referer?}]}, up to 500 per call, with an owner key holding events:write; it answers {accepted, dropped: {human, over_allowance, invalid}, allowance}. The MCP tools are datastated_analytics_crawls (alias site_crawls) and datastated_crawls_report; the CLI command is datastated crawls <site>, and --json adds the paths and per-day counts.

Questions? Email us at hello@datastated.com.