Web crawlers explained: how they work and 10 common crawlers

A web crawler is a bot that browses the internet to discover, download, and index pages. Also called a spider or spiderbot, it starts from known URLs called seeds, downloads them, and follows their links to find new ones.

At Exa, our search index tracks 2 trillion URLs as of October 2026, and our own crawler, ExaSearchBot, keeps it current.

This guide covers how a crawl works, what crawlers do, ten common crawlers, how to control and verify them, and why we run our own crawler for AI search.

How web crawlers work

A crawler runs the same loop for every URL: fetch the page, parse it, and queue the links it finds.

1. Seed list. A crawl begins with URLs the operator trusts: popular sites, previous crawls, sitemaps, and site-owner submissions. The seed list sets which parts of the web the crawler reaches first.
2. Fetching. The crawler requests each URL over HTTP, identifying itself with a user-agent string. Some also render JavaScript like a browser. Fetch limits vary: Googlebot reads the first 2 MB of a supported file and the first 64 MB of a PDF.
3. Parsing. The crawler extracts text, metadata, and links from the HTML, and notes directives such as a noindex tag.
4. Queueing. New links enter a queue. Already-seen URLs are dropped, and the rest are ranked by importance and update frequency.
5. Politeness. A well-behaved crawler reads the site's robots.txt before fetching, skips disallowed paths, and limits its request rate. Google says Googlebot should access most sites no more than once every few seconds on average.

What web crawlers are used for

Search engines and AI companies crawl the open web. Price trackers and SEO teams run the same loop on narrower targets. Here are some concrete examples:

• Search engines (Googlebot, Bingbot). These crawlers build the indexes search engines rank for queries.
• AI training and retrieval (GPTBot, ClaudeBot). AI companies collect text for model training and use separate bots to retrieve pages for search answers.
• Price tracking. Retailers and comparison sites crawl a fixed list of product URLs on a schedule, extracting the same few fields, such as price and stock level, across many stores.
• Site auditing. SEO teams crawl their own sites to find broken links, redirect chains, missing titles, and orphaned pages, seeing the site as a search crawler does.

10 common web crawlers

These are ten of the user agents you're most likely to find in your logs.

1. Googlebot: Google's crawler for Google Search, mostly run as a smartphone crawler.
2. Bingbot: Bing's standard crawler, run by Microsoft, which handles most of Bing's daily crawling.
3. GPTBot: OpenAI's crawler for content that may be used to train its generative AI foundation models.
4. ClaudeBot: Anthropic's crawler for web content that could contribute to training its models.
5. PerplexityBot: Perplexity's crawler for finding and linking to websites in its search results. Perplexity says it does not collect content to train AI foundation models.
6. Applebot: Apple's crawler for search in Spotlight, Siri, and Safari. Its data may also help train Apple's foundation models.
7. DuckDuckBot: DuckDuckGo's crawler for improving its search results.
8. Amazonbot: Amazon's crawler for improving its products and services. The data may be used to train Amazon AI models.
9. CCBot: The nonprofit Common Crawl's crawler, which builds an open repository of web crawl data anyone can download and analyze.
10. Bytespider: ByteDance's crawler, which Cloudflare's bot reference classifies as an AI crawler.

Search crawlers versus AI crawlers

Search and AI crawlers fetch pages the same way but differ in what they keep and what the site gets back.

• A search crawler builds a link index. It stores enough of each page to rank it, then shows a title and snippet that send readers, and traffic, to the site.
• An AI crawler gathers text. Training crawlers (GPTBot, ClaudeBot, CCBot) collect pages for datasets that teach models language and facts. Retrieval crawlers (OAI-SearchBot, Claude-SearchBot) index pages so assistants can cite them in answers.
• A user-triggered fetcher retrieves one page on request. Because a person initiated the fetch, OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

Operators use separate user agents so site owners can choose which to allow. OpenAI's robots.txt settings work independently, so a site can allow OAI-SearchBot and block GPTBot.

Controlling crawlers on your site

If you're trying to control crawlers on your site, robots.txt is the first place to look. However, whether or not it is actually obeyed is up to the crawler.

• robots.txt directives. A User-agent line names the bot, and Disallow and Allow lines set accessible paths. To block a bot such as GPTBot, ClaudeBot, or CCBot entirely, give it a group with Disallow: /. Support for other fields varies: Google ignores crawl-delay, Applebot and Amazonbot don't follow it, and Anthropic's bots honor it.
• Cloudflare AI Crawl Control. Every Cloudflare plan includes a dashboard that identifies visiting AI services, lets you allow or block each crawler, and tracks crawlers that violate your robots.txt rules.
• Rate limits. If a search crawler overloads your server, set Bingbot's hourly crawl rate in Bing Webmaster Tools or follow Google's instructions to reduce Googlebot's rate.
• noindex. A robots.txt rule stops crawling but doesn't remove a page that's already indexed. To remove it, serve a noindex robots meta tag or an X-Robots-Tag: noindex response header, and leave the path crawlable so the bot sees the directive. If the page is in our index, we remove it after ExaSearchBot's next re-fetch.

Verifying a crawler

User-agent strings are easy to fake, so verify the request itself. Google, Bing, and Apple publish verification steps. OpenAI, Perplexity, and Common Crawl publish IP lists, which can go stale when an operator adds capacity. We instead sign every ExaSearchBot request with HTTP Message Signatures (RFC 9421), which you can validate against our published Ed25519 keys regardless of IP address. Cloudflare, Akamai, and other providers that support Web Bot Auth verify these automatically.

Why an AI search API runs its own crawler

LLMs need search results with relevant passages from the full page, fresh information, and sources that general search engines may overlook. An API that resells another engine’s results inherits that engine’s choices on all three.

That's why at Exa, we built our own search infrastructure to make those choices ourselves. ExaSearchBot discovers and refreshes public web pages, obeys robots.txt, and limits requests to each site. As of August 2026, our index serves 100 billion pages from the URLs it tracks.

Our documentation lists SEC filings, earnings calls, clinical trials, court records, sanctions lists, and CVE databases in the index alongside general web pages.

FAQ

Is a web crawler the same as a web scraper?

A crawler discovers and downloads pages by following links. A scraper extracts specific fields, such as prices, from pages it already has. Many tools do both.