The Economics of AI Web Crawling in 2026
How much does it cost to crawl the web? Analysis of compute, bandwidth, and legal costs for AI training crawlers.
The Scale of AI Crawling
In 2026, the major AI companies collectively crawl billions of pages per day. OpenAI's GPTBot, Anthropic's ClaudeBot, Google's various crawlers, and Meta's AI training scrapers are the largest consumers of web bandwidth after traditional search engines. This crawling isn't free — it costs the AI companies real money and costs website operators in bandwidth and compute.
Cost to the Crawler
Running a web crawler at scale requires: compute for HTTP requests and HTML parsing ($0.01-0.05 per 1000 pages), bandwidth for downloading content ($0.05-0.12 per GB), storage for indexing and processing ($0.02 per GB/month), and legal/compliance teams to handle robots.txt and copyright. At billion-page scale, this adds up to millions per month.
Cost to Website Operators
Every bot visit consumes server resources. For sites on serverless platforms like Vercel or Cloudflare, each bot request costs $0.000001-0.00001 in compute. Sounds tiny, but a popular site getting 100K bot visits/day pays $30-300/month just serving bots. Larger sites report 40-60% of their traffic being bots.
The robots.txt Economy
robots.txt has become the de facto negotiation tool between sites and AI crawlers. Some publishers block all AI bots. Others allow specific crawlers in exchange for partnerships (like news organizations granting access to Google in exchange for traffic). The legal status is still evolving — several lawsuits are testing whether ignoring robots.txt constitutes trespass.
Who Benefits?
The economics are asymmetric. AI companies extract billions in training data value while website operators bear the bandwidth costs. Some sites are fighting back: serving different content to bots, implementing CAPTCHAs, or demanding licensing fees. Others see bot traffic as free promotion — if ChatGPT or Claude references your site, that drives real traffic.
Our Data
From our tracking on global-chat.io: 9 bots generate ~220 visits across a small site. Extrapolating, a site with 100 pages and decent SEO can expect 500-2000 AI bot visits per month. The bandwidth cost is negligible ($0.01/month), but the opportunity cost — what if those bots drove real referral traffic? — is the interesting question we're testing.