Here’s a startling truth: while AI giants like Anthropic and OpenAI are devouring the web’s data at an unprecedented rate, they’re giving back almost nothing in return. But here’s where it gets controversial—is this a fair exchange, or are these companies exploiting the web’s most valuable resource without contributing to its sustainability? Let’s dive into the details and uncover what’s really happening behind the scenes.
Recent data from Cloudflare, a company that supports roughly 20% of the world’s websites, reveals a troubling trend. Anthropic and OpenAI are crawling websites extensively, yet they’re sending very few referrals back to those sites. This crawl-to-refer ratio has worsened since early September, suggesting that AI companies are taking far more than they give. And this is the part most people miss—this imbalance isn’t just about data; it’s about the very foundation of the web’s ecosystem.
Historically, tech companies adhered to an unspoken grand bargain of the web: they could scrape data for free as long as they directed users back to the original sources. This allowed websites to monetize their content through ads, subscriptions, and other methods. However, in the era of generative AI, this deal is crumbling. AI chatbots and answer engines now provide direct responses, reducing the need for users to visit the original websites. The result? Websites lose traffic, revenue, and visibility.
Cloudflare’s data highlights this shift by tracking how often Big Tech bots crawl websites versus how many referrals they generate. For instance, a crawl-to-refer ratio of 100:1 means a company’s bots visit sites 100 times for every referral sent. Is this a fair metric for ethical behavior in the AI era? It’s a question worth debating.
Anthropic, in particular, stands out as a glaring example. According to Cloudflare, its crawling activity has surged since September 2025, while its referrals remain minimal. OpenAI follows a similar pattern, with its crawl-to-refer ratio also deteriorating. This isn’t just about data extraction—it’s about the financial burden these practices place on website owners. Business Insider reported in late 2024 that AI bots were causing traffic costs to spike dramatically, with one web developer seeing a client’s cloud-computing costs double in just a few months.
When approached for comment, Anthropic has questioned Cloudflare’s methodology, claiming it couldn’t confirm the ratios. OpenAI, meanwhile, has remained silent. Both companies have launched web search features for their AI chatbots, which they claim generate more referral traffic. But is this enough to balance the scales?
Here’s a thought-provoking question for you: Should AI companies be required to compensate websites for the data they use, or is the current model sustainable? Let’s not forget that Google, with its traditional search engine, still sends significant traffic to websites, though it’s increasingly integrating AI-driven answers. Does this make Google a better steward of the web, or is it just a matter of time before it follows suit?
As we move forward, Business Insider will continue to monitor Cloudflare’s data to track how this behavior evolves. The stakes are high—the web’s health depends on it. What do you think? Are AI companies nurturing the web or exploiting it? Share your thoughts in the comments below.
Sign up for BI’s Tech Memo newsletter here to stay updated on this and other critical tech issues. You can also reach out to me directly at abarr@businessinsider.com.
Categories: Anthropic | OpenAI | Google | Artificial Intelligence | Generative AI
