How to Block AI Crawlers in 2026 (robots.txt + llms.txt)

There is a specific moment almost every developer running a real website eventually hits. You open your server logs or your Cloudflare analytics dashboard to chase down a slow endpoint or a mysterious spike in bandwidth, and instead of finding a traffic surge from a blog post going viral, you find page after page of requests from user agents you have never seen before. GPTBot. ClaudeBot. Bytespider. Amazonbot. PerplexityBot.

Dozens of these requests per second, hitting every route on your site, ignoring your pagination, crawling straight through your search results pages, and showing absolutely no sign of ever stopping. You did not sign up for this traffic. Nobody asked your permission. And now it is your job to figure out what to do about it.

This guide is written for that exact moment. It is a practical, technical walkthrough of how AI crawlers actually work, which ones matter, how to write a robots.txt file that reflects an actual decision instead of a guess, why that file alone will not protect you, how to verify a crawler is telling the truth about who it is, how to block the ones you do not want at the server and edge level, and whether the newly hyped llms.txt file is worth your time. It draws on official documentation from Google, Cloudflare, and Vercel, along with independent adoption research, so that by the end you can make a genuinely informed call instead of copying a robots.txt snippet from a forum post and hoping for the best.

You are not imagining it, and you are in good company

If the scene described above feels dramatic, it is worth knowing that some of the most respected infrastructure teams on the open web have been living through a much worse version of it for two straight years now, and their numbers put real weight behind what would otherwise sound like an exaggeration.

Documentation host Read the Docs, which serves technical documentation for a huge share of the Python ecosystem on a genuinely modest budget, published its own bandwidth numbers publicly rather than just complaining in the abstract. After blocking a set of aggressive AI crawlers, the team reported bandwidth for its downloaded files dropped 75%, falling from roughly 800 gigabytes a day down to around 200. One single crawler had downloaded 73 terabytes of zipped HTML files in a single month, a bug in that crawler’s own code that caused it to repeatedly redownload files that had never changed, since it ignored the standard caching headers built specifically to prevent exactly that kind of waste. The team put a direct dollar figure on the damage from just one of these incidents: over 5,000 dollars in bandwidth charges from a single misbehaving bot.

Drew DeVault, the founder of the open source git hosting platform SourceHut, described his own experience in blunt, memorable terms in a blog post whose title alone tells you everything about the emotional toll of this problem: please stop externalizing your costs directly into my face. DeVault reported spending between 20 and 100 percent of his time in any given week simply mitigating aggressive crawler traffic, describing dozens of brief service outages every single week as a direct result. SourceHut eventually resorted to blocking entire cloud provider networks outright, including Google Cloud and Microsoft Azure in their entirety, according to reporting from The Register, simply because so much of the abusive traffic originated from those networks that individual filtering had stopped being practical.

The GNOME project, which maintains one of the most widely used desktop environments in the Linux world, faced a similar breaking point on its own GitLab infrastructure and responded by deploying a tool called Anubis, a proof of work challenge system built by developer Xe Iaso specifically to force automated crawlers to spend real computational effort before they can access a page, effectively raising the cost of scraping high enough to make it uneconomical at scale. Iaso later described the futility of relying on simpler blocking methods alone in a post bluntly titled a desperate cry for help, writing plainly that it is futile to block AI crawler bots because they lie about who they are, rotate through residential proxy addresses, and generally do whatever it takes to blend in with ordinary human visitors. A separate industry report cited in that same piece, compiled by LibreNews, found some open source projects now see as much as 97% of their total traffic originating from AI crawlers alone, a figure that turns the entire framing of this problem upside down. On sites like that, the bots are not an occasional nuisance mixed into normal traffic. They are effectively the entire audience, with real human visitors reduced to a rounding error.

This is not only an open source problem either. Technology publication Trusted Reviews crashed multiple times in a single day in August 2025 after AI crawlers scraped the site 1.6 million times within twenty four hours, according to reporting from Press Gazette covered by Media Copilot. Chris Dicker, chief executive of the site’s parent company Candr Media Group, noted publicly that all that scraping activity had translated into just 603 actual human visitors arriving from generative AI platforms afterward, a conversion rate so small it barely registers, and dramatically lower than what the same site would typically see from a comparable volume of traditional search engine crawling.

None of these teams are careless, under resourced amateurs. They are some of the most technically capable infrastructure operators on the internet, and they are still getting overwhelmed. Keep that firmly in mind as you read the rest of this guide. Nothing here promises to make this problem disappear completely. It promises to meaningfully reduce it, the same way it has for teams facing a genuinely worse version of the exact same fight.

The three kinds of AI crawlers, and why the difference matters

Before touching a single configuration file, it helps enormously to understand that not all AI crawlers do the same job, because blocking the wrong one can quietly cost you far more than the bandwidth it saves.

The first category is training crawlers. These bots exist purely to harvest content that gets folded into a future model training run. GPTBot from OpenAI, ClaudeBot from Anthropic, the Google Extended token that governs Gemini and Vertex AI training, Applebot Extended, and Meta’s external agent all fall into this bucket, alongside the nonprofit Common Crawl bot, CCBot, whose archives have quietly fed the training data of GPT, Llama, Mistral, and most other major language models at some point. Blocking a training crawler costs you essentially nothing in the short term. It simply keeps your specific pages out of whatever dataset that company builds next.

The second category is search and retrieval crawlers. These are the bots that build the index a chatbot actually searches through when it answers a live question. OAI SearchBot powers ChatGPT’s search feature, Claude SearchBot supports Claude’s web answers, and PerplexityBot builds the index behind Perplexity’s results. Blocking one of these is a completely different decision than blocking a training crawler, because it does not just opt you out of a future dataset. It removes your content from eligibility to ever be cited in that assistant’s answers today, in front of a growing number of people who are increasingly asking questions inside a chat window instead of typing them into a traditional search box.

The third category is user triggered fetchers, sometimes also called browsing agents. ChatGPT User, Claude User, and Perplexity User only fire when an actual human, sitting in front of ChatGPT, Claude, or Perplexity, asks a specific question that requires looking at your specific page right now. Blocking these does not protect you from bulk scraping at all, since each request is tied to one real person’s one real question. It simply breaks that person’s experience the moment they try to get an answer that depends on your content.

Treating all of these as one undifferentiated wave of “AI bots” is the single most common mistake developers make here, and it usually results in either blocking everything, which quietly removes a site from an entire emerging category of discovery, or blocking nothing, which is the default most platforms shipped with until quite recently.

The complete AI crawler user agent reference for 2026

Reference lists like this one age quickly, since new user agents appear roughly every few months as new AI products launch, so treat this as a solid starting point rather than a permanent list, and revisit official documentation from each vendor periodically to confirm nothing has changed.

OpenAI runs three distinct user agents. GPTBot handles training data collection. OAI SearchBot populates ChatGPT’s search feature. ChatGPT User fires when a live ChatGPT session fetches a specific page on a user’s behalf.

Anthropic follows a nearly identical three way split. ClaudeBot, and the older anthropic ai token, handle training data collection. Claude SearchBot supports Claude’s web answer feature. Claude User fires for live, user triggered fetches.

Google’s system works a little differently, and this is the point developers most commonly get wrong. Googlebot is the classic search indexing crawler, and it also feeds Google’s AI Overviews and AI Mode results, because both of those features run through the exact same crawl infrastructure as regular search. Blocking Googlebot removes you from Google Search entirely, not just from AI features, so it is almost never the right move. Google Extended is a separate robots.txt token, not a separate crawler, that specifically controls whether Gemini and Vertex AI are allowed to use your content for model training. You can safely disallow Google Extended without affecting your Google Search rankings at all, since it only opts you out of the training use case.

Perplexity runs PerplexityBot for building its search index and Perplexity User for live, user triggered fetches. As covered in detail elsewhere, Perplexity has a well documented history of continuing to access content even after both of these named agents were explicitly blocked, a point worth remembering as you read the rest of this guide.

Beyond the big three AI labs, a handful of other crawlers show up constantly in server logs. Amazonbot crawls on behalf of Amazon’s various AI and shopping features. Applebot Extended governs Apple’s generative AI training specifically, distinct from the classic Applebot that powers Siri and Spotlight search. Meta’s external agent, written in logs as meta externalagent, collects training data for Meta’s AI products. Cohere AI and Diffbot both run smaller but persistent training crawlers. And then there is Bytespider, ByteDance’s crawler, which security infrastructure firm HAProxy reported was responsible for nearly 90% of all AI crawler traffic across its customer base in 2024, with a well documented pattern of ignoring disallow rules entirely, according to HAProxy’s own published analysis referenced by agency Soar. If you only have the energy to explicitly block one crawler today, developers who track this closely consistently point to Bytespider as the one worth prioritizing.

How to write a robots.txt file for AI crawlers

Once you know which category each crawler falls into, writing the actual file becomes a matter of making three separate decisions rather than one blanket choice.

If your priority is keeping your specific content out of future model training runs while still remaining visible in AI powered search results and chatbot citations, your robots.txt should disallow the training specific tokens while explicitly allowing the search and retrieval tokens. That looks roughly like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

If your priority is broad visibility across every AI assistant and you are comfortable with your content contributing to future training data in exchange for that visibility, a simpler blanket allow list covering the major, compliant crawlers is more appropriate, and most commercial sites without a specific licensing concern land here by default according to a 2026 configuration guide published by agency Crawlcrawl.

And if your priority is keeping your content out of AI entirely, whether for legal, competitive, or licensing reasons, you disallow everything explicitly, including the search and retrieval bots, fully understanding that this also means your content becomes ineligible for citation inside any of these tools going forward.

None of these three approaches is objectively correct. They represent a genuine tradeoff between training data control, present day visibility, and total exclusion, and the right answer depends entirely on your specific business, not on whichever configuration happens to be pasted into the top result for a quick search.

Why robots.txt alone will not save you

Here is the part most quick tutorials skip entirely, and it matters more than anything covered so far. Robots.txt has no enforcement mechanism whatsoever. It works only because the overwhelming majority of crawler operators voluntarily choose to respect it, in roughly the same way most drivers voluntarily stop at a red light even when no camera is watching. OpenAI, Anthropic, and Google all publicly commit to honoring it and generally do, largely because ignoring it carries real reputational risk for companies that size. Smaller or less accountable operators face no such pressure.

The Bytespider statistic mentioned earlier is the clearest illustration of this in practice, but it is far from the only one. Cloudflare separately ran a direct investigation into Perplexity’s crawling behavior after receiving customer complaints, and found the crawler continuing to access sites that had explicitly disallowed it, using rotating IP addresses and a browser fingerprint disguised as ordinary Chrome running on a Mac specifically to avoid being identified and blocked again. That investigation ended with Cloudflare delisting Perplexity’s crawler from its Verified Bots program entirely. The lesson generalizes cleanly. A disallow rule in robots.txt is a polite request, honored by well behaved operators and simply ignored by anyone determined enough to keep scraping anyway. Treat it as your first line of defense and your policy of record, never as your actual security boundary.

How to verify a crawler is actually who it claims to be

Because user agent strings can be typed by literally anyone and cost nothing to fake, checking the User Agent header alone tells you almost nothing trustworthy about who is really making a request. A malicious scraper can claim to be GPTBot just as easily as it can claim to be Googlebot, and plenty do specifically to slip past simple allow lists built for legitimate crawlers.

Google’s own developer documentation lays out the correct method clearly, and the same technique applies equally well to any other major crawler operator that publishes verification data. The process, documented directly by Google, is called forward confirmed reverse DNS. You take the IP address from your server logs, run a reverse DNS lookup on it using a standard command line tool like host or dig, and check whether the resulting hostname ends in a domain the company actually controls, such as googlebot.com or google.com for Google’s crawlers. Critically, you do not stop there, because an attacker who controls their own DNS could theoretically fake that first step. You then run a forward DNS lookup on that hostname and confirm it resolves back to the exact same IP address you started with. An attacker cannot fake this second step without also controlling the actual DNS records for the domain being impersonated, which is precisely what makes this method reliable rather than just theoretically sound.

The practical downside is speed. Running two DNS lookups on every single request that claims to be a bot adds real latency, particularly at scale, which is why production implementations almost always cache verified and rejected results so the same IP address is not looked up on every single visit. Infrastructure vendors have built entire small products around exactly this pattern. An Nginx module documented by GetPageSpeed performs the user agent check, runs the reverse and forward DNS lookups only when a request claims to be a known bot, and caches the result in Redis to avoid repeating the lookup on every subsequent visit from the same address, while normal, non bot labeled traffic passes through untouched with zero added latency. HAProxy Enterprise ships a comparable feature under the name Verify Crawler, built specifically because, as the company puts it plainly in its own documentation, impersonating a well known search engine crawler has historically been an easy way for attackers to evade detection simply because so few site operators had the time to verify claims like this themselves.

If building this yourself feels like more infrastructure than your team wants to own, most major AI labs and search engines also publish their crawler IP ranges directly in machine readable JSON format, letting you skip the DNS lookup step entirely and simply check whether an incoming request’s IP address falls inside a published range. That approach is faster per request, since it is a simple range comparison rather than a live DNS query, though it does require you to keep your local copy of those ranges reasonably fresh as providers occasionally update them.

Blocking AI crawlers at the server and edge level

For most production sites, the most practical and lowest maintenance approach is not building custom verification logic from scratch. It is using the tools already built into your existing infrastructure provider, since companies like Cloudflare and Vercel already maintain their own continuously updated directories of verified bots and update them as new crawlers appear, sparing you from tracking every new user agent by hand.

Cloudflare’s Block AI Bots feature, documented in its developer reference, is a single managed rule that blocks a maintained list of AI crawlers including Amazonbot, Applebot, Bytespider, ClaudeBot, DuckAssistBot, Google’s CloudVertexBot, GoogleOther, GPTBot, Meta’s external agent, PetalBot, TikTokSpider, and CCBot, alongside unverified bots that behave in ways consistent with that category, all without you needing to maintain the underlying list yourself. It ships with three simple configuration options, letting you turn blocking off entirely, apply it across your whole site, or restrict it specifically to pages that carry advertising, and as of mid 2025 this blocking rule became the default setting applied automatically to every newly created Cloudflare domain, a shift the company publicly framed as Content Independence Day.

Cloudflare also runs a genuinely clever secondary defense called AI Labyrinth, built specifically for the non compliant crawlers that ignore standard blocking directives entirely. Rather than serving those bots a blunt error page, which tends to simply signal to the operator that they have been detected and prompt them to rotate infrastructure and try again, AI Labyrinth quietly feeds misbehaving crawlers into an endless maze of realistic looking but entirely meaningless AI generated pages, wasting their computing budget instead of tipping them off. Human visitors never see these decoy links at all, and they carry nofollow tags specifically so they cannot affect your search rankings, while legitimate, well behaved crawlers like Googlebot and Bingbot simply never trigger the trap in the first place because they respect the disallow rules pointing away from it.

If your stack runs on Vercel, the equivalent tool is called BotID, and it takes a fundamentally different approach worth understanding, since it focuses on behavioral detection rather than user agent matching. According to Vercel’s own documentation, the basic mode runs invisibly and for free, actively identifying sophisticated automation tools like Puppeteer and Playwright that never announce themselves through a suspicious user agent at all, while a deeper analysis mode, priced at roughly a dollar per thousand requests, layers in more advanced behavioral signals for genuinely high value routes like checkout and payment endpoints. Critically, the same tool exposes whether a request comes from a recognized, verified agent such as ChatGPT’s browsing feature, letting you deliberately allow that specific traffic through while still blocking everything else automated, which solves the exact allow versus block precision problem robots.txt alone cannot handle on its own. A typical integration is genuinely small, often just a single line wrapping your existing Next.js configuration plus a short server side check inside the specific route you want to protect.

If you are running your own infrastructure directly rather than sitting behind a managed platform, the same layered principle still applies even without any of these specific products. A basic Nginx or Next.js middleware layer can inspect the incoming user agent string and immediately reject any request matching a known, non compliant crawler signature before it ever reaches your application code, which is exactly the pattern demonstrated in Vercel’s own official middleware examples for user agent based routing decisions at the edge. A minimal version of that middleware pattern in Next.js looks roughly like this:

export function middleware(request) {
  const userAgent = request.headers.get('user-agent') || ''
  const blockedPatterns = /Bytespider|PetalBot|MJ12bot/i

  if (blockedPatterns.test(userAgent)) {
    return new Response('Blocked', { status: 403 })
  }
}

This kind of simple pattern match is fast, costs essentially nothing to run, and stops any crawler honest enough to identify itself accurately in its user agent string. It will do nothing at all against a crawler willing to lie about who it is, which is exactly why the verification methods covered earlier in this guide, and the behavioral tools covered just above, exist as a second and third layer rather than a replacement for this first one.

Does llms.txt actually work

Given how much attention llms.txt has received across developer forums and marketing blogs over the past year and a half, it deserves an honest, evidence based answer rather than another round of hype, because the actual data tells a genuinely different story than the buzz around it suggests.

For anyone unfamiliar, llms.txt is a proposed convention, not an official standard backed by any recognized standards body like the W3C or IETF, where a site publishes a plain Markdown file at the root of its domain summarizing its most important content in a format theoretically easier for a language model to parse quickly than a full HTML page. It is philosophically similar to a sitemap, but aimed at machine readers of a very different kind.

The honest problem is adoption and actual usage, and the numbers here are considerably worse than most coverage of the topic lets on. A study covering roughly 300,000 domains by SE Ranking found an llms.txt adoption rate of just 10.13%, and among the fifty most frequently cited domains inside actual AI search results, only a single one had bothered to publish the file at all, a detail reported by agency Derivatex in its own 2026 review of the standard. Usage by the crawlers themselves is even more discouraging than raw adoption numbers alone suggest. The same Limy analysis, drawn from monitoring more than 500 million actual AI bot traffic events across a ninety day window, found that GPTBot, ClaudeBot, PerplexityBot, OAI SearchBot, and Google Extended overwhelmingly skip the llms.txt file entirely and crawl the regular HTML pages directly instead, with only 408 requests out of that entire half billion event sample ever specifically targeting the file.

Google has been unusually direct about this in public, which is worth taking seriously given how much weight the company’s statements on crawling and indexing normally carry. In July 2025, Google’s own Gary Illyes confirmed on the record that Google does not support llms.txt and has no plans to start, and fellow Googler John Mueller separately compared the entire concept to the long since discredited keywords meta tag, an SEO relic from the late 1990s that search engines stopped trusting decades ago once it became clear site owners would simply stuff it with whatever they wanted, according to reporting from Limy’s ongoing tracking of the standard’s real world reception. Separately, research cited by industry publication Search Engine Land found that roughly eight out of nine sites that implemented llms.txt saw no measurable change whatsoever in their AI referral traffic afterward, a figure noted in independent coverage by LinkBuildingHQ.

None of this means llms.txt is a scam or that publishing one is actively harmful, and it genuinely is not. It costs an hour of your time, it forces some useful clarity around your own information architecture as a byproduct, and the specific pattern of copying every single page into its own duplicate Markdown file, which some early implementers did enthusiastically, can create real duplicate content problems worth actively avoiding regardless of whether the file itself ever gets read by anything. The honest, current answer for a developer deciding where to spend a limited afternoon is simply that llms.txt sits closer to a low cost, low confidence bet on a future standard than a proven lever that moves AI visibility today, and treating it as a solved, must have checklist item, the way a surprising number of agencies currently market it, does not match what the actual crawler traffic data or Google’s own public statements currently show.

Protecting APIs and AI powered endpoints, not just pages

Everything covered so far addresses crawlers reading your public pages, but a growing and genuinely distinct category of risk sits specifically around API routes and AI powered features you have built yourself, particularly anything that calls out to an expensive third party model on the backend.

An unprotected AI endpoint, meaning a route that accepts user input and forwards it to a paid model API to generate a response, is a fundamentally different kind of target than a static page. A scraper hitting your blog wastes your bandwidth. A bot hitting an unprotected AI endpoint in a tight loop can generate a genuinely alarming bill within hours, since every single request potentially costs real money the moment it reaches the model provider, regardless of whether a real person ever asked for anything. This is precisely the gap products like Vercel’s BotID were built to close, letting a route specifically wrap its most expensive operations behind a bot check while leaving cheap, ordinary page loads completely untouched and free of any added friction. The same logic applies even without a dedicated product, and a simple rate limit tied to session or IP address on any route that calls a paid model API is one of the cheapest, highest value protections a developer can add in a single afternoon, well before reaching for anything more sophisticated.

It is worth being honest, in closing this section, that even the strongest combination of these techniques is not a permanent, complete fix. Agentic AI browsers, meaning tools where a real person directs an AI assistant to browse and act on their behalf inside a completely legitimate, often logged in browser session, do not need to fake a user agent, spoof a fingerprint, or rotate through a proxy network at all, because technically nothing about the underlying request is actually fake. That specific challenge, and the genuinely unresolved legal and technical questions it raises, is covered in far more depth elsewhere in this series, but it is worth keeping in the back of your mind here. Every technique in this guide meaningfully raises the cost and difficulty of scraping your site. None of them, alone or combined, promises to make it perfectly impossible.

A practical checklist before you ship anything

Pulling everything above into a short, usable sequence, a reasonable process for a developer actually implementing this today looks something like the following. Start by deciding, deliberately and in writing somewhere your team can reference later, which of the three tradeoffs from earlier in this guide actually matches your business: full training data protection, present day AI visibility, or total exclusion. Write your robots.txt to reflect that specific decision using the correct category of tokens rather than a generic copy pasted template, and revisit it on a recurring quarterly schedule, since new crawlers appear regularly enough that a file written once and forgotten drifts out of date within a year.

Turn on your infrastructure provider’s managed AI bot blocking feature if one is available to you, since Cloudflare and Vercel both maintain these lists so you personally do not have to. Add a lightweight user agent based rule at your own application or edge layer as a second line of defense for anything a managed feature might miss. If you run anything genuinely sensitive, whether that is a paid AI endpoint, an authentication flow, or a checkout page, add real behavioral bot detection specifically on those routes rather than relying on user agent matching alone, since user agent strings are trivially easy to fake and sensitive routes are exactly where that gap matters most. Treat llms.txt as an optional, low cost afterthought rather than a priority, given what the actual crawler traffic data currently shows about how rarely it gets read. And check your logs periodically, not just once during initial setup, because this entire landscape is genuinely still shifting underneath everyone building on it right now, and a configuration that made perfect sense six months ago can quietly stop matching reality without anyone noticing until traffic or billing looks strange again.

The bottom line

There is no single file, product, or setting that makes this problem go away permanently, and any guide that promises otherwise is selling something. What actually works is layering several imperfect defenses that each catch a different category of the problem: a robots.txt file that reflects a real, considered decision rather than a guess, verification for the requests that matter most, a managed blocking feature from your infrastructure provider so you are not maintaining a crawler list by hand, and genuine behavioral protection on the specific routes where a bot slipping through would actually cost you money or data. None of it is glamorous, and none of it is finished once you ship it. But taken together, it is a genuinely solid, well informed starting point, built on what is actually documented and actually measured right now, rather than on whatever configuration happened to rank highest the last time you searched for a quick answer.