How to Block Bad Bots Without Blocking Real Customers

Every team that decides to get serious about bot traffic makes the same mistake in the first month, and it is always the same mistake.

They turn something on.

A firewall rule. A managed bot ruleset. A challenge on the login page. An aggressive robots directive. Something that felt overdue, went live on a Tuesday, and produced an immediate satisfying drop in junk traffic. The dashboard looked better on Wednesday.

Then, somewhere between three days and three weeks later, a different number moves. Organic impressions slide. A payment provider’s callback starts failing. Conversion rate on mobile drops four points with no other explanation. Support gets a trickle of messages from people who cannot complete checkout, and the trickle is small enough to look like noise. Nobody connects it to the Tuesday change, because the Tuesday change was a security decision and this is a revenue problem, and those live in different meetings.

The uncomfortable truth about bot defence is that the failure mode is silent. A blocked attacker generates a log line. A blocked customer generates nothing. They do not file a ticket, they do not tweet, they do not email you. They go somewhere else and you never learn their name. Your enforcement looks like it is working precisely because the thing it broke does not announce itself.

This guide is about doing it properly. Not about whether to defend, which is settled, but about how to defend with a system that is auditable, reversible and calibrated to the actual cost of each kind of error. It covers the taxonomy of automated threats worth defending against, the enforcement actions available and when each is appropriate, how to verify a crawler instead of trusting it, why challenges have quietly stopped working, what the research says about the real cost of blanket blocking, the legal ground you are standing on, and a staged rollout that surfaces your mistakes before your customers do.

We build ClickBaton, which classifies and manages automated traffic, so we have an obvious interest here. What follows is written to be useful whether you buy anything or not, and every material claim is linked to a primary source.

One note on scope before we start. If your question is how much of my traffic is real and what are my true conversion metrics, that is a measurement problem and a different discipline. This article is about enforcement: what to do once you know. The two disciplines share a detection layer and diverge completely after it.

The First Principle: Blocking and Detecting Are Different Jobs

Almost every expensive error in this field comes from collapsing two questions into one.

Question one is a detection question. What is this request? Who or what generated it? What is it here to do? This question has a probabilistic answer, expressed as a confidence score, and it should be asked of every request that reaches you.

Question two is a policy question. Given what we believe about this request, what should we do to it? This question has a business answer, and it depends on what you sell, what the request is trying to reach, what it costs you to be wrong in each direction, and what your obligations are.

Teams that merge these end up with rules that read like “block anything suspicious,” which is not a policy, it is a mood. It cannot be tuned because it has no threshold. It cannot be audited because it produces no record of what it believed. And it cannot be reversed cleanly because nobody wrote down why it was doing what it was doing.

Separating them gives you something much more useful: a confidence score on one axis and a graduated set of responses on the other. Low confidence and a harmless target gets logged. High confidence and a payment endpoint gets stopped. Everything in between gets something proportionate. The threshold becomes a dial that a named person owns, rather than a binary that nobody wants to touch.

There is a second reason to keep them apart, and it matters more than it sounds. Most automated traffic should be measured and excluded from your analytics while remaining perfectly welcome to fetch the page. Googlebot belongs in that category. So does every uptime monitor, link preview service and accessibility scanner pointed at you. If your only tool is a block, you will inevitably reach for it on traffic that was never a threat, because you have no other way of expressing “this is not a customer.”

A Taxonomy Worth Borrowing

Before deciding what to stop, it helps to have precise names for what you are stopping. The industry already has them, and using shared vocabulary makes conversations with vendors and internal stakeholders dramatically shorter.

The OWASP Automated Threats to Web Applications project maintains an ontology of automated attacks, published as the Automated Threat Handbook and organised as a list of threat events comprising twenty one numbered threat events. It has been in circulation since 2015 and has become the de facto standard for classifying malicious automation. Its virtue is that it describes what the automation is trying to achieve, rather than what technology it uses, which means it stays accurate as the tooling changes.

The events that matter most to commercial sites, in rough order of how often we see them cause real damage:

Credential stuffing and credential cracking. Testing stolen username and password pairs at scale, or brute forcing weak ones. Aimed at your login endpoint and increasingly at the API behind it.

Account creation. Manufacturing accounts for later misuse: promotion abuse, review manipulation, marketplace fraud, or simply aging them for a future attack.

Carding and card cracking. Testing stolen payment credentials against your checkout to find which ones are live.

Scraping. Bulk collection of your content, pricing or inventory. Usually not illegal, frequently commercially damaging, occasionally so aggressive it becomes an availability problem.

Scalping and expediting. Acquiring limited inventory faster than a human can, whether that is concert tickets, sneakers, appointment slots or discounted stock.

Denial of inventory. Holding items in carts or reservations without completing purchase, making stock unavailable to real buyers.

Skewing. Inflating counts that matter: poll results, review scores, referral counts, advertising metrics.

Sniping. Acting at the last possible moment, classically on auctions.

Vulnerability scanning and footprinting. Mapping your application looking for weaknesses.

Denial of service. Exhausting a resource, which the OWASP handbook explicitly extends beyond network floods to include business logic denial of service, account lockout, resource depletion and denial of wallet.

That last category deserves a note, because the scale has changed. Cloudflare’s H1 2026 DDoS Threat Report recorded 23.2 million network layer attacks and 29.64 trillion malicious HTTP requests in the first half of 2026 alone, working out to roughly 5,343 network layer attacks per hour. The number of attacks exceeding one terabit per second grew from 130 in the first quarter to 805 in the second, a rise of 519 percent. Most attacks remain small and brief, with the great majority under 50 megabits per second and ending inside ten minutes, but the tail has got a great deal heavier.

Notice what this taxonomy does not contain. There is no OWASP threat event called “being a bot.” Every entry describes an abusive outcome. That framing is the correct one for policy, because it forces you to answer a useful question: which of these am I actually exposed to, and what is each one worth to prevent?

Most sites are exposed to three or four of them, not twenty one. Knowing which three saves an enormous amount of wasted defensive effort.

The Asymmetry That Should Govern Every Rule

Here is the single most important idea in this article, and the one most likely to change how your team writes rules.

There are four possible outcomes when you make a decision about a request. Two are correct. Two are errors. The two errors do not cost the same, and they do not have the same visibility.

A false allow lets a bot through. The cost is one bot’s worth of bandwidth, one bot’s worth of polluted analytics, and whatever that specific bot came to do. It is a real cost. It is also bounded, recoverable and, crucially, visible: the bot leaves traces in your logs that you can find later and learn from.

A false block turns away a person who wanted to give you money. The cost is that customer, their lifetime value, and quite possibly their opinion of your brand. It is unbounded, unrecoverable, and almost entirely invisible, because people who cannot use your site overwhelmingly do not tell you. They assume it is broken, they assume it is their connection, they leave.

That asymmetry has a direct operational consequence: you cannot manage your false block rate by waiting for complaints. Complaints are a lagging indicator with a catastrophic sampling bias. If you are getting complaints about being blocked, you are far past the point where you should have noticed.

Who ends up in the false block bucket? Not a random sample of your audience. A specific and predictable set of people:

Users on privacy focused browsers and hardened configurations. Anti fingerprinting features make a browser look unusual by design, which is exactly what many detection systems score as suspicious.

Users behind corporate proxies and VPNs. Shared egress addresses concentrate traffic, which trips rate limits and reputation checks. This population skews toward employed professionals at large companies, which for a B2B business is precisely the audience you most want.

Users of assistive technology. Screen readers, switch access devices and voice control produce interaction patterns that behavioural models frequently score as non human, because they are not the patterns the model was trained on.

Users on older hardware, slow connections or unusual locales. Failed challenge timeouts, missing browser features and geography that does not match your typical traffic all push scores down.

Users in markets your model has seen little of. Any detection system is calibrated against the population it has observed. If you expand into a new region, your false block rate there will be higher on day one and nobody will tell you.

Read that list again and notice something: it correlates with disability, with corporate employment, with lower income, and with emerging markets. A poorly calibrated bot defence is not a neutral tax on your traffic. It falls hardest on specific groups, which turns a technical decision into a legal and ethical one. We return to that shortly.

The practical response is to instrument the error you cannot see. Concretely:

Track challenge pass rates by segment. Country, device class, browser, network type. A segment passing at 60 percent when your baseline is 96 percent is a defect, not a threat signal.

Track abandonment at every enforcement point. If your checkout challenge has a 12 percent abandonment rate, you now know its cost in the same units as its benefit.

Sample your blocks and review them by hand. A weekly review of fifty randomly selected blocked sessions, read by a person, catches things no dashboard will. This is dull and it is the highest value hour in the whole programme.

Give people a way out. Every enforcement point should have a documented path for a wrongly blocked human to reach a person. Almost nobody will use it. The few who do are worth ten dashboards.

The Case Against Blanket Blocking, With Evidence

There is a widespread intuition that blocking automated traffic is close to free: you lose bots, you keep humans, the only cost is engineering time. There is now empirical evidence that this intuition is wrong, and it comes from a natural experiment run at scale by an entire industry.

When large language models arrived, news publishers responded by blocking AI crawlers in their robots files, and they did so far more aggressively than other sectors. That created a clean setting for measurement: many similar sites, staggered adoption dates, and observable traffic.

Researchers Hangcheng Zhao of Rutgers Business School and Ron Berman of The Wharton School studied it. Their working paper, Strategic Response of News Publishers to Generative AI, combines traffic data, browsing panel records, historical robots files, job postings and page content across the top 500 news publishers.

The finding that matters here is blunt. Using a difference in differences design, they estimate that after blocking generative AI crawlers, large publishers experienced roughly a 23 percent decline in total traffic and a 13.9 percent decline in panel measured traffic relative to before blocking.

The second number is the one to sit with. Panel data measures household browsing, which means it is counting people. Human traffic fell too. This was not the mechanical removal of bot visits from a bot inclusive metric. It was a loss of actual readers.

The mechanism the authors propose is reduced brand exposure: if a publisher blocks a crawler, its material is less likely to surface in AI generated summaries and answer tools, so fewer people encounter the brand at all and fewer arrive by any route.

Two honest caveats, because this finding gets over cited in both directions.

The estimate moved between paper versions. A later revision using a weekly window over six weeks put the decline at roughly 7 percent rather than 23 percent monthly. Different windows, updated methodology, same direction. Cite the direction confidently and the magnitude carefully.

It is a working paper about news publishers. News is a category where discovery is unusually dependent on third party surfaces. A business selling industrial pumps on long sales cycles should not assume the same elasticity. The authors also found that effects varied by publisher size, with some mid sized publishers seeing increases rather than declines.

With those caveats, the strategic lesson stands and generalises: access and visibility are the same commodity. Anything that reaches your content is a potential distribution channel, and cutting it off has a cost on the demand side that will not show up in your security metrics. That does not mean allow everything. It means the blocking decision is a commercial decision wearing a technical costume, and it belongs in front of the people who own revenue.

Why Challenges Stopped Working

For twenty years the default answer to automation was to ask the visitor to prove they were human. That answer has quietly inverted, and a large number of sites are still running the old playbook.

The research picture is unambiguous. A study out of ETH Zurich examining Google’s image challenge system reported that the authors could solve 100 percent of the captchas, against 68 to 71 percent in previous work, using object detection models. More pointedly, they found no significant difference in the number of challenges a human and a bot had to solve in order to pass. A separate paper describing a low resource attack on a competing service reported an automated solver accuracy of 95.93 percent, concluding the reverse Turing test was broken. Earlier work reached similar conclusions about object detection based solvers, describing the scheme as essentially broken because its tests are easily solvable.

Now hold that against the human side of the ledger. Analysis compiled by the United Nations University campus computing centre notes that give up rates for audio challenges reach 50 percent, and solving one takes an average of 28.4 seconds against 9.8 seconds for a visual challenge. The same piece observes that roughly 16 percent of the world’s population experiences significant disability, for whom these challenges range from difficult to impassable.

Put the two together and you get the defining fact of this section. A challenge that machines pass in under a second and that half of your audio dependent users abandon entirely is not a security control. It is a filter with the polarity reversed.

There is a compliance dimension too, and it has sharpened recently. Challenges sit awkwardly under accessibility guidelines: they are non text content that cannot have a meaningful text alternative without defeating their own purpose. The World Wide Web Consortium has flagged the tension for over a decade. With the European Accessibility Act now in force and web accessibility litigation continuing at volume, a challenge that blocks screen reader users is a live risk rather than a theoretical one.

None of this means friction is useless. It means the shape of useful friction has changed.

Invisible verification beats interactive challenge. Systems that run background checks and only escalate when signals are poor keep the cost off the great majority of legitimate users. Cloudflare’s Turnstile is the widely deployed example of the pattern: it runs a client side challenge in the background, offers managed, non interactive and fully invisible modes, and states WCAG 2.2 AA compliance. It also works without routing your traffic through a particular network, which matters for adoption.

Proof of work shifts the cost to the right party. Rather than testing perception, you impose a small computational cost that is negligible for one page view and material at scraping scale. The open source Anubis project, released in January 2025 and now maintained by Techaro, does exactly this with a SHA256 challenge, and has been adopted by a striking list of infrastructure projects including GNOME’s GitLab, the Linux kernel mailing list archives, FFmpeg, Wine, FreeCAD and UNESCO.

Anubis is also an instructive lesson in limits, and its own community says so. Codeberg reported in 2025 that many scraper operators had learned to solve the challenges, and the project’s documentation is candid that deploying it will likely result in your site not being indexed by some search engines, which it describes as a feature rather than a bug. Read that sentence as a design constraint, not a slogan. It is the correct tool for a code forge under a scraping siege. It is the wrong tool for a commercial site that needs to be found.

Reserve interactive challenges for a narrow band. Medium confidence, high value action, and nothing else. A challenge on your checkout for everybody is a tax on revenue. A challenge on a password reset attempt from an unrecognised device on a datacentre address is proportionate.

And measure the challenge itself. Pass rate, abandonment rate, time to complete, all segmented. If you cannot state your challenge abandonment rate for mobile users in your third largest market, you do not know what your challenge costs.

Verifying a Crawler Instead of Trusting It

A large share of enforcement mistakes trace back to one root cause: somebody wrote a rule that matched on a user agent string.

A user agent is free text. Any client can send any value. Rules built on it fail in both directions at once. They let attackers in, because impersonating a trusted crawler is trivial. And they keep customers out, because legitimate tools and services also send unusual strings.

The scale of impersonation is no longer marginal. DataDome found that known agents are actively used as cover, with one major operator’s agent impersonated across 16.4 million spoofed requests in a two month window, and another operator’s crawler showing an impersonation rate near 2.4 percent of requests. Separately, industry analysis has noted that fake search crawlers became a favoured vehicle in denial of service attacks precisely because defenders hesitate to block anything claiming to be a search engine.

The fix is verification, and the major operators have made it possible. Google’s documentation on verifying requests from Google crawlers and fetchers lays out two methods, and you should implement both.

Forward confirmed reverse DNS. Take the source address from your logs. Run a reverse lookup. Confirm the hostname sits under an expected domain. Then run a forward lookup on that hostname and confirm it resolves back to the original address. Both directions must agree. A reverse record alone proves nothing, because a reverse record can be set by whoever controls the address block.

Published range matching. Google publishes its crawler ranges as machine readable files, split across several categories covering common crawlers, special case crawlers, user triggered fetchers and agents. Fetch them on a schedule and match the source address against the correct file. Microsoft, and several AI operators, publish equivalents.

Signed requests, where available. The emerging standard is Web Bot Auth, built on HTTP Message Signatures. The bot signs each request with a key and publishes its public keys in a discoverable directory; you verify the signature. Cloudflare folded message signatures into its Verified Bots programme, and AWS WAF added Web Bot Auth support in November 2025, automatically allowing verified agent traffic by default. This is qualitatively different from everything else on the list, because a signature is checkable while a string is a claim.

Four implementation notes from doing this in production.

Cache verification results, but not forever. Reverse and forward lookups per request will hurt. Cache the verdict against the address for hours, not days, and audit the cache against the published ranges periodically, because crawler infrastructure moves.

Handle the unverifiable category explicitly. Not every operator supports verification. Some publish ranges and no reverse DNS, some publish nothing at all. A claim you cannot check is not a claim you should honour, but it is also not automatically an attack. It belongs in its own bucket with its own policy, usually rate limited and excluded from reporting rather than blocked.

Failing verification is not the same as being hostile. This is the single most useful mental adjustment. When a request claiming to be a search crawler fails your checks, the correct action is to stop treating it as a search crawler. It reverts to being an ordinary unknown request and is judged on everything else you know about it.

Watch for the reverse failure. Managed rulesets that detect fake crawlers produce their own false positives. Cloudflare documents several: cloud platform services sending requests with related user agents from outside standard ranges, site scanning tools that do not share their crawler’s address ranges, and third party monitors that set bot like user agents. AWS documents a parallel set, noting that mobile applications typically have non browser user agents which its rules block by default. If you run a mobile app against your own API, that default will eventually bite you.

The Blast Radius of Blocking a Search Crawler

This deserves its own short section because it is the highest consequence single mistake available to you, and it is common.

Blocking a search crawler does not produce an immediate visible failure. It produces a gradual de indexing that begins days later and takes far longer to reverse than it took to cause. Pages drop out. Rankings decay. Recovery waits on recrawl schedules you do not control. By the time it is obvious in your traffic reports, weeks have passed and the cause is buried in a firewall change nobody logged as a marketing event.

The ways it happens are mundane:

A managed ruleset gets tightened and starts matching crawler user agents along with the bad traffic it was aimed at.

A rate limit is set globally rather than per class, and a crawler that requests thousands of pages an hour trips it exactly as designed.

A robots edit meant for one bot catches several with similar names.

A migration goes live and nobody checks whether the new configuration still permits what the old one did.

An aggressive bot mode is enabled on a plan where it applies to everything, including verified crawlers.

A tarpit or challenge is applied broadly, and search crawlers either get trapped in synthetic content or fail the challenge silently.

That last one carries a second order risk that is easy to miss. If your deception layer generates synthetic pages and a search crawler indexes them, you have published thin machine generated content under your own domain. The quality signal problem that creates is entirely self inflicted.

The controls that prevent all six are unglamorous:

Allow verified crawlers explicitly and first, using verification rather than user agent matching, and place that rule above your general bot rules in evaluation order.

Always allow the robots file and sitemaps, unconditionally, from anywhere.

Never rate limit by a global counter. Rate limits belong per class and per endpoint.

Monitor crawler access continuously. Not manually, on a schedule. You want an alert the day a crawler that was allowed yesterday starts receiving errors, not a discovery three weeks later. Server log analysis is the ground truth here, since it shows what actually happened at your origin rather than what your configuration says should happen.

Treat any change to bot rules as a change with search consequences. That means it goes through whatever review process your other search affecting changes go through, and it gets a note in the same changelog your marketing team reads.

The Enforcement Ladder in Detail

With detection separated from policy, and the cost asymmetry understood, here is the actual set of actions available, from lightest to heaviest.

Observe

Log the request, record the classification and the confidence score, and do nothing else. The visitor is unaffected.

This is the permanent baseline. Every request gets observed, forever, regardless of what else happens to it. Observation is what makes every other rung tunable, because it gives you the counterfactual: what would have happened if we had enforced.

Any new rule starts life here and stays for at least two weeks. No exceptions. A rule you have not watched in observe mode is a rule whose false positive rate you are about to discover in production.

Tag and exclude

Classify the request, exclude it from your analytics and from the conversion events you transmit to advertising platforms, and otherwise serve it normally.

This rung is where most automated traffic on a healthy site should end up, and it is chronically underused because it does not feel like doing anything. It is, in fact, doing the most valuable thing: making your numbers true without risking a single customer. It has a zero false positive cost by construction, because nothing about the visitor’s experience changes.

Throttle

Slow the response, cap the request rate, or queue.

Rate limiting is the most underrated control in this entire discipline because it is proportionate by nature. It does not deny access, it denies scale, and scale is what makes most automated abuse economical. A scraper that can fetch three pages a second is a business. One limited to three a minute usually is not.

The algorithm choice matters less than people think, but it does matter. The main families are fixed window, sliding window log, sliding window counter, token bucket and leaky bucket, and they trade memory, burst tolerance and accuracy differently. Fixed windows are simplest and allow a burst at the boundary. Token bucket absorbs bursts deliberately, which suits developer APIs where bursts are a feature. Leaky bucket enforces a constant output rate, which suits protecting a fragile downstream service.

The design decisions that actually determine whether rate limiting helps or hurts:

Limit per class, not globally. A verified search crawler, a logged in customer, an anonymous visitor and an API client have different legitimate ceilings. One number for all of them will be wrong for all of them.

Limit per endpoint. Your search endpoint, your login, your pricing API and your blog have wildly different abuse profiles and cost profiles. Sensitive endpoints need much lower thresholds: login attempts per minute should not share a budget with page views.

Return the right status and header. A 429 with a Retry After header tells a well behaved client exactly what to do. Silently dropping requests teaches good clients nothing and bad clients nothing.

Add jitter to retry guidance. If every limited client retries at exactly the advertised interval, they synchronise and hit you together.

Degrade before you deny. For read endpoints under pressure, serving a cached or simplified response is better than an error for everyone involved.

Challenge

Require the client to do something before proceeding, ideally invisibly.

Per the previous section, this rung has narrowed considerably. Use it at medium confidence on high value actions. Prefer invisible verification. Instrument abandonment. Never make it the first line of defence, and never apply it uniformly to all traffic.

Tarpit and deceive

Serve the request, but serve something worthless, and make the worthlessness expensive to discover.

This is the most interesting recent development in enforcement, and its logic is genuinely different from everything above. A block tells an attacker they have been detected, which is useful information they will act on. Deception does not.

Cloudflare’s AI Labyrinth is the productised version: on detecting unauthorised crawling, rather than blocking, it links to a series of generated pages convincing enough to entice a crawler to traverse them, wasting its time and compute while the real content stays untouched. The company describes a secondary benefit that is arguably the primary one: no real human goes four links deep into a maze of generated nonsense, so anything that does is almost certainly a bot, which turns the trap into a high confidence detection signal. Cloudflare’s documentation notes the links use nofollow attributes and are intended not to affect search performance, and that crawlers respecting no crawl instructions will simply ignore the honeypot.

Deception carries risks worth naming plainly. Your decoy content must never be indexable, or you have polluted your own domain. Your detection must be very good before you deploy it, because a customer trapped in a maze is a worse outcome than a customer who saw an error page. And archival and research crawlers get caught in these, which is a real cost to the commons that reasonable people weigh differently.

Block

Refuse the request.

Reserve it for very high confidence and genuinely hostile intent: credential stuffing in progress, carding, active exploitation, attack traffic. Blocking should be the rarest action your system takes, and every blocking rule should have a named owner and a documented rollback.

One Policy Per Traffic Class

The single structural change that improves most bot programmes is replacing one global posture with a policy per class of traffic.

Search crawlers. Allow, verify, exclude from reporting. Never rate limit aggressively, never challenge. This traffic is your distribution.

AI search and retrieval bots. Allow, verify, rate limit generously, exclude from reporting. These can produce citations and correlate with live human interest. Blocking them is the decision the publisher research above should make you think twice about.

AI training crawlers. This is the genuinely contested row, and it is a licensing decision rather than a security one. It belongs with legal and executive, not with the security team alone. Whatever you decide, rate limit and exclude from reporting.

Agentic browsers. Allow, verify where signatures exist, and treat with care: some of these sessions are a customer buying something through an intermediary. Challenge only on sensitive actions and prefer cryptographic identity over behavioural heuristics, because behaviour cannot distinguish an agent shopping for a real person from an agent committing fraud, while a signature and a mandate can.

Monitoring, preview and accessibility tools. Allow, exclude from reporting, throttle only if volume is genuinely disruptive. Blocking a link preview service means your links look broken when people share them, which is a marketing loss disguised as a security win.

Undeclared scrapers. Rate limit, exclude, challenge on sensitive paths, tarpit if extraction is aggressive, block only with strong evidence.

Credential and payment abuse. The one row where blocking is the default. These are attacks, they are identifiable, and the cost of allowing them dwarfs the cost of the occasional false positive. Even here, prefer step up authentication over a hard block where the action is a login rather than an exploit.

Two rows in that grid will be contested inside your organisation, and that is the point. A policy matrix makes disagreement explicit and resolvable, where a single global setting just makes it invisible.

The Legal Ground You Are Standing On

Enforcement decisions sit on top of a legal landscape that many teams are surprised by, and the surprise usually runs in the direction of assuming more protection than exists.

Scraping public data is largely lawful in the United States, and the courts have said so repeatedly. The line of cases begins with hiQ Labs against LinkedIn, where the Ninth Circuit affirmed that the Computer Fraud and Abuse Act does not prohibit scraping publicly available data, reasoning that publicly accessible computers are not accessed without authorisation. In January 2024 the same judge who presided over hiQ granted summary judgment for Bright Data against Meta, in a ruling widely read as reinforcing the distinction between logged out public data and data behind authentication.

The practical reading for a site owner:

Terms of service are weaker than you think against a party with no account. The Bright Data ruling turned substantially on whether the defendant was bound by terms while logged out, and on the effect of terminating its accounts.

Authentication is the meaningful legal boundary. Data behind a login is in a materially different position from data on a public page. This is a strong argument for putting genuinely sensitive data behind authentication rather than relying on obscurity plus terms.

Creating fake accounts crosses a line that scraping does not. The hiQ record treated the hiring of contractors to create fake profiles for logged in collection as a distinct and problematic category.

Your technical defences are your primary remedy. If the law will not stop bulk collection of your public pages, rate limits, verification and deception are not a supplement to a legal strategy, they are the strategy.

Elsewhere, the picture differs. Some abuse categories have dedicated statutes. In the United States, the Better Online Ticket Sales Act prohibits circumventing purchase limits and security measures on ticket sellers, and enforcement intensified after a March 2025 executive order directed the Federal Trade Commission to prioritise it. The agency has since brought several actions, and there is a detail in that regime worth generalising well beyond ticketing: the Act targets circumvention of the limits you actually set. If you never post purchase limits, there is nothing to circumvent. Your own published rules are what make enforcement possible.

Two more legal considerations that bear directly on how you enforce:

Accessibility law applies to your bot defences. Web accessibility litigation runs at high volume in the United States, and the European Accessibility Act came into force in June 2025 with WCAG 2.2 as the operative standard. A challenge that excludes screen reader users is exposure. This is not a reason to remove protection; it is a reason to choose invisible verification over interactive puzzles.

Fingerprinting and behavioural signals sit in a contested privacy area. The analysis varies by jurisdiction and by purpose, and security and fraud prevention purposes generally stand on firmer ground than marketing analytics. Two practical mitigations: document the purpose properly, and lean on signals that carry no personal data at all, such as network ownership, transport fingerprints, header ordering and request patterns. We are not lawyers and this is not legal advice. Have counsel review anything you deploy.

Defending the Login: The One Place Blocking Earns Its Keep

If there is a single endpoint where aggressive enforcement is unambiguously correct, it is authentication. The economics there are inverted from everywhere else on your site: the attack volume is enormous, the legitimate volume is predictable, and the cost of a successful attack is severe.

The volumes are genuinely difficult to internalise. Verizon’s analysis of single sign on provider logs found that credential stuffing accounted for a median 19 percent of all authentication attempts, which is to say roughly one in five login attempts on a typical service is somebody trying stolen credentials. The same research found that only 49 percent of a typical user’s passwords were distinct from one another, which is the entire reason the technique works.

Merchant network data goes further. Forter’s benchmarks reported that across its network, login traffic ran 31 percent legitimate and 69 percent bot or malicious in the second half of 2024. Imperva’s research places financial services as the most targeted sector for account takeover, and notes that API directed attacks made up 44 percent of advanced bot traffic, which is the detail most login defences miss: the attack has usually moved to the API behind your form, not the form itself.

The defensive stack that works, roughly in order of impact per unit of effort:

Remove the reusable secret. Phishing resistant passkeys leave no shared credential to steal or replay, which removes the mechanism credential stuffing depends on entirely. Everything else on this list mitigates. This one eliminates.

Check credentials against known breach corpora at set time and at login. If the password a user is choosing is already in circulation, stuffing will find it eventually. Refusing it costs one moment of friction and removes a permanent liability.

Rate limit per account, per address, per fingerprint and globally, with different thresholds for each. Attackers spread across all four dimensions, so limiting on one just shapes their traffic.

Step up rather than block. For a login attempt that looks wrong but might be a real person on a new device in a new country, an additional verification factor is proportionate and a hard block is not. Blocking a real customer out of their own account is a support call, a bad review, and sometimes a lost customer.

Never leak which factor failed. Distinct responses for wrong username and wrong password turn your login into an account enumeration oracle.

Protect the recovery path with the same rigour as the login path. Account recovery is frequently the softest surface in the whole system, and attackers know it.

Watch the API, not just the form. Your mobile application, your single page app and your partner integrations all authenticate through endpoints that may have none of the controls your web form has.

Invalidate sessions on credential change and make session tokens hard to replay. Increasingly the attack is not stealing the password, it is stealing the session.

The reason this section exists in an article about not blocking customers is that it is the exception that proves the rule. Blocking is right here because the confidence is high, the evidence is strong, the endpoint is narrow, and the alternative is catastrophic. Those four conditions rarely co occur elsewhere on your site, which is exactly why blocking should be rare elsewhere on your site.

Business Logic Abuse: When the Bot Follows Every Rule

There is a class of automation that passes every technical check you can devise, because it does nothing technically wrong. It requests pages your site offers, at rates your site permits, using a browser your site supports, and it destroys your business anyway.

This is business logic abuse, and it is the hardest category to defend because the defence has to understand what your application is for.

Scalping and expediting. Acquiring limited inventory faster than a person can. Tickets, sneakers, consoles, appointment slots, discounted stock, domain names. Every request is legitimate in isolation.

Denial of inventory. Holding stock in carts or reservations without ever purchasing, so real buyers see sold out. Costs the attacker nothing and costs you the sale twice.

Coupon and promotion abuse. Enumerating discount codes, or creating accounts in bulk to harvest first order offers. The account creation is real, the promotion redemption is real, the customer is not.

Review and rating manipulation. Skewing your social proof in either direction, on your own site or a marketplace.

Price and inventory scraping for competitive repricing. Not abuse in a legal sense at all, and frequently the largest single category by volume in retail and travel.

Loyalty point theft. Draining accumulated value from compromised accounts, often more attractive to attackers than payment fraud because it is less monitored.

Technical signals help but rarely settle these cases. What settles them is application aware limits expressed in business units rather than request units.

The pattern to adopt: for each abusable operation, define a rate that a real customer would never exceed, expressed in the language of the operation. Not requests per second. Carts per hour per account. Reservations per address per day. Promotion redemptions per payment instrument. Password resets per account per week. Search queries per session before a purchase becomes implausible.

Then add three things:

Make holds expensive. Cart reservations should expire fast and require progressively more commitment to extend. If holding inventory is free, it will be held.

Publish the limits. As the ticketing statute illustrates, an explicit limit is what makes circumvention a nameable act. It also tells honest high volume customers where the line is before they trip it.

Verify at the point of value, not the point of entry. A person who has completed a payment with a verified instrument has done something a script finds expensive. Push your strongest checks to the moment of value transfer rather than the front door.

There is a strategic point underneath all of this. You cannot buy a product that understands your business logic. Vendors provide excellent detection of automation. Whether five carts per hour is normal depends on whether you sell trainers or industrial fasteners, and only you know that. This is the part of bot defence that is irreducibly yours.

Architecture: Where to Enforce

Enforcement can happen at four places in the request path, and the choice determines what evidence you have and what you can do about it.

At the edge, before your origin. Your content delivery network or edge platform. Cheapest per request, protects origin capacity, sees network and transport signals clearly, and stops attacks before they cost you anything. It knows least about your application: it cannot tell a cart hold from a page view.

At the reverse proxy or gateway. Your own infrastructure, still ahead of application code. Sees full request context, can enforce rate limits with your own state, and can be reasoned about without a vendor dashboard. Costs you the connection.

In application middleware. Knows who the user is, what the operation means, and what the business rules are. Everything above has already been spent by the time it runs.

At the data or transaction layer. Last line. Knows the true state, catches things nothing upstream could see, and is far too late to be your only control.

The pattern that works is cheap checks early, expensive checks late, and one shared verdict across all four. Network and transport verification at the edge, application aware limits in middleware, value transfer checks at the transaction layer, and a single classification with a single session identifier flowing through all of them.

That last requirement is the one most implementations miss. If your edge platform blocks something and your application never learns about it, you cannot reconcile your numbers, you cannot investigate a complaint, and you cannot tell whether a drop in conversions came from a customer problem or an enforcement change.

Some practical constraints worth designing around:

Never put a synchronous third party verification call in the critical path of an ordinary page load. Verify asynchronously and score after the fact for everything except your highest value actions.

Fail open on infrastructure, fail closed on money. If your detection service is unreachable, an ordinary page request should be served. A payment authorisation should not.

Keep enforcement configuration in version control. Rules that live only in a vendor dashboard have no history, no review and no rollback story. A rule change that nobody can diff is a rule change nobody can undo.

Log the verdict with the request, always. Classification, confidence, contributing signals, action taken, rule version. If you cannot replay a decision six months later, you cannot defend it when somebody senior asks why traffic dropped.

How ClickBaton Handles This

Disclosed interest again: we make a product in this space. The design choices below are the ones we would argue for regardless of who builds it.

We separate the verdict from the action, structurally. Classification produces a score and a set of contributing signals. Policy consumes that score and decides what happens. They are different systems with different owners, which means you can change your appetite for risk without touching detection, and improve detection without accidentally changing enforcement.

We verify crawlers with a ladder rather than a list. A request claiming a known identity passes through registry lookup, published range matching, forward confirmed reverse DNS, and signature verification where the operator supports it. Failing a rung demotes the claim rather than triggering a block. Given the volume of agent impersonation now in circulation, this ladder has become the single highest value component we ship.

We classify agents by function and intent rather than by brand. A training crawler, a search indexer, a live retrieval fetcher and an agentic browser are four different rows in your policy, not one row labelled AI. That distinction is what makes a defensible per class policy possible at all.

We ship graduated actions, with tarpitting configurable rather than default. Deception is powerful and it is easy to misuse. It sits behind explicit configuration, above a confidence threshold you set, and never applies to verified crawlers.

We keep every excluded request with its label and evidence. You can see what was acted on, how much, from where, and why, and you can reprocess a historical period when your rules improve. An invisible filter is an assumption. A visible, auditable, reprocessable one is a measurement.

We assume you will check our work. Every score decomposes into the signals that produced it. If we cannot show why, we do not expect to be believed, and neither should you believe any other vendor who cannot.

You do not need us for this. Cloudflare, Akamai, DataDome, HUMAN, Fastly and others solve overlapping parts of the problem, and a competent team with edge logs, published crawler ranges and a weekend can build a serviceable verification layer alone. What we would argue against is doing nothing because a complete solution looks expensive. Forward confirmed reverse DNS on your top declared crawlers is an afternoon of work and it will tell you something surprising.

Choosing a Vendor Without Being Sold To

If you do buy, the category has matured enough that the evaluation criteria have shifted. Forrester renamed its market evaluation from bot management to bot and agent trust management for its Q2 2026 edition, on the reasoning that the market has moved beyond blocking bots to governing all automated traffic including AI agents and crawlers. That renaming is a useful signal about where the discipline is going, whatever you make of any particular vendor’s placement.

Six questions we would ask, in this order:

What fraction of requests do you reach a verdict on? Coverage before accuracy, always. A product that examines 40 percent of your traffic and clears all of it is telling you it looked at 40 percent.

What is your false positive rate on legitimate human traffic, and how was it measured? If the answer is a number with no methodology, it is marketing. Ask specifically how they measure an error nobody reports.

Can I see the signals behind an individual verdict? Explainability is not a nice to have. When somebody complains they cannot check out, you need to reconstruct why in minutes, not open a support ticket.

Can I run in observe mode indefinitely, and can I reprocess history? If a product cannot tell you what it would have done, you cannot tune it safely.

What happens when your service is unreachable? Fail open or fail closed, and can I choose per endpoint?

How do you handle verified crawlers and signed agents? If the answer involves user agent matching anywhere, keep looking.

And one thing to be sceptical about across the whole category, ours included: detection effectiveness claims are unfalsifiable from the outside. Nobody can show you the attacks they missed. What you can verify is coverage, explainability, false positive instrumentation and the ability to reverse a decision. Weight those.

A Staged Rollout That Surfaces Mistakes Early

Here is the sequence we would follow, structured so that every phase produces evidence before the next phase adds risk.

Weeks one and two: observe

Instrument classification with no enforcement whatsoever. Every request gets a verdict and a score, written to a store you can query. Nothing changes for any visitor.

Establish your baseline. Total requests, requests by class, verified crawler share, unverifiable claims, share of traffic your system reaches a verdict on at all.

Run forward confirmed reverse DNS across your declared crawlers and count how many claims fail. This number usually changes the conversation.

Map your existing enforcement. Every rule currently live at your edge, in your firewall, in your robots file and in your application. Most teams discover rules nobody remembers adding. Write them down before you add more.

Exit gate: you can state your traffic composition and your verification coverage from data.

Weeks three and four: shadow

Write your rules and run them in log only mode. For every rule, record what it would have done and to whom.

Count the would be false positives by hand. Take the population each rule would have acted on, sample it, and read it. Look specifically for the categories from earlier: privacy browsers, corporate networks, assistive technology, unfamiliar markets.

Segment the shadow results. A rule with an acceptable aggregate false positive rate can still be catastrophic in one country or on one device class.

Exit gate: you have a measured false positive estimate per rule, not an assumed one.

Week five: narrow enforcement

Enforce one rule, on one path, against one traffic class. Preferably a high confidence, high harm case such as credential stuffing on the login endpoint.

Choose a light action first. Rate limit before challenge, challenge before block.

Watch four numbers daily: conversion rate on the affected path, support contact volume, crawler access status, and the rule’s own action count.

Exit gate: a week of enforcement with no unexplained movement in the four numbers.

Weeks six to eight: expand

Add one traffic class per week. Never two. When something breaks you want an unambiguous cause.

Escalate action level separately from coverage. Do not simultaneously add a new class and move from throttle to block.

Keep the observe layer running underneath everything. It stays on permanently.

Exit gate: your policy matrix is implemented, each row was enforced separately, and each has a rollback.

Ongoing

Weekly: false positive review by hand, challenge pass rate by segment, crawler access verification, action counts by rule.

Monthly: policy matrix review, rule version audit, review of anything sitting in the unverifiable bucket.

Quarterly: rescore a historical period with current rules and compare, re run your verification coverage measurement, and prune rules that have not fired.

The gate that matters more than any of the others: a named person can roll back any enforcement change in under fifteen minutes, and that person has actually practised doing it. A rollback path that has never been exercised is a hypothesis.

Robots Files, Content Signals and the Limits of Politeness

Every discussion of crawler policy eventually arrives at the robots file, and it is worth being precise about what that file is and is not.

A robots file is a request, not a control. It is a published statement of preference that well behaved clients honour voluntarily. It has no enforcement mechanism. A crawler that ignores it faces no technical consequence.

This produces a failure mode we see constantly: a site has a carefully written robots file expressing one policy, and a firewall or edge configuration enforcing a completely different one. When those disagree, the firewall wins and your robots file becomes a document describing a policy that is not in effect. Auditing that gap is a five minute job that almost nobody does.

That said, the file has grown a genuinely useful new capability. In September 2025, Cloudflare published the Content Signals Policy, an extension that lets a site declare not just what may be crawled but what may be done with the content afterwards. It defines three signals with deliberately narrow meanings:

search covers building a search index and returning links and short excerpts, and explicitly excludes AI generated summaries.

an AI input signal covers feeding content into a model at answer time, such as retrieval augmented generation.

an AI training signal covers training or fine tuning models.

Each takes a value of yes or no, expressed as a single comma delimited directive line, and absence means no expressed preference rather than permission or refusal. The exact token names are published in the specification. Cloudflare automatically applied the policy across more than 3.8 million domains using its managed robots service, with defaults permitting search and refusing training, while leaving the answer time signal neutral rather than guessing at customer intent.

Why bother with a mechanism that can be ignored? Three reasons, and they are better than they first appear.

It separates access from use, which robots directives never could. Previously your only lever was whether a crawler could fetch the page. Now you can permit fetching for one purpose and object to another, which matches how the disagreement actually works.

It may carry legal weight. Cloudflare’s own framing notes that operator preferences expressed in robots files can have legal significance, while being clear that courts and regulators may conclude otherwise. An explicit, machine readable, publicly published statement of terms is a materially stronger position than silence, whatever the eventual doctrine.

It is the cheapest thing on this entire list. It is a text file.

The correct posture is to treat signals as declaration and your edge as enforcement, and to make sure they agree. Declare your preference in the file, enforce the part you care about at the perimeter, and audit the two against each other on a schedule. A declaration nobody enforces is a wish. Enforcement nobody declared is a surprise.

When Access Becomes Priced

A shift is under way that changes the framing of this entire discussion, and it deserves attention even if you decide to ignore it for now.

Cloudflare announced that from September 15, 2026, its default settings will block mixed use crawlers from any page that hosts advertising, applying to new customers, new sites added by existing customers, and all existing free accounts. The reasoning is that an advertisement signals the site owner intended a human to arrive, which makes scraping that page for training or answer generation a different transaction from search indexing. Alongside it, the earlier Pay Per Crawl programme became Pay Per Use, compensating publishers when content is actually used in an answer rather than merely fetched.

Set aside whether this specific policy is good for the open web, which reasonable people dispute. The operational consequence is what matters: if access becomes priced, access must be metered, and metering is a measurement problem before it is an enforcement problem.

Three practical implications:

Defaults chosen by your infrastructure provider are still your policy. If your site was onboarded after a default changed, that default was selected for you. The only way to know what is actually in effect is to test it, not to read your own configuration.

Attribution by operator becomes a commercial requirement. You cannot participate in any compensation arrangement, or negotiate one, without being able to say what each operator took and what it returned.

Enforcement granularity becomes a revenue lever. The ability to allow one purpose and price another is only available to sites that can distinguish purposes at the request level.

Objections and Honest Limits

An article that only argues one side is advocacy. Here are the strongest counters and what we actually think.

“You are arguing for doing nothing while attackers eat us.”

No. We are arguing for graduated enforcement with instrumented error rates. Every rung above observe is on the table, including blocking, and on your login endpoint blocking should be aggressive. The argument is about defaults and sequencing, not about whether to defend.

“Observe mode for two weeks is too slow when we are under attack.”

Correct, and active incidents are the explicit exception. When credential stuffing is running right now, you enforce now and instrument afterwards. The two week discipline is for permanent rules, not incident response. The failure to avoid is leaving incident rules in place permanently without ever reviewing them, which is how most sites accumulate a firewall nobody understands.

“Our false positive rate is fine, we would have heard.”

This is the belief this article exists to challenge. You would not have heard. Blocked people leave silently and disproportionately come from populations already underrepresented in your feedback. Measure it: challenge pass rate by segment, sampled manual review of blocks, and a documented escape path. If the measurement comes back clean, you have bought certainty cheaply.

“The publisher research does not apply to us, we are not a news site.”

Probably true in magnitude and probably false in direction. News discovery is unusually dependent on third party surfaces, so the elasticity is higher there than for a business with direct relationships and long sales cycles. But the mechanism, access equals visibility, is general. Treat 23 percent as a ceiling for a highly discovery dependent business and not as a forecast for yours.

“Tarpits are unethical.”

There is a genuine argument here and we do not think it is silly. Deception wastes another party’s resources by design, it catches archival and research crawlers that serve the public interest, and at scale it degrades the commons. Our position is that it is proportionate against aggressive extraction that ignores published preferences and disproportionate as a default posture, which is why we ship it as configuration rather than as a default. Sites with a strong public interest mission should weigh it more carefully than a commercial retailer.

“We will just block everything that is not a browser and be done.”

That rule blocks your mobile app, your partner integrations, your payment provider callbacks, your monitoring, your accessibility tooling and, depending on how it is written, several crawlers you depend on for distribution. It is the single most common self inflicted outage in this field.

“This is a lot of process for a small site.”

Scale it down, do not skip it. For a small site the whole programme is: verify crawlers properly, exclude non human traffic from your reporting, rate limit your login and your forms, use invisible verification rather than puzzles, and look at your logs once a week. That is a day of work and a recurring half hour.

And the limit we hold most firmly. No enforcement system is correct. Ours is not. What a good one produces is a calibrated probability, a graduated response, an auditable record and a measured error rate in both directions. Anyone selling certainty in this category is describing a marketing position, not a control.

The Five Mistakes We See Most Often

Matching on user agent strings. It fails in both directions simultaneously. Verify instead, using published ranges, forward confirmed reverse DNS and signatures.

Enforcing before observing. A rule whose false positive rate you have not measured is a rule you are about to test on customers.

One global posture instead of a policy per class. It guarantees that your setting is wrong for most of your traffic, and it hides the disagreements that should be explicit.

Treating the challenge as the defence. Machines pass them, humans abandon them, and regulators increasingly have opinions about them. They are one narrow tool for a medium confidence band, not a perimeter.

Rules with no owner and no rollback. The rule that eventually costs you a quarter of your organic traffic will be one that was added during an incident, never reviewed, and understood by nobody still on the team.

A Field Guide to the Traffic You Must Not Block

Before you write a single rule, enumerate the legitimate non browser clients that already touch your infrastructure. Almost nobody does this, and it is the cheapest possible insurance against the most expensive possible mistake.

Every item below is automated. None of it is a browser. All of it is either something you depend on or something a customer depends on. A rule that says “block non browser clients” takes out the entire list.

Payment provider callbacks and webhooks. Your processor posts to your endpoint to confirm an authorisation, a capture, a refund or a dispute. It presents no browser signature, executes no JavaScript, and arrives from a datacentre address. Block it and orders silently stop reconciling.

Your own mobile applications. Native clients authenticate and fetch data through the same API your website uses, with a non browser user agent by design. This is documented by AWS as a leading false positive scenario in bot controls, and it is the single most common self inflicted outage we see.

Partner and integration traffic. Fulfilment providers, inventory systems, shipping carriers, tax engines, CRM syncs, marketing automation, review platforms. Each one is a machine calling you on a schedule, and each one breaks something operational when it stops.

Uptime and synthetic monitoring. Your own observability stack, plus whatever your customers or your compliance obligations require. Ironically, blocking these means your monitoring reports green while the site is broken for humans.

Continuous integration and deployment checks. Smoke tests, accessibility audits, performance budgets, visual regression runs. These usually come from a cloud provider’s address range, which is exactly the range most aggressive rules target first.

Link preview services. Every time somebody shares your URL in a messaging app, a mail client, a social platform or a collaboration tool, something fetches the page to build a card. Block these and your links render as bare text or broken previews everywhere they are shared, which is a marketing loss that will never be traced back to a firewall rule.

Email scanning and security gateways. Corporate mail security follows links in messages before delivering them, which means your transactional emails, password resets and confirmation links get fetched by a machine before the recipient ever sees them. Rules that consume a single use token on first fetch will break for every customer behind such a gateway.

Accessibility tooling. Screen readers do not usually fetch pages independently, but accessibility auditing services, testing harnesses and browser extensions used by disabled visitors frequently produce non standard signatures. Blocking these is both a customer loss and a legal exposure.

Feed readers and syndication clients. Smaller than they used to be, still real, and disproportionately used by exactly the technically engaged audience many businesses want.

Translation and proxy browsers. Data saving browsers, translation proxies and some regional access services fetch and rewrite your pages on the user’s behalf. The visitor is a person; the request is not shaped like one.

Corporate egress and VPNs. Not automation, but shares the failure mode. Hundreds or thousands of employees behind one address will trip any naive rate limit, and the population behind those addresses skews toward exactly the professional buyers a business market wants.

Security scanners you commissioned. Your own penetration testers, your compliance scanning vendor, your bug bounty participants operating under a documented policy.

Ad verification and brand safety vendors. If you run advertising, verification partners fetch your pages to confirm placement, viewability and brand safety. Blocking them can cause your own inventory to be marked unmeasurable.

Archive and preservation crawlers. Whether you want to be preserved is a values question. It is worth answering deliberately rather than by accident.

The exercise itself is simple and takes an afternoon. Pull a week of origin logs, group by network owner and by user agent family, and walk the top hundred rows with somebody from engineering and somebody from marketing in the room. You are looking for the rows where the answer to “what is this?” is “no idea”, and for the rows where the answer is “oh, that is critical”. Both categories are findings.

Then do the part that actually protects you: give every legitimate non browser client an identity you can verify, rather than an exemption you match on a string. An allow rule keyed to a user agent is a hole any attacker can walk through. An allow rule keyed to a shared secret, a client certificate, a signed request or a verified address range is a control.

From Signal to Action: What Each Layer Licenses You to Do

Detection produces evidence. Policy converts evidence into action. The conversion is where most systems are sloppy, because they treat all signals as interchangeable inputs to one score.

They are not interchangeable. Each class of signal has a different false positive profile, a different forgery cost, and therefore justifies a different maximum action. Here is how we map them.

Network ownership and address reputation

What it tells you: which network the request came from, and whether that network is consumer broadband, mobile, cloud hosting, a known proxy service, or something else.

Forgery cost: low to moderate. Residential proxy services exist precisely to defeat this, and they are large and cheap.

False positive profile: high in specific pockets. Corporate egress, VPN users, mobile carrier grade address translation and legitimate cloud based clients all look wrong on this signal alone.

Maximum action it licenses on its own: tag and throttle. Never challenge and never block on network reputation as a single signal. A datacentre address is a reason to look harder, not a verdict.

Declared identity

What it tells you: what the client claims to be, via the user agent string and related headers.

Forgery cost: zero. It is free text.

False positive profile: enormous in both directions.

Maximum action it licenses on its own: nothing at all. Record it. Use it to decide which verification path to run. Never act on it.

Verified identity

What it tells you: that the client is who it claims to be, proven by forward confirmed reverse DNS, published range membership, a valid cryptographic signature, or a shared secret you issued.

Forgery cost: very high for signatures and range plus reverse DNS combined.

False positive profile: low, with one important exception: infrastructure changes. Operators do reassign ranges and rotate infrastructure, so a verification failure is occasionally the operator’s fault rather than an impostor’s.

Maximum action it licenses: full allow, including exemption from other rules. This is the only signal strong enough to grant privilege rather than merely withhold suspicion. It is also why verification is the highest leverage investment in the whole stack.

Transport fingerprint

What it tells you: whether the client’s TLS handshake, HTTP framing and header ordering are consistent with the browser it claims to be.

Forgery cost: moderate to high. Most automation frameworks do not bother, because the operator sets the user agent and never thinks about the layer beneath it. Sophisticated operators do address it.

False positive profile: low but non zero. Unusual TLS stacks, corporate inspection proxies that terminate and re establish connections, older embedded clients and some privacy tooling all produce unusual fingerprints legitimately.

Maximum action it licenses on its own: throttle, and challenge when combined with one other signal. A mismatch between a claimed modern browser and a standard library handshake is among the strongest single signals available, and it is still not sufficient for a block by itself.

Client and rendering environment

What it tells you: whether the client executes JavaScript, renders consistently, and presents a coherent set of capabilities.

Forgery cost: low for the obvious markers, moderate for coherence across many properties.

False positive profile: the worst of any layer. Privacy browsers, hardened enterprise configurations, assistive technology, older devices, unusual locales and users who block scripts all fail these checks legitimately. Anti fingerprinting features exist specifically to make this signal unreliable, which is a design goal for the browser and a problem for you.

Maximum action it licenses on its own: tag. Use it to lower confidence and require corroboration. Never block on a rendering signal alone. This is the layer that generates the demographically skewed false positives described earlier, and it is the layer most products lean on hardest.

Behavioural signals

What it tells you: whether interaction patterns look like a person driving the session.

Forgery cost: moderate, and falling. Simulating plausible pointer movement and scroll cadence is a solved problem for anyone motivated.

False positive profile: moderate to high, concentrated among assistive technology users, keyboard only navigation, and anyone whose interaction pattern differs from the model’s training distribution.

Maximum action it licenses on its own: throttle, or challenge at a high threshold. Useful as corroboration, dangerous as a primary basis for denial.

Application and business context

What it tells you: whether the sequence of operations makes sense for a customer. Twelve carts in an hour. Forty password resets against different accounts. A checkout with no preceding product view.

Forgery cost: high, because evading it requires the attacker to sacrifice the thing they came for. Scale is the point of automation, and this signal measures scale in business units.

False positive profile: low if your limits are set from your own data, high if you copied them from an article.

Maximum action it licenses: up to and including block, on the specific operation. Note the qualifier. Business logic evidence justifies stopping the operation, not banishing the visitor.

Outcome evidence

What it tells you: what actually happened. The payment authorised. The lead answered. The order shipped without a dispute. The account was still active in week four.

Forgery cost: highest available. Faking it means genuinely buying something.

False positive profile: essentially nil, but it arrives too late to inform the request that produced it.

Maximum action it licenses: retrospective everything, including reversing a decision, updating a reputation, retraining a threshold and refunding a wrongly blocked customer’s patience.

The rule that falls out of this table: the action you take should never exceed what your weakest contributing signal can support. If your block was driven by a rendering check with two corroborating network signals, you did not have block grade evidence. You had throttle grade evidence and you rounded up.

Three Incidents, and What We Actually Did

Abstractions are easier to agree with than to apply. Here are three scenarios, composited from real patterns, with the sequence of decisions written out.

Incident one: a catalogue under extraction

What we saw. A retail site with a large public catalogue. Origin requests up 240 percent over four days. Analytics sessions up 11 percent. Bandwidth bill on track to double. Conversion rate flat in absolute terms, which meant it had collapsed relative to traffic.

First move: reconcile the logs against analytics. The gap between a 240 percent origin increase and an 11 percent session increase told us immediately that the incremental population was not executing JavaScript. That single comparison, which took four minutes, eliminated half the possible explanations.

Second move: group by network owner. The increase resolved to a long tail of consumer networks across many countries. Not a datacentre burst. This is the signature of a residential proxy operation, and it meant address based controls were going to be close to useless.

Third move: transport fingerprint comparison. The incremental population claimed current browser versions. Their TLS handshakes did not match those versions. Better than 90 percent mismatch across the incremental traffic. That is as close to conclusive as this discipline gets.

Fourth move: shape of the requests. Sequential traversal of category pages, then product pages, in catalogue order. Consistent interval. No asset requests. No checkout, no cart, no login. Pure extraction, no interaction with anything that costs money.

What we did, in order. Tagged and excluded the population from analytics and from conversion feeds the same day, because that is reversible and free. Applied a per class rate limit on catalogue paths, keyed to transport fingerprint mismatch rather than address, set at a level well above any human browsing pattern. Watched for two days. Then, once we could show the limited population was not touching any revenue path, enabled deception on the extraction paths only, with verified crawlers explicitly exempted.

What we deliberately did not do. We did not block. Blocking would have told the operator exactly what we had detected, and the operator had a proxy pool large enough to restart from a clean position within hours. Throttling and deception cost them time and compute without telling them why.

Outcome. Origin load returned to baseline over about ten days as the operation lost interest. Reported sessions fell nine percent, which required a briefing to leadership before anyone saw the chart. Conversion rate as reported rose, because the denominator got honest.

Incident two: credential stuffing on a software login

What we saw. Login attempt volume up eleven times over a weekend. Success rate near zero. Support tickets about account lockouts from real customers who had done nothing wrong.

The lockout detail is the important one. The attack’s primary damage was not account compromise. It was denial of service against real users through the account lockout mechanism, which is a documented automated threat category in its own right. The defence was hurting customers on the attacker’s behalf.

What we did, in order. Changed lockout from account keyed to source keyed within the first hour, so that an attacker hammering one account could no longer lock its legitimate owner out. Applied simultaneous rate limits across four dimensions, per account, per address, per fingerprint, per global, at different thresholds. Added step up verification for attempts that failed the transport check, rather than blocking them, because a real customer on a new device also fails a device check. Blocked outright only the subset showing both a transport mismatch and attempts against more than five distinct accounts from one source, which is behaviour no customer produces.

Then the slower work. Breach corpus checking at password set time, so the credentials the attacker was testing would stop working. Passkey enrolment promoted in the product. Session invalidation on credential change. Recovery flow hardened to the same standard as login, because it was softer and would have been the next target.

What went wrong. Our first rate limit was set globally rather than per class and briefly throttled a customer’s own integration, which authenticated legitimately at high volume through the API. Their engineer noticed within twenty minutes, which was luck rather than design. We had not enumerated our legitimate non browser clients first, which is why that section exists in this article.

Incident three: the false positive we caused

What we saw. Nothing. That is the point.

Three weeks after a routine tightening of a managed ruleset, a finance team member mentioned in passing that reconciliation had been taking longer. Investigation found that a payment provider’s webhook callbacks were being challenged. The provider retried, some retries succeeded, some did not, and a small percentage of order confirmations had been arriving late or not at all for most of a month.

No alert fired. The webhook endpoint returned a challenge page with a 200 status, which every monitor read as healthy. The provider’s retry logic masked the problem well enough that it looked like intermittent slowness rather than a systematic block.

What we changed afterwards. Every machine to machine endpoint moved to an explicit allow keyed to a verifiable identity rather than inheriting general rules. Monitoring switched from status code checks to content assertions, because a challenge page is a 200. A weekly report of enforcement actions grouped by destination path, so that any action against an endpoint that should never see enforcement is visible immediately. And a rule review that asks, for every rule, which paths it could possibly apply to.

The lesson generalises. The costly false positives in bot defence are not usually customers at the front door. They are integrations, callbacks and machine clients that fail quietly and are retried by systems designed to hide failure. Your monitoring almost certainly cannot see them today.

Sector Playbooks

The framework holds everywhere. The priorities change substantially by business model, and applying a retail playbook to a publisher wastes months.

Retail and e commerce. Your dominant automated pressure is catalogue and price extraction, running constantly rather than in bursts, plus scalping on limited stock and denial of inventory through cart holds. Priorities in order: application aware limits expressed in carts and reservations rather than requests, aggressive expiry on holds, verified crawler exemptions so you never damage product discovery, and deception reserved for extraction paths only. Resist the urge to block scraping outright: much of it comes from comparison and affiliate surfaces that also send you buyers, and separating those from pure extraction is worth the effort.

Business software. Your exposure concentrates at authentication and at free trial signup. High value credentials, generous trials and self serve provisioning are an attractive combination. Priorities: passkeys and breach corpus checking, per account and per source limits with source keyed lockout, verification on trial creation rather than after, and step up rather than block on anything that might be a real person on a new device. Your legitimate non browser client list is long and includes customer built integrations, so enumerate it before you tighten anything.

Publishing and media. Your problem is inverted. You are not trying to keep automation out, you are trying to price it and understand it. The Rutgers and Wharton findings should weigh heavily on your blocking decisions, since discovery in your sector is unusually dependent on third party surfaces. Priorities: verified operator identity in your logs, per purpose policy using content signals plus edge enforcement, deception only against operators that ignore your published preferences, and careful attention to the fact that your decoy content must never be indexable.

Travel, ticketing and events. You face the heaviest automated pressure of any sector and you should calibrate expectations accordingly. Availability and search endpoints are the target rather than content pages. Scalping is a named legal category with statutory backing in some jurisdictions, and that backing depends on you having published limits to circumvent. Priorities: queue systems rather than raw rate limits for on sale events, per payment instrument and per identity limits, explicit published purchase limits, and evidence collection good enough to support enforcement action rather than merely mitigation.

Marketplaces and classifieds. Automated pressure on both sides of the market plus fake account creation plus review manipulation. Your listing pages are one target and your signup flow is the other. Priorities: separate policy for authenticated and anonymous browsing, account creation verification with outcome feedback, and explicit classification of partner scraping you have actually agreed to, which otherwise gets caught by the same rules as hostile extraction.

Financial services. Your pressure skews adversarial rather than extractive, and the regulatory environment raises the cost of both error types. Imperva’s research places financial services as the most targeted sector for account takeover. Priorities: everything in the login section, applied harder, plus API first defence because the attack has usually moved there, plus documented evidence trails because you will need to explain your decisions to someone.

Local and service businesses. Small volumes mean single incidents dominate. Priorities: the cheapest version of everything. Verified crawler handling, form protection with invisible verification, per address limits on your contact form, and a weekly look at your logs. You do not need an architecture. You need four controls and somebody paying attention.

The through line across all seven: work out what an adversary or an aggregator gains from your specific site, and you will predict the shape of your automated traffic before you measure it. The generic question is always the same. The answer never is.

The Metrics of a Healthy Enforcement Programme

You cannot manage what you do not measure, and the metrics that matter for enforcement are different from the ones that matter for traffic measurement. Here is the set we would put on a dashboard and review on a fixed cadence.

Action distribution. What proportion of requests received each action: observed, tagged, throttled, challenged, tarpitted, blocked. This should be a steep pyramid. If your blocked share is larger than your throttled share, your policy is inverted and somebody should say so out loud.

Verification coverage. The share of requests claiming a known identity that you actually verified, versus those you accepted or rejected on the claim alone. Below 90 percent, every other number here is suspect.

Unverifiable share. The proportion of traffic where the operator publishes nothing checkable. This is a policy gap, not a detection failure, and it should be reviewed monthly because operators do start publishing.

Challenge pass rate, segmented. By country, device class, browser family and network type. Segmented, always. An aggregate pass rate of 95 percent can conceal a segment at 55 percent, and that segment is a market you are quietly losing.

Challenge abandonment rate. How many people started a challenge and never completed the action behind it. This is the direct cost of your friction, in the same units as its benefit.

Enforcement actions by destination path. Grouped by endpoint. Any action against a path that should never see enforcement, such as a webhook receiver or a health check, is an incident regardless of volume.

False positive estimate, from sampling. Take fifty blocked or challenged sessions a week, read them by hand, and record how many look human. It is a small sample and it is vastly better than nothing. Track the estimate over time rather than obsessing over its precision in any one week.

Escape path usage. How many people used your documented route to report being wrongly blocked, and what happened to them. Near zero usage is expected. Zero usage over months usually means the path is broken or invisible rather than unnecessary.

Verified crawler access health. For each crawler you depend on: is it currently receiving successful responses, and has that changed? Alert on change, daily. This is the early warning for the most expensive mistake available to you.

Rule age and rule fire rate. Every rule, when it was added, who owns it, when it last fired. Rules that have not fired in ninety days are candidates for removal, and rules with no owner are candidates for removal regardless. The firewall that costs you a quarter of your organic traffic will contain a rule from an incident three years ago that nobody remembers.

Mean time to rollback, measured by drill. Not estimated. Actually timed, at least quarterly, by having somebody roll back a real rule in a real environment. If it takes forty minutes, you will hesitate to enforce anything aggressive, and hesitation is its own failure mode.

Origin load reduction. The infrastructure benefit, which is the number that funds the programme. Requests prevented, bandwidth saved, origin compute avoided.

Two notes on how to present these.

Publish the threshold next to the number. Every one of these metrics depends on a confidence cutoff. If the cutoff is invisible, somebody will compare two periods with different cutoffs and reach a confident wrong conclusion.

Version your rules and record the date of every change. Then, when a chart shows a step change, you can tell instantly whether the world changed or your policy did. This single discipline prevents more bad analysis than any other.

Testing Your Own Defences

Almost nobody tests bot defences, which is strange given that everybody tests everything else. The result is that most organisations discover their configuration only during an incident, which is the worst possible time.

Four kinds of test, in ascending order of effort.

Continuous crawler access checks. Automated, daily, from outside your network. Request a set of representative pages while presenting as each crawler you depend on, and assert on content rather than status code, because a challenge page returns 200. Alert on any change. This is the highest value test by a wide margin and it is a small script.

Synthetic legitimate client checks. For every entry on your non browser client list: your mobile app’s API calls, your payment webhook receiver, your partner integrations, your monitoring. Exercise each one on a schedule and assert on a successful business outcome, not a status code. The webhook incident described earlier would have been caught in one day by this test and instead took three weeks.

Adversarial testing against your own site. With authorisation, in writing, from someone who can grant it. Run a headless browser against your login and see what happens. Try a modest scrape of your catalogue. Attempt a plausible carding pattern against a test payment path. You are not trying to prove you are unbreakable. You are trying to find out at which rung your defences actually engage, and whether the action they take is the one you configured.

False positive simulation. The test nobody runs and everybody needs. Deliberately visit your own site under the conditions that generate your worst false positives: a privacy hardened browser, a corporate VPN, an older device, a screen reader, a connection routed through a market you rarely see traffic from. Complete a purchase or a signup in each configuration. If any of them cannot complete, you have found a revenue leak that no amount of log analysis would have surfaced.

Two operational rules for all of this. Run tests from outside your own network, because your office address is probably on an allow list and will not experience what customers experience. And record test results over time, because the useful signal is usually a change rather than a level.

Governance: Who Owns the Dial

Technical programmes fail for organisational reasons more often than technical ones, and bot enforcement fails in a specific and predictable way: it becomes nobody’s job, so it accumulates rules and never sheds them.

One named person owns the enforcement policy. Not a team, a person. They approve threshold changes, maintain the policy matrix, and are accountable for both error rates. In most organisations this belongs to security or platform engineering, but it must have a standing relationship with whoever owns revenue.

Every rule has an owner, a purpose and an expiry. Written down, in version control, next to the rule. The purpose field should say what it is for in one sentence. The expiry field forces a review rather than automatically deleting anything.

Bot rule changes go through the same review as other search affecting changes. Because they are search affecting changes. Whatever process your team uses for a robots edit or a redirect map should cover a firewall tightening.

Marketing and security share one dashboard. Not two. The most common organisational failure here is that security sees threats blocked and marketing sees traffic lost, and neither view contains the other’s number. One dashboard with both, reviewed together monthly, resolves an astonishing number of arguments before they start.

Incident rules are tagged as incident rules and expire by default. The single most common source of accidental damage is an emergency rule written at two in the morning that was never removed. Tag them at creation, review them within a week, and make removal the default rather than the exception.

Finance sees the infrastructure saving. The programme costs money in tooling and time. It saves money in origin capacity, bandwidth and fraud losses. If only the cost side is visible, the programme will lose its budget in the first difficult quarter.

Write down what you do not know. Every enforcement system has blind spots. Document yours, publish them next to the metrics, and revisit quarterly. A stated limitation is a strength. An unstated one surfaces at the worst possible moment, usually in front of someone senior.

What This Actually Costs to Run

An honest accounting, because the sales material in this category rarely provides one.

Initial implementation. For a single property with a competent engineer: two to four weeks of part time work to reach verified crawler handling, per class rate limits, invisible verification on sensitive forms, and an observe layer. Less if you are already behind an edge platform with these capabilities, since much of the work becomes configuration rather than code.

Ongoing operational load. Roughly half a day a week once established. A weekly false positive review, a look at the action distribution and challenge pass rates, and a check on crawler access health. Plus a half day a quarter for the deeper review and the rollback drill.

Tooling. Ranges from nothing to substantial. Verification against published ranges, forward confirmed reverse DNS and basic rate limiting are free and are most of the value. Edge platform bot management sits in the middle. Dedicated bot and agent trust products at enterprise tier start in the low thousands per month and go up considerably from there.

The cost people forget: your time budget for false positives. Every enforcement layer you add creates a small permanent stream of investigation work. Budget for it explicitly, or it will be absorbed invisibly by whoever is least able to refuse it.

The saving people forget: origin capacity. When a large share of your requests are automated, a large share of your infrastructure spend is serving machines. Multiply your monthly infrastructure cost by your non human request share and you usually get a number big enough to fund the whole programme, which makes it a much easier conversation with an engineering budget holder than a security conversation would be.

Build versus buy, honestly. Build the verification layer: it is small, it is well documented by the operators themselves, and owning it means you understand it. Buy the detection layer if your risk profile justifies it, because maintaining transport fingerprint databases and behavioural models against an adapting adversary is a full time job for a team, not a side project. Always own the policy layer, because it encodes decisions only you can make.

What Happens Next

Three shifts are already visible, and each changes what enforcement means.

Identity is replacing inference. For twenty years, deciding what a client was meant guessing from evidence. Cryptographic bot identity replaces the guess with a check. Cloudflare, AWS, Akamai and others already verify signed requests in production, and an IETF working group has been chartered, though the drafts are still individual rather than adopted and implementations have not fully converged on wire format details. The practical instruction for your roadmap: start recording whether a request was signed as a first class field in your data model now. In eighteen months, signed versus unsigned will be one of the most useful dimensions you have, and only if you have history.

Access is becoming priced rather than merely permitted. Once compensation arrangements exist for content access, the enforcement question changes from allow or deny to allow, deny, or charge. That third option only exists for sites that can attribute load by operator and purpose at the request level. Metering precedes monetising.

The binary is dissolving. The interesting question is no longer whether a request came from software, because nearly all of them do. It is whether there is a human intent behind it and whether that intent is legitimate. Forrester renaming its market category from bot management to bot and agent trust management is a lagging indicator of a shift that has already happened in practice: an agent acting for a real customer with a real payment instrument is not a bot in any sense that should matter to your policy, and treating it as one will cost you sales.

And a caution. Every technique in the detection half of this discipline will decay. Transport fingerprinting works today because most automation ignores it. Behavioural analysis works today because most automation simulates behaviour badly. Both statements have a shelf life measured in single digit years. The two ends of the stack do not decay: cryptographic verification at the bottom and business outcome evidence at the top. Invest disproportionately in the two ends, because the middle is rented.

The Weekend Audit: Finding Out What You Already Enforce

Before adding anything, find out what is already running. Nearly every site we look at is enforcing rules that nobody currently employed there wrote, and roughly a third are enforcing something that is actively costing them.

This audit takes a weekend and requires no new tooling.

Step one: enumerate every place a request can be stopped. Registrar and DNS level controls. Edge or content delivery network rules, including managed rulesets you did not write and defaults you did not choose. Load balancer and reverse proxy configuration. Application middleware. Framework level protections. Plugin or extension level protections, which on content management platforms are frequently the culprit. Your robots file. Your firewall at the host level.

Write them all in one list. Most teams find between four and seven layers, and most had been thinking about two.

Step two: for each layer, list the rules currently active, when they were added and by whom. For managed rulesets, list which groups are enabled and at what sensitivity. For platform defaults, find out what the default actually is rather than what you assume, because defaults change and your site may have inherited a new one at onboarding.

Step three: test crawler access from outside. Request a representative page while presenting as each crawler you care about, from an address outside your network, and check the response body rather than the status code. Verify that your robots file and sitemaps are reachable unconditionally. This is where sites discover that a crawler they depend on has been receiving errors for weeks.

Step four: test every non browser client you depend on. Work through the list from earlier in this article. Payment callbacks, mobile API calls, partner integrations, monitoring, preview services. Exercise each and assert on a real outcome. Anything you cannot test, note as unknown, because unknown is a finding.

Step five: pull a week of enforcement actions and group by destination path. You are looking for actions against paths that should never see them: webhook receivers, health checks, API endpoints your own applications use. Every one of those is a defect.

Step six: sample fifty actioned requests and read them. Not aggregate them, read them. What was the claimed identity, what network did it come from, what did it request, in what order, and does it look like a person or a machine to you? An hour of this teaches more about your traffic than a month of dashboards.

Step seven: check your robots file against your actual enforcement. Does the file permit things your edge blocks? Does it forbid things your edge allows? When they disagree, the enforcement wins and the file is fiction. Reconcile them, and decide deliberately which one you want to be the source of truth.

Step eight: write down what you found, including what you could not determine. Then delete the rules nobody can justify. This is the highest value hour of the whole exercise and it is the one most likely to be skipped, because deleting a security rule feels like taking on risk. In practice, an unowned rule of unknown purpose is the risk.

The deliverable is a single page: layers, active rules, owners, crawler access status, non browser client status, and a list of rules to remove. That page is the foundation for everything else in this article, and it is also the artefact that makes the case for resourcing the work.

Special Cases That Break Every Rule You Just Wrote

Steady state policy is the easy part. The moments that generate the most damage are the ones where normal traffic stops looking normal, and where your rules, correctly implemented, do exactly the wrong thing.

Sale events and product launches. Everything spikes at once. Legitimate traffic looks like an attack: high volume, high concurrency, repeated requests from the same addresses as people refresh. Your rate limits, tuned for a normal Tuesday, will fire on customers. Plan for this explicitly with a documented event mode, which raises thresholds on customer paths while keeping or tightening them on abuse paths, and which someone is authorised to enable without a meeting. For genuinely constrained inventory, a queue is a better instrument than a limit, because it manages fairness rather than merely denying access.

Marketing campaigns. A television spot, an email send or a viral post produces a burst from a narrow geography in a narrow window, frequently arriving with no referrer. Every one of those characteristics reads as suspicious to a naive rule. Get campaign calendars into the same channel as your enforcement changes.

Market expansion. The first month of traffic from a new country is the month your false positive rate there is highest, because your models have seen little of it and your reputation data is thin. Nobody in that market will tell you they could not check out. Deliberately lower enforcement in new markets for the first sixty days, and monitor conversion rate there against your established markets as a defect signal.

Migrations and replatforming. New infrastructure means new addresses, new configuration and a fresh set of defaults. The checklist item that gets missed is verifying that crawler access and non browser client access survived. Add both to your migration runbook as blocking items rather than post launch checks.

Mobile application releases. A new client version changes user agent, request patterns and sometimes transport characteristics. If your rules were tuned against the previous version, the new one may look novel. Coordinate app releases with whoever owns enforcement, which requires those two people to know each other.

Security incidents elsewhere in the industry. After a large breach, credential stuffing volume rises across every site, because a fresh corpus is circulating. Your login defences will be tested by people who were not previously interested in you. Treat major public breaches as a trigger to review authentication controls, not as somebody else’s problem.

Your own testing and load work. Penetration tests, load tests and quality assurance runs all generate traffic that is by definition abusive in shape. Give them verifiable identities in advance rather than address exemptions, and remember to remove access afterwards.

Third party outages. If your verification provider, your challenge provider or your reputation feed is unreachable, what happens? Decide before it happens, per endpoint. Ordinary page requests should be served. Payment authorisations should not. A default of fail closed everywhere turns a vendor’s bad afternoon into your bad afternoon.

The general principle is that your enforcement configuration should have modes, not just settings. Normal, event, incident and degraded, each with defined thresholds, each with a named person who can switch between them, and each documented well enough that the person on call at three in the morning does not have to reason from first principles.

When Your Customer Arrives as a Machine

Everything above treats automation as something to govern. A growing share of it is your customer, arriving through an intermediary, and the enforcement question for that category is genuinely new.

The volume is not hypothetical. HUMAN Security’s analysis of more than one quadrillion interactions found that traffic from AI agents and agentic browsers grew 7,851 percent year over year, with more than 95 percent of AI driven traffic concentrated in retail and e commerce, streaming and media, and travel and hospitality. DataDome found agentic browser traffic concentrated in e commerce and retail at roughly 20 percent of volume, real estate at 17 percent and travel at 15 percent.

The hard part is stated most clearly by HUMAN’s own researchers: an agent browsing products, accessing an account and completing a checkout could be acting for a real customer or executing a fraud operation autonomously, and the behaviour is identical while the intent is not.

That sentence has a direct consequence for enforcement. Behavioural analysis cannot resolve this category. No amount of pointer entropy or scroll cadence distinguishes a legitimate purchase agent from a fraudulent one, because both are agents doing agent things. The signals that do resolve it are structural rather than behavioural:

Verified operator identity. Which platform is this agent from, proven cryptographically rather than claimed. This is the primary reason signed request standards matter commercially rather than merely technically.

Mandate and authorisation. Can the agent demonstrate that a specific person authorised this specific action? The payment and commerce protocol work happening across the industry is largely an attempt to make this checkable.

Payment instrument validity. A completed authorisation against a real instrument is expensive to fake at scale, which makes it one of the strongest available signals.

Downstream outcome. Did the order ship without dispute. Did the account remain in good standing. This arrives too late for the request but is the ground truth that should calibrate everything else.

The policy shape we would recommend today:

Do not block verified agents from public content. They are frequently doing comparison work on behalf of a buyer, and being absent from that comparison is a distribution loss you will never see itemised.

Do apply rate limits, generously. Agents evaluate many options quickly and can generate request patterns that would be abusive from a browser. Generous limits per verified operator handle this without denying access.

Do require stronger evidence at the point of value. Adding to a cart is cheap and should be easy. Completing a purchase, changing an address, redeeming loyalty value or accessing account data deserves verification appropriate to the risk.

Do classify agent traffic separately in every report. An agent session and a scraper session in the same bucket makes both numbers useless, and makes it impossible to have an evidence based conversation about whether this channel is worth investing in.

Do not build your policy around today’s operator list. It will change. Build it around verifiable identity and declared purpose, so that a new entrant is handled by your existing rules rather than requiring an update.

And an honest limitation: measurement frameworks for this channel are immature and anyone offering certainty is overselling. The reasonable response is not to wait for maturity. It is to capture the raw signals now, particularly whether a request was signed and by whom, so that when the frameworks arrive you have history to apply them to.

Explaining Enforcement to People Who Do Not Care About Bots

The last failure mode worth naming is communication, because a technically correct programme that nobody else understands will be overruled during the first difficult quarter.

Three audiences, three framings.

For leadership, lead with the trade you are managing. Not “we blocked 40 million requests.” That number sounds impressive and means nothing to them. Instead: “automated traffic is a third of our infrastructure cost, we have reduced it by half, and our measured false positive rate on customer paths is under a tenth of a percent.” Cost saved, risk reduced, error rate stated. Always state the error rate voluntarily, because the first time somebody else raises it you want to have already been talking about it.

For marketing and revenue, lead with what you protect. They care about two things: that search and AI discovery keep working, and that customers can complete purchases. Show them crawler access health and challenge pass rate by segment. Tell them before you ship anything that could affect either. Give them a veto on anything touching the conversion path, which sounds like a concession and is actually how you get their support for everything else.

For support and sales, give them a script and a route. They are the people a wrongly blocked customer eventually reaches. They need to recognise the symptom, have a phrase for it, and know exactly who to send it to. A support team that can identify an enforcement problem in one message is your best false positive detector and costs nothing to create.

And the piece of internal communication that matters most: brief people before reported numbers move, not after. Enforcement done properly reduces reported traffic. That is the intended effect, it looks like a decline on a chart, and if leadership encounters the decline before they encounter the explanation, you will spend the next month defending the work instead of extending it.

Deception in Depth: How to Use Tarpits Responsibly

The tarpit deserves fuller treatment than it usually gets, because it is the newest tool in the enforcement set, the most misunderstood, and the one with the widest gap between good and bad implementations.

The core insight is about information, not damage. When you block a request, you send the operator a signal: something about that request was detected. A competent operator treats a block as feedback, changes one variable, and tries again. Over a few cycles they learn the shape of your detection by probing it, which means every block is a small donation of intelligence to your adversary.

Deception withholds that signal. The operator receives a valid looking response, their pipeline reports success, and their data quietly fills with material that costs them storage and compute and returns nothing. They learn nothing about your detection because, from their point of view, nothing happened.

The second insight is that traps are also sensors. Cloudflare makes this point explicitly about its own implementation: no real human follows a chain of generated pages four links deep, so anything that does is almost certainly automated, which turns the trap into a very high confidence detection signal that feeds back into classification. A well built honeypot path is one of the few places you can get near certainty about a client’s nature.

Now the implementation rules, because this is easy to do badly.

Decoy content must be unindexable, without exception. Robots exclusion on the paths, noindex directives, nofollow on the links that lead there, and verified search crawlers exempted from the trap entirely. If a search engine indexes your maze, you have published thin machine generated pages under your own domain and created a quality problem far worse than the scraping you were preventing.

Detection quality must precede deployment. A tarpit applied at a low confidence threshold will eventually catch a person, and a person trapped in a maze of nonsense has a worse experience than one who saw a clear error message. Deploy above a high threshold and nowhere else.

Never trap verified crawlers, ever. This should be enforced structurally, as a rule that runs before the tarpit rule and cannot be reordered by accident.

Keep the decoy plausible but valueless. The point is to waste time, not to spread misinformation. There is a version of this technique that deliberately serves false facts in the hope of degrading a model, and we think it is a bad idea: it is difficult to contain, it can be attributed back to you, and the reputational risk is asymmetric. Plausible and empty beats false and pointed.

Watch your own costs. Generating decoy content consumes your resources too. If your trap costs you more per request than the scrape did, you have built an expensive way to lose.

Log everything the trap catches, separately. The intelligence is the point. Fingerprints, addresses, request patterns and timing from trapped clients are among the cleanest training data you will ever collect about automation, precisely because the population is unambiguous.

And weigh the ethics honestly. Deception wastes another party’s resources by design. It catches archival crawlers and academic researchers alongside commercial extractors. At scale it adds noise to a commons that many people depend on. Our position is that it is proportionate against aggressive extraction from operators who ignore published preferences, and disproportionate as a default posture. Organisations with a public interest mission should hold themselves to a stricter standard than a commercial retailer needs to.

The honest summary: a tarpit is a precision instrument with a wide blast radius if you aim it carelessly. Configure it deliberately, threshold it high, exempt verified traffic structurally, and review what it catches every week.

Rate Limiting That Does Not Punish Your Best Customers

Rate limiting is the most useful rung on the ladder and the one most often implemented in a way that produces exactly the wrong outcome: it catches your heaviest legitimate users first, because heavy legitimate use looks like abuse from a distance.

The teams that get this right share a set of habits.

Identify the actor properly before you count. A limit keyed to address alone will treat an entire office, university or mobile carrier as one user. A limit keyed to a session alone is trivially defeated by discarding sessions. Combine: authenticated identity where available, session where not, address and fingerprint as supporting dimensions, and different ceilings on each.

Set limits from your own distribution, not from an article. Pull the actual distribution of requests per minute for the endpoint, per real customer, over a month. Look at the ninety ninth percentile. Set your limit meaningfully above it. If your limit sits below your ninety ninth percentile, you have decided to throttle one percent of your customers as policy.

Give authenticated users more room than anonymous ones. They have proven something and they have more at stake. Give paying customers more room than free ones. Give verified partners more room than unknown clients. Tiering is not a premium feature, it is basic proportionality.

Separate read from write, and cheap from expensive. Serving a cached page and executing a search across your catalogue cost you wildly different amounts. One budget for both is wrong for both.

Communicate the limit. Return a 429 with a Retry After header, publish limits in your API documentation, and expose current usage to authenticated clients where you can. A developer who can see the limit will design around it. One who cannot will retry aggressively and look like an attacker.

Add jitter to your retry guidance so that limited clients do not synchronise and produce a thundering herd at the exact moment your advertised interval expires.

Degrade before you deny on read paths. Serving a cached or simplified response to a client approaching its limit is better for everyone than an error, and it buys you headroom without a hard failure.

Watch who hits limits repeatedly. A client that constantly bumps against a limit is either abusive or badly configured, and those need different responses. Frequently it is a customer’s integration with a retry loop, and a five minute conversation fixes what a block would have escalated.

Never let a limit be global by accident. The most common serious mistake in this area is a single counter applied across all traffic types, which means a legitimate crawler doing exactly what you want consumes budget that then denies a customer. Per class, per endpoint, always.

Handling the Complaint You Will Eventually Receive

At some point somebody will tell you they cannot use your site. How you handle that first report determines whether you find the other cases or never hear about them again.

Treat it as a defect report, not a support annoyance. The person who bothered to tell you represents an unknown number who did not. Their message is a sample from a population you cannot otherwise observe.

Capture what you need in the first reply. Approximate time, page attempted, country, device and browser, and whether they were using a VPN or corporate network. Ask kindly and ask once; a form with twelve fields will lose them.

Reproduce before you theorise. Match the report against your enforcement logs for that window. In most cases you will find the exact action and the rule that caused it in a few minutes, provided you have been logging verdicts with requests. If you cannot find it, that itself is a finding: your logging is insufficient.

Fix the class, not the case. Adding an exemption for one person solves nothing. Ask what population that person belongs to, count how many of them you have, and fix the rule.

Tell them what happened. People are remarkably forgiving of a system that made a mistake and explained it. They are much less forgiving of silence.

Record it. Date, symptom, rule involved, population affected, fix applied. Six months later the pattern will recur in a slightly different form, and the record turns a two day investigation into a ten minute one.

And make the reporting path visible. An error page that says only that access was denied gives a wrongly blocked customer nowhere to go. An error page that says access was restricted, provides a reference code, and offers a route to a human converts an invisible loss into a data point. The reference code matters more than it sounds: it turns “your site is broken” into a log lookup.

A Thirty Day Plan for a Team of One

Not every reader has a security team. If you are a founder, a solo developer or a marketer who has been handed this problem, here is the compressed version. It assumes a few hours a week and no budget beyond what you already pay for hosting.

Week one: find out what is happening. Pull a week of origin or edge logs. Group requests by network owner and by declared identity. Run forward confirmed reverse DNS on the top few declared crawlers and count the failures. List every place a request could currently be stopped, and every non browser client you depend on. Do not change anything. The deliverable is one page describing your situation, and it is worth more than any rule you could write this week.

Week two: fix verification and exclusions. Implement verified crawler handling using published ranges plus forward confirmed reverse DNS, and place that rule above everything else. Exclude verified crawlers and obvious automation from your analytics and from any conversion events you send to advertising platforms. This step has no false positive cost at all, which makes it the safest high value change available to you.

Week three: protect the two endpoints that matter. Your login and your forms. Add invisible verification rather than a puzzle. Add per address and per account rate limits with source keyed lockout. Add a honeypot field and a submission timing threshold to your forms. Verify contact details asynchronously rather than blocking submission.

Week four: instrument the error you cannot see. Set up a daily external check that requests a page as each crawler you depend on and asserts on content. Set up a weekly check for each non browser client. Add a reference code to any error or challenge page and a route to reach you. Then read fifty actioned requests by hand.

Ongoing, thirty minutes a week. Crawler access health, action counts by rule, anything that arrived through your escape path. Once a quarter, delete rules that have not fired and time yourself rolling one back.

That is the whole programme at small scale. It is not sophisticated and it does not need to be. It closes the most damaging failure modes, it costs almost nothing, and it leaves you able to answer the four questions that matter when somebody asks why traffic moved.

A Short Glossary

Shared vocabulary shortens arguments. These are the terms used throughout this article, defined the way we use them.

Automated threat. An abusive outcome achieved through automation, named by what it does rather than what tool it uses. The OWASP ontology numbers twenty one of them.

Verified operator. A client whose claimed identity has been proven, by published range membership, forward confirmed reverse DNS, a valid cryptographic signature, or a secret you issued. The only category strong enough to be granted privilege.

Unverifiable claim. A client claiming an identity that the operator provides no means of checking. Neither trusted nor hostile, and deserving of its own policy.

Forward confirmed reverse DNS. Resolving an address to a hostname, then resolving that hostname back to the address, and requiring both to agree. The standard method for verifying declared crawlers.

Transport fingerprint. The pattern formed by a client’s TLS handshake, protocol framing and header ordering, which reveals what software is actually making the request regardless of what it claims.

Graduated response. A policy in which the action taken scales with the confidence held and the harm at stake, rather than a binary allow or deny.

False block. Denying or obstructing a legitimate visitor. Silent, unbounded, and the error this whole article is organised around.

Shadow mode. Running a rule with logging but no enforcement, in order to measure what it would have done before it does it.

Step up. Requiring additional verification for a specific action rather than denying access to the visitor.

Tarpit. Serving plausible but valueless content to a client believed to be extracting at scale, wasting its resources without informing it that it was detected.

Business logic abuse. Automation that breaks nothing technically and damages you commercially, by doing permitted things at impermissible scale.

Agentic traffic. Requests made by software acting on behalf of an identified person, which may represent a genuine customer arriving through an intermediary.

Content signal. A machine readable declaration of how content may be used after it is fetched, distinct from whether it may be fetched at all.

Escape path. A documented route by which a wrongly obstructed person can reach a human. The only direct measurement you will ever get of your false block rate.

Frequently Asked Questions

Will blocking bots hurt my SEO?

Blocking search crawlers will, badly and with a slow recovery. Blocking other automated traffic will not, provided your rules verify crawler identity rather than matching user agent strings, always permit your robots file and sitemaps, and rate limit per class rather than globally. The risk is almost never the intent of the rule, it is the collateral scope of it.

Is it safe to block AI crawlers?

It is a commercial decision, not a security one, and the evidence suggests it is not free. Research from Rutgers Business School and The Wharton School found that large news publishers who blocked generative AI crawlers experienced roughly a 23 percent decline in total traffic and about a 14 percent decline in panel measured human traffic, with a later revision estimating around 7 percent on a weekly window. The proposed mechanism is reduced brand exposure. Split the decision by purpose: training, search, retrieval and agentic browsing are four different questions.

Should I still use a CAPTCHA?

Not as a general perimeter. Research solvers have reported effectively complete success against common image challenges, while audio alternatives take roughly 28.4 seconds on average and are abandoned about half the time. Prefer invisible verification that only escalates on poor signals, and reserve any interactive challenge for medium confidence attempts at high value actions.

How do I verify that a request is really from a search crawler?

Two methods, and use both. Run a reverse DNS lookup on the source address, confirm the hostname sits under the operator’s expected domain, then run a forward lookup on that hostname and confirm it resolves back to the same address. Separately, match the address against the operator’s published range files, which the major operators publish in machine readable form. Where the operator supports signed requests, verify the signature, which is stronger than either.

What should I do with a request I cannot verify either way?

Put it in its own bucket with its own policy. Some operators publish nothing, which makes their claims uncheckable rather than false. The usual correct treatment is rate limit, exclude from reporting, and do not grant the privileges you would grant a verified crawler.

What is the difference between rate limiting and blocking?

Rate limiting denies scale, blocking denies access. Most automated abuse is only economical at scale, so throttling frequently removes the incentive without removing the possibility. It is also far more forgiving of a detection error: a wrongly throttled customer is slightly inconvenienced, while a wrongly blocked one is gone.

Are tarpits worth deploying?

They are effective and they carry real trade offs. The strongest argument for them is that a block tells an attacker they were detected while deception does not, and that no human follows a maze of generated pages four links deep, which turns the trap into a high confidence detection signal. The arguments against are that decoy content must never be indexable, detection must be excellent before deployment, and archival crawlers get caught in them. Use above a high confidence threshold, never against verified crawlers, and never as a default.

Is scraping my public site illegal?

In the United States, largely not. The Ninth Circuit held in hiQ Labs against LinkedIn that the Computer Fraud and Abuse Act does not prohibit scraping publicly available data, and a 2024 summary judgment in Meta against Bright Data reinforced the distinction between logged out public data and data behind authentication. The practical consequences are that authentication is the meaningful boundary, terms of service are weak against a party with no account, and your technical controls are your primary remedy rather than a supplement to a legal one.

How aggressive should I be on my login page?

Aggressive. It is the clearest case for strong enforcement anywhere on your site, because confidence is high, the endpoint is narrow and the downside is severe. Verizon’s analysis of single sign on logs found credential stuffing at a median 19 percent of all authentication attempts. Prefer passkeys, check credentials against breach corpora, rate limit across several dimensions at once, protect the recovery path as carefully as the login, and step up rather than hard block where the visitor might be a real person on a new device.

How do I know if I am blocking real customers right now?

You will not learn it from complaints. Measure challenge pass rate segmented by country, device class, browser and network type, and treat any segment far below your baseline as a defect. Track abandonment at every enforcement point. Sample fifty blocked sessions a week and read them by hand. Publish an escape path and watch what arrives through it.

What about mobile apps and partner integrations?

They are the most common casualty of blunt rules, because they legitimately present non browser user agents. AWS documents this explicitly as a false positive scenario in its bot controls. Enumerate every legitimate non browser client you have before you write a rule, and give each one an identity you can verify rather than an exemption you match on a string.

Does rate limiting need to be complicated?

No. Pick one algorithm and apply it per class and per endpoint. Sliding window counters are a reasonable default because they smooth the boundary burst problem with fixed memory. Token bucket is better where bursts are a legitimate feature, such as a developer API. Return a 429 with a Retry After header and add jitter to the guidance so limited clients do not synchronise their retries.

Where should enforcement live: edge, gateway, application or database?

All four, doing different jobs. Cheap network and transport checks at the edge, application aware limits in middleware, value transfer checks at the transaction layer, and one shared verdict with one session identifier flowing through all of them. Enforcement your application never learns about is enforcement you cannot reconcile, investigate or reverse.

What is the single highest value change if I only do one thing?

Verify your declared crawlers properly instead of trusting user agent strings, and stop your general bot rules from applying to the verified ones. It takes an afternoon, it eliminates the most damaging category of false positive, and it closes the impersonation route that attackers currently rely on.

How long before I see results?

Infrastructure effects are immediate: throttling and blocking reduce origin load the day they go live. Reporting accuracy improves as soon as you start excluding automated traffic. Fraud and abuse reduction shows up over weeks as operators lose interest and move on. The one thing that takes longest is confidence in your own false positive rate, which needs a few cycles of sampling and review before the number means anything.

Should I block traffic from cloud hosting providers?

Not wholesale. Cloud ranges carry a high concentration of automation, which makes them a strong signal, and they also carry your payment provider callbacks, your monitoring, your partner integrations, several legitimate crawlers and an increasing amount of agent traffic acting for real customers. Use network ownership to lower confidence and to justify throttling. Do not use it alone to deny access.

What do I do about traffic from countries I do not sell to?

Be careful. Geographic blocking is popular because it is easy and it produces a satisfying drop in junk traffic. It also blocks travelling customers, expatriates, people behind VPNs terminating elsewhere, and researchers or journalists looking at your business. If you must restrict by geography, restrict specific sensitive operations rather than the whole site, and never restrict your public content, which is your distribution.

My edge provider already has bot protection turned on. Is that enough?

It is a good foundation and it is not a policy. Managed rulesets encode a vendor’s average judgement about an average site, and your site is not average. At minimum, find out which groups are enabled, what the sensitivity is set to, whether verified crawlers are exempted, and what the defaults were when your site was onboarded. Then check whether any of it is currently acting on paths that should never see enforcement.

How do I convince my team this is worth doing?

Lead with the infrastructure number rather than the security one. Multiply your monthly hosting and bandwidth cost by your non human request share, and present that as the recoverable amount. It is usually large enough to fund the work on its own, it lands with an engineering budget holder rather than requiring a risk conversation, and the security and data quality benefits then arrive as bonuses rather than as the entire pitch.

The Closing Argument

The instinct behind most bot defence is protective and correct. Something is taking from you, and you want it stopped. That instinct built the entire security industry and it is not wrong.

What it needs is a second instinct alongside it, and that one is harder to hold because it works against the first. Every control you deploy has a cost that lands on somebody, and the somebody is not always the attacker.

The attacker adapts. That is the whole shape of the thing. Your rule blocks a technique, the technique changes, the rule stays. Over enough cycles you accumulate a perimeter that stops last year’s attacks and this year’s customers, and nothing in your monitoring will tell you which is which, because the attacker leaves a log line and the customer leaves nothing at all.

So the discipline is not really about detection technology. Detection technology is a rented advantage, good for eighteen months at a time. The discipline is about building a system where you can answer four questions on any given Tuesday:

What did we decide about this request, and why?

How often are we wrong, in each direction, measured rather than assumed?

Who owns the threshold, and can they move it today?

Can we undo this in fifteen minutes?

A team that can answer those four can run aggressive enforcement safely, because they will notice when it goes wrong. A team that cannot answer them is running on faith, and faith works right up until the quarter where organic traffic falls twenty percent and nobody can say why.

Block the attackers. Block them hard, at the login, at the checkout, at the endpoints where the evidence is strong and the harm is real. But start every rule at observe, verify before you trust, throttle before you block, measure the error you cannot see, and make sure somebody’s name is on the dial.

The goal was never a site with no bots on it. That site does not exist and cannot be built. The goal is a site where automation is governed rather than merely resisted, where every class of traffic has a policy somebody chose deliberately, and where the people trying to give you money always get through.

If you want to see which of your visitors are actually being stopped, and what it is costing you, that is what we built ClickBaton to show. And if you build it yourself instead, this article did its job.

References

Threat taxonomy and standards

1. OWASP Foundation. Automated Threats to Web Applications. Project home and threat ontology.
2. OWASP Foundation. Denial of Service. Automated threat event definition.
3. OWASP Foundation. Vulnerability Scanning. Automated threat event definition.
4. F5 Community. Bots, Fraud and the OWASP Automated Threats Project.

Crawler verification and identity

5. Google. Verify Requests from Google Crawlers and Fetchers. Official verification guidance.
6. Cloudflare. Message Signatures Are Now Part of Our Verified Bots Program.
7. Amazon Web Services. AWS WAF Announces Web Bot Auth Support.
8. DataDome. The AI Traffic Report: High Volume, Low Visibility and a Growing Risk. Agent impersonation data.
9. Search Engine Land. What Is Googlebot Fraud. Detection and blocking guide.

False positives and operational risk

10. Cloudflare. Fake Bot Detection Blocking Legitimate Requests. Documented false positive scenarios.
11. Amazon Web Services. AWS WAF Bot Control False Positives.
12. NameSilo. When Blocking User Agents Backfires: SEO and Security Risks.

The cost of blocking

13. Zhao, H. (Rutgers Business School) and Berman, R. (The Wharton School). Strategic Response of News Publishers to Generative AI. Working paper.
14. Zhao, H. and Berman, R. The Impact of LLMs on Online News Consumption and Production. Earlier version with the 23 percent estimate.
15. PPC Land. Blocking AI Crawlers Cost News Publishers 7% of Traffic, Study Finds. Coverage of the revised estimate.
16. PPC Land. Blocking AI Crawlers Backfired: News Publishers Lost 23% of Traffic. Coverage of the original estimate.

Challenges, CAPTCHA and accessibility

17. Plesner, A. et al. Breaking reCAPTCHAv2. ETH Zurich.
18. A Low Cost Attack Against the hCaptcha System. arXiv preprint.
19. An Object Detection Based Solver for Google’s Image reCAPTCHA v2. arXiv preprint.
20. United Nations University Campus Computing Centre. Why CAPTCHAs Are Losing Ground to AI.
21. Smashing Magazine. The Accessibility Problem With Authentication Methods Like CAPTCHA.
22. Cloudflare. Turnstile Documentation. Widget modes and accessibility conformance.
23. Cloudflare. Turnstile Challenge Types.

Proof of work and deception

24. Wikipedia. Anubis (software). Proof of work reverse proxy, origin and adoption.
25. Cloudflare. Trapping Misbehaving Bots in an AI Labyrinth.
26. Cloudflare. AI Labyrinth Documentation.
27. Neuralab. Creative Defense Against AI Crawlers: From Labyrinth to HTML Bombs.

Rate limiting and architecture

28. Digital Applied. API Rate Limiting Strategies: 2026 Engineering Reference.
29. DevWithTools. Rate Limiting Explained: Algorithms and Best Practices.
30. Zuplo. 10 API Rate Limiting Best Practices.
31. Arcjet. Rate Limiting Algorithms: Token Bucket vs Sliding Window vs Fixed Window.

Account takeover and credential abuse

32. MojoAuth. Account Takeover and Credential Stuffing: The 2026 Threat Landscape. Citing Verizon DBIR findings.
33. Security Boulevard. Account Takeover Statistics 2026.
34. HUMAN Security. Credential Stuffing and Account Takeover Attacks Remain Nagging Business Problems.
35. Thales. 2025 Imperva Bad Bot Report. Sector targeting and API attack share.

Denial of service scale

36. Cloudflare. DDoS Threat Report H1 2026.
37. BleepingComputer. DDoS Attacks Over 1 Tbps Surged Fivefold in the Second Quarter.

Legal landscape

38. California Lawyers Association. Ninth Circuit Holds Data Scraping Is Legal in hiQ v. LinkedIn.
39. Farella Braun and Martel. Major Decision Affects Law of Scraping and Online Data Collection, Meta Platforms v. Bright Data.
40. Quinn Emanuel. Meta v. Bright Data: Significant Decision For Web Scraping Industry.
41. Fenwick. The Ninth Circuit Reaffirms That Data Scraping Does Not Violate the CFAA.
42. Futura Tickets. US Ticket Selling Laws 2026: BOTS Act and FTC Fee Rule.
43. HUMAN Security. Stopping Scalper Bots: How to Protect Fans, Customers, and Revenue.

Crawler policy and access economics

44. Cloudflare. Content Signals Policy. Specification and generator.
45. Cloudflare. Cloudflare Gives Creators New Tool to Control Use of Their Content.
46. TechCrunch. Cloudflare’s New Policy Pushes AI Companies to Pay for Publishers’ Content.
47. MLQ News. Cloudflare Sets September 15 Deadline for AI Companies to Separate Training Crawlers.

Market and vendor landscape

48. DataDome. The Forrester Wave: Bot and Agent Trust Management Software, Q2 2026.
49. HUMAN Security. HUMAN Recognized as a Leader in The Forrester Wave for Bot and Agent Trust Management Software.
50. Kasada. Kasada Named a Leader in The Forrester Wave: Bot and Agent Trust Management, Q2 2026.

Last reviewed August 2026. Figures reflect the most recent published editions of each source at that date. Working papers and pending litigation are identified as such in the text, and readers should check the originals for updates before citing.