Web Scraping CoreElite tier

Identify Which Anti-Bot System Is Blocking Your Scraper

A 403, 429, or empty 200 is not a diagnosis — it is a symptom shared by half a dozen vendors. This hub maps the observable signals (cookies, response headers, status codes, body markers) to the system that produced them, then states honestly whether rotating a residential IP can change that layer.

TMWritten by Thomas Müller·Reviewed by Liam O'Connor·Last verified September 4, 2026
14 min read
Editorial board contrasting guessing the anti-bot from a status code with matching a vendor signal matrix.

Key takeaways

  • Identify the vendor from client-visible artifacts before changing proxies, TLS clients, or concurrency.
  • AKA_A2 proves Akamai delivery/performance features — it is Adaptive Acceleration, not a Bot Manager clearance cookie.
  • A 403 with no x-kpsdk-* headers is not Kasada; look at the CDN/WAF in front (observed: Server: DOSarrest on some retail properties).
  • DataDome returns the same HTTP 403 shape for device check, CAPTCHA, and ban — parse the body, do not trust the status alone.
  • F5 Shape leaves almost nothing in the response; the fingerprint is in request telemetry headers your client is expected to send.
  • Run a three-way test (plain curl, curl with Chrome UA, curl through residential) before blaming IP reputation.

On this page

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

Signal observedWhat it indicatesWhich layerDoes a proxy change it
cf-ray / __cf_bm / challenge HTMLCloudflare edge markers or Managed Challenge / Turnstile pageCDN / Bot Management edge — TLS, JS challenge, and IP reputation layersResidential IP helps reputation and velocity; JS challenges still need a real browser. See /solutions/scrape-behind-cloudflare.
Server: AkamaiGHost + Reference #18… or Reference 0…Access Denied HTML with Mime-Version: 1.0; optional _abck / ak_bmsc cookiesAkamai Bot Manager / edge deny — Reference #18.x is access denied; Reference #9.x is malformed URL (not a bot block)Residential helps IP/geo velocity. TLS/HTTP/2 fingerprint and sensor clearance are not fixed by IP alone. See /solutions/fix-akamai-access-denied-scraping.
x-datadome: protected + datadome cookieHTTP 403 HTML or JSON pointing at captcha-delivery.com; little or no brandingDataDome device check / CAPTCHA / ban — identical 403 envelope for multiple outcomesParse the challenge URL or dd object. IP rotation addresses reputation-shaped bans; client/TLS failures need a browser-grade stack. See /solutions/handle-datadome-blocks-python.
access-control-expose-headers lists x-kpsdk-*HTTP 429 (or 403 on some API routes) with x-kpsdk-ct; cookie pair name + name-ssnKasada challenge gate — 429 is often the challenge, not a classic rate-limit budgetA 403 without x-kpsdk-* is not Kasada. Residential helps ASN reputation; the challenge itself needs a capable client. Deep page planned.
X-Iinfo / visid_incap_* / incap_ses_*Imperva edge headers and Incapsula-lineage cookies; may also see reese84 or ___utmvcImperva Cloud WAF and/or Advanced Bot Protection — reese84/___utmvc are Distil-lineage signals, not Incapsula brandingResidential helps ASN and velocity. JS tokens are client-side. Deep page planned.
_px3 / meta name=description content=px-captchaPerimeterX/HUMAN risk cookie or press-and-hold challenge templateHUMAN Bot Defender — often ships with high blockingScore so MONITOR mode scores without hard-blocking by defaultClean residential IP improves the thin signal set when no valid _px3 exists. Challenge UI needs a browser. Deep page planned.
No branded response + endpoint asymmetryHomepage 200s; a login POST or availability XHR returns opaque JS / empty data; no Stable vendor cookieOften F5 Shape / Bot Defense — fingerprint lives in request telemetry headers, not a named response cookieDo not confuse TS* / BIGipServer* (BIG-IP ASM/LTM) or reese84 (Imperva) with Shape. Residential helps ASN/velocity only.
HTTP 200 + Access Denied title / empty shellStatus looks healthy; body is a block or unavailable page (e.g. lowes.com Access Denied title)Soft block / content assertion failure — status-code monitors miss itAssert title, selectors, and body length. Proxy may or may not help depending on layer — never accept 200 alone.
Detector rows of cookie, header, and path markers used to identify Cloudflare, Akamai, DataDome, Kasada, HUMAN, and Imperva.
Markers live in cookies, headers, and paths — save the raw Set-Cookie and response headers, not a memory summary.

How to diagnose and fix this scraper failure

  1. 1

    Capture status, Server, Set-Cookie, and the first 500 bytes of body

    Before rotating anything, log the full response for one failing URL: status code, Server header, every Set-Cookie name, custom x-* headers, and a body prefix. That tuple is what the matrix below matches. Do not discard 200s — soft blocks live there.

    Field note: Save raw headers to a file. Memory summaries drop Mime-Version, x-datadome, and access-control-expose-headers — the exact tells you need.

  2. 2

    Run the three-way diagnostic recipe

    Plain curl, curl with a current Chrome User-Agent, and curl through a residential proxy must all hit the same URL. Two-way tests misattribute UA-triggered blocks to IP reputation. Record all three outcomes before changing your scraper framework.

    bash (three-way)
    # 1) plain
    curl -sS -D - -o /tmp/body1.html "https://TARGET/" -w "\nHTTP %{http_code}\n" | head -n 40
    
    # 2) Chrome UA only
    curl -sS -D - -o /tmp/body2.html "https://TARGET/" \
      -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
      -w "\nHTTP %{http_code}\n" | head -n 40
    
    # 3) residential proxy (Premium HTTP 2099)
    curl -sS -D - -o /tmp/body3.html "https://TARGET/" \
      -x "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099" \
      -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36" \
      -w "\nHTTP %{http_code}\n" | head -n 40

    Field note: If (1) and (2) fail identically but (3) succeeds, IP reputation is implicated. If all three fail the same way, look at TLS/JS — not the proxy pool size.

  3. 3

    Match signals to a vendor, then follow the deep page

    Use the playbook table on this page: Cloudflare (cf-ray), Akamai (AkamaiGHost + Reference #), DataDome (x-datadome), Kasada (x-kpsdk-*), Imperva (X-Iinfo / visid_incap_), PerimeterX (_px3 / px-captcha meta), Shape (response silence + endpoint asymmetry). Then open the vendor guide — do not invent a bypass from a generic 403 article.

  4. 4

    Assert content, not only status, on every accepted response

    Treat HTTP 200 with an Access Denied title, challenge shell, or missing price node as a failure. lowes.com has been observed returning 200 with an Access Denied title — status-code-only monitors green-light poison.

    Python (content assert)
    def accept(resp, min_len=2000, must_have=("price",)):
        if resp.status_code != 200:
            return False
        text = resp.text or ""
        open_t, close_t = "<" + "title>", "</" + "title>"
        title = text[text.find(open_t): text.find(close_t) + len(close_t)].lower()
        if "access denied" in title or "just a moment" in title or "px-captcha" in text.lower():
            return False
        if len(text) < min_len:
            return False
        return all(tok in text.lower() for tok in must_have)
  5. 5

    Only then rotate residential IPs — and only for IP-shaped rows

    When the matrix says the failure is ASN reputation, velocity, or geo, switch to Premium or Elite residential with country targeting. When it says TLS, JS challenge, or sensor token, keep the same IP and fix the client (curl_cffi / Playwright). Publishing what a proxy cannot fix is part of a correct diagnosis.

What goes wrong when this scraper fails

Your scraper failed and the first instinct is to rotate proxies. That is wrong when the signal is a TLS mismatch, a JavaScript challenge, or a vendor that returns HTTP 200 with a block body. The useful first question is which anti-bot (or CDN WAF) produced the response — because each one exposes different artifacts and each one treats IP reputation as a different weight. This page is the cross-vendor entry point: match what you observed, then follow the deep guide for that vendor.

When residential proxies fix this — and when they cannot

Residential proxies change the IP-shaped layers: datacenter ASN penalties, shared-IP reputation, per-IP velocity, and geo gates. They do not execute JavaScript, forge TLS/JA3, or mint vendor sensor tokens. Use them after you know which layer failed — otherwise you will rotate forever on a client fingerprint the edge never tied to the IP.

Three-way curl columns: plain, Chrome user-agent only, and residential proxy.
Run plain curl, Chrome UA, and a residential exit on the same URL before you change the scraper framework.

How Aethyn residential proxies help here

Once the signal matrix points to an IP-shaped failure, Aethyn gives you country/city/ISP targeting and sticky or rotating residential exits on published ports — so the three-way diagnostic and the fix use the same credential format.

  • Premium HTTP 2099 / SOCKS5 1099 and Elite HTTP 5499 / SOCKS5 3499 — dedicated ports, not one shared endpoint
  • Username suffixes for country, city, ISP, and sticky sessions (1–1440 min) documented at /docs/targeting
  • Self-serve credentials so you can A/B plain curl vs residential without a sales call
  • Elite tier when Bot Management–class targets penalize commodity residential reputation
  • No claim that residential alone defeats a named vendor — pair with curl_cffi or Playwright when the matrix says fingerprint/JS

Best practices that keep scrapers reliable

  • Identify before you rotate — one logged response beats ten blind retries
  • Always run the three-way curl matrix on a new target
  • Treat cookie presence as deployment evidence, not clearance
  • Disambiguate CDN-in-front from the bot vendor behind it
  • Keep public-data collection and ToS limits in scope — no CAPTCHA-solving recipes here
  • Prefer Elite residential when Bot Management–class edges penalize commodity exits

Common mistakes that burn proxy budget

  • Blaming the proxy for Akamai/DataDome fingerprint failures
  • Calling every 429 a rate limit (Kasada challenges often use 429)
  • Treating AKA_A2 as a bot cookie
  • Assuming Reference #9.x is an access denial (malformed URL)
  • Identifying Shape by TS* or reese84 cookies (wrong products)
  • Accepting HTTP 200 without body assertions

Common questions about this scraper problem

How do I identify which anti-bot is blocking my scraper?
Log status, Server, Set-Cookie names, custom headers, and a body prefix, then match them on this page’s signal matrix. Cloudflare shows cf-ray; Akamai often shows AkamaiGHost plus a Reference #; DataDome shows x-datadome: protected; Kasada exposes x-kpsdk-* in access-control-expose-headers.
Does a residential proxy bypass Akamai or DataDome by itself?
No. Residential exits improve IP reputation, velocity, and geo. They do not fix TLS/HTTP fingerprint mismatches or JavaScript challenges. Use the matrix row’s “does a proxy change it” guidance per vendor.
Why did plain curl fail but a browser work on the same IP?
That pattern usually means client fingerprint or JS challenge, not IP ban. Confirm with the three-way test: if Chrome-UA curl still fails and only a full browser succeeds, you need Playwright or equivalent — not more IPs.
What is the difference between Reference #18 and Reference #9 on Akamai?
Reference #18.x denotes access denied. Reference #9.x indicates a malformed URL / request problem — not a bot block. Do not rotate proxies for a #9.
Is every HTTP 403 with no branding DataDome?
No. Confirm x-datadome: protected and/or the datadome cookie or captcha-delivery challenge URL. Generic 403s are also issued by CDNs and origin WAFs.
What Aethyn ports should I use for these tests?
Premium HTTP 2099, Premium SOCKS5 1099, Elite HTTP 5499, Elite SOCKS5 3499 on proxy.aethyn.io — username form aethyn-XXXXX-country-us[-city-X][-isp-X][-session-X-lifetime-N].
Is scraping behind anti-bot systems legal?
Collect only public data you are allowed to access, respect robots.txt and site Terms, and avoid circumventing authentication or CAPTCHA as a product feature. This guide is diagnostic, not a bypass manual.

References

Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions