Web ScrapingElite tier

How to Prevent IP Blocking During Web Scraping

IP bans are the single most common reason scrapers fail — but 'the IP got banned' is usually shorthand for several different detection signals firing at once. This guide breaks down exactly what modern anti-bot systems measure and how to design a pipeline that reads as ordinary traffic.

TMWritten by Thomas Müller·Reviewed by Sarah Chen·Updated September 4, 2026
13 min read
Editorial board contrasting a single IP lever with a four-layer block score of reputation, velocity, TLS, and behavior.

Key takeaways

  • Blocking is a score, not a switch: IP reputation, request velocity, TLS/HTTP fingerprint, and behavior all feed it.
  • Fixing one signal rarely helps — a perfect residential IP with a Python TLS fingerprint still gets flagged.
  • Velocity is per-IP, so the fix is a large pool rotated thin, not a few fast IPs.
  • Soft blocks (200 OK with altered/empty data) are more dangerous than hard 403s because they corrupt data silently.
  • Back off and rotate on the first challenge; retrying the same path immediately is what turns a throttle into a ban.

On this page

What goes wrong when this scraper fails

Nearly every scraping project hits the same wall eventually: a run that worked yesterday now returns 403s, 429s, a CAPTCHA, or — most insidious — a 200 OK with stripped or altered content your parser quietly records as valid. Teams then play whack-a-mole, swapping proxies or adding random sleeps, without a model of what is actually being detected. The result is a pipeline that is perpetually one site update away from breaking.

Why this failure mode happens

Modern anti-bot stacks (Cloudflare, Akamai, DataDome, PerimeterX) score every request across four layers. First, IP reputation: datacenter ASNs are pre-discounted and shared by thousands of scrapers. Second, velocity: how many requests an IP and subnet send, and how evenly. Third, fingerprint: your TLS handshake (JA3/JA4) and HTTP/2 frame ordering — a Python or Go client looks nothing like Chrome even with a spoofed User-Agent. Fourth, behavior: header order, cookie handling, mouse/scroll on JS pages. Cross a threshold on the weighted sum and you are challenged, regardless of how good any single signal is.

Four columns for reputation, velocity, TLS fingerprint, and behavior feeding one score.
Velocity is per IP. Widen the pool. Do not push a few exits faster.

Challenges that make this hard to automate

  • Datacenter IP ranges that arrive pre-flagged and throttle as a whole subnet
  • Per-IP velocity thresholds that one greedy worker can trip for everyone
  • TLS/JA3 and HTTP/2 fingerprints that betray non-browser HTTP clients
  • Inconsistent header order, cookies, and Accept-Language that read as automation
  • Soft blocks that return altered or empty data instead of an honest error code

Approaches that usually fail

  • Sprinkling time.sleep() everywhere — slower, but does nothing about reputation or fingerprint
  • Cycling a handful of datacenter IPs — they are flagged together as one subnet
  • Randomizing only the User-Agent — ignores the TLS/HTTP fingerprint that gives you away
  • Retrying aggressively on errors — the fastest way to escalate a throttle into a hard ban

When residential proxies fix this — and when they cannot

Residential proxies fix the reputation layer (real consumer IPs) and, just as importantly, give you a pool large enough to keep each IP's velocity near human levels. That addresses two of the four detection layers directly. Pair them with coherent headers and — for the toughest targets — a real browser engine that carries a genuine TLS fingerprint, and your requests stop standing out as a fleet of bots.

Session rail from a first 403 through retrying the same path into a ban, then fail-closed.
Soft 200s with empty bodies are more dangerous than honest 403s. Assert content.

How Aethyn residential proxies help here

Reputation and velocity are the two layers a proxy network owns, and they are the two that block most scrapers. Aethyn is built around both, with rotation and stickiness controlled in the username.

  • High-reputation residential IPs that begin with real-user trust, not a flagged ASN
  • A large pool plus per-request rotation so per-IP velocity stays near human levels
  • Sticky sessions with explicit lifetimes for flows that genuinely need continuity
  • Country targeting so the exit geography is plausible for the site you hit
  • Elite tier for the most aggressively defended targets (Cloudflare/Akamai-class)

How to implement this with residential proxies

  1. 1

    Rotate IPs per request to keep velocity human-scale

    Start with the layer that blocks most scrapers. Use the base username so every request exits from a fresh residential IP, which keeps per-IP velocity near what a real person generates. This is why pool size beats request speed: 100k requests across 50k IPs is invisible; 100k across 50 IPs is a siren.

    Python (requests)
    import requests
    
    PROXY = "http://aethyn-XXXXX:PASSWORD@proxy.aethyn.io:5499"
    proxies = {"http": PROXY, "https": PROXY}
    r = requests.get("https://example.com", proxies=proxies, timeout=30)
    print(r.status_code)

    Field note: Resist the urge to find 'the safe requests-per-IP number'. There isn't one — it varies by target and time. Keep velocity low and scale by widening the pool, not by tuning a magic per-IP limit.

  2. 2

    Match the fingerprint, not just the User-Agent

    This is the step most guides skip. Even from a perfect residential IP, a plain requests or aiohttp call ships a TLS handshake (JA3/JA4) and HTTP/2 frame order that scream 'Python', so a spoofed Chrome User-Agent fools nobody. For protected targets, use a client that impersonates a real browser's TLS stack — or drive an actual browser, which carries a genuine fingerprint for free.

    Python (curl_cffi)
    from curl_cffi import requests as cffi
    
    # Impersonates Chrome's TLS/JA3 + HTTP/2 fingerprint, not just the UA string
    r = cffi.get(
        "https://example.com",
        impersonate="chrome",
        proxies={"http": PROXY, "https": PROXY},
        timeout=30,
    )
    print(r.status_code)

    Field note: A good litmus test: hit a JA3-reflecting endpoint (e.g. a TLS-fingerprint check service) through your client and through Chrome. If the hashes differ wildly, the IP was never your real problem — the handshake was.

  3. 3

    Keep headers and cookies coherent per session

    Send a complete, realistic header set and keep it stable for the life of a session — don't randomize the User-Agent on every request from the same cookie jar, which is itself a tell. Reuse cookies within a sticky session; a site that sets a cookie and never sees it again knows you are not a browser.

    Python (session)
    session = requests.Session()
    session.headers.update({
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Sec-Fetch-Site": "none",
    })

    Field note: Header order matters too. Browsers send headers in a consistent order; some HTTP clients alphabetize or reorder them. Clients like curl_cffi preserve browser-like ordering, which is one less thing to give you away.

  4. 4

    Treat the first challenge as a signal to slow down

    Detect 403/429/CAPTCHA explicitly and respond by backing off with jitter, rotating to a fresh IP, and lowering concurrency — never by retrying the same request immediately. Crucially, also detect soft blocks: a 200 OK whose body is suspiciously short or missing your target selector is a block in disguise.

    Python (detect + backoff)
    import time, random
    
    def classify(resp, marker_selector_found):
        if resp.status_code in (403, 429):
            return "hard_block"
        if resp.status_code == 200 and not marker_selector_found:
            return "soft_block"      # 200 but the content you expected is gone
        return "ok"
    
    def backoff(attempt):
        time.sleep(min(60, 2 ** attempt) + random.random())  # exponential + jitter

    Field note: Log block rate as a first-class metric and alert on it. A slow climb almost always means concurrency crept up or a target tightened defenses — catching it early beats discovering a week of corrupted data later.

  5. 5

    Cache aggressively and request only what you need

    The cheapest request is the one you never send. Cache responses, skip pages that have not changed (ETag/Last-Modified where available), and fetch only the fields you actually use. A smaller footprint is both cheaper and less detectable — fewer requests means fewer chances to trip any threshold.

Best practices that keep scrapers reliable

  • Think in four layers — reputation, velocity, fingerprint, behavior — and fix the one that's failing
  • Rotate per request and scale by pool width, never by per-IP speed
  • Match the TLS/HTTP fingerprint (browser-impersonating client or real browser) for protected sites
  • Keep headers, cookies, and order coherent for the life of a session
  • Detect soft blocks (200 OK, missing content) and back off with jitter on any challenge
  • Cache, skip unchanged pages, and request only what you use
  • Stay within robots.txt and each site's Terms of Service

Common mistakes that burn proxy budget

  • Blaming 'the IP' when the TLS/JA3 fingerprint is what's flagged
  • Cycling a few datacenter IPs that throttle together as one subnet
  • Randomizing the User-Agent while ignoring every other signal
  • Retrying instantly on a 429 and escalating a throttle into a ban
  • Recording soft-blocked (altered/empty) 200s as valid data
  • Chasing a mythical 'safe' requests-per-IP number instead of widening the pool

Common questions about this scraper problem

Why do I still get blocked even with residential proxies?
Because a proxy only fixes two of the four detection layers (reputation and velocity). If your HTTP client has a non-browser TLS/JA3 fingerprint, inconsistent headers, or you're retrying on errors, you'll still be flagged. Match the fingerprint and back off properly alongside the residential IPs.
What signals do anti-bot systems actually use?
Roughly four: IP reputation (datacenter vs residential), request velocity per IP/subnet, network fingerprint (TLS JA3/JA4 and HTTP/2 frame order), and behavior (header order, cookies, interaction on JS pages). They're combined into a weighted score, so one weak signal can sink an otherwise clean request.
What is a 'soft block'?
A 200 OK that returns altered, truncated, or empty content instead of the page you asked for. It's more dangerous than a 403 because nothing errors — your parser happily stores garbage. Always validate that expected content is present before trusting a response.
Rotating or sticky sessions to avoid bans?
Rotate per request for stateless scraping to keep velocity low. Use a short sticky session only when a flow genuinely needs one IP (login, pagination, cart), then go back to rotating.
How fast can I scrape without getting banned?
There's no universal number — it depends on the target, locale, and timing. Keep per-IP velocity human-scale and scale total throughput by widening the proxy pool, not by pushing individual IPs harder.
Do I need a real browser to avoid blocks?
Not always. Raw HTTP with a browser-impersonating TLS client (e.g. curl_cffi) handles many sites cheaply. Reserve a full headless browser for the most defended or JavaScript-gated targets, where its genuine fingerprint and behavior help.
How should I handle CAPTCHAs?
Treat them as a threshold signal: slow down, rotate to a fresh IP, and reduce concurrency. For JS-heavy targets a real browser through the proxy reduces challenge frequency. Solving services exist but are a last resort, not a substitute for lowering your footprint.
Which tier handles the toughest sites?
Elite. Its higher-reputation pool and city/ISP targeting are best for aggressively defended targets like those behind Cloudflare or Akamai enterprise protections.
Is avoiding blocks the same as ignoring the rules?
No. The goal is to look like ordinary traffic, not to evade legitimate access controls. Respect robots.txt and Terms of Service, avoid authenticated or personal data you have no right to, and collect only public information.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions