Web ScrapingPremium tier

How to Build a Proxy Rotation System

Plenty of teams build a proxy rotation layer when they shouldn't, and skip the parts that matter when they do. This guide covers what a real rotation system needs — health checks, ban detection, weighting, and session affinity — and when a managed rotating endpoint makes most of it unnecessary.

DWWritten by David Wu·Reviewed by Liam O'Connor·Updated September 4, 2026
12 min read
Editorial board contrasting naive round-robin through a dead IP with health checks, cooldown, weighting, and affinity.

Key takeaways

  • Rotation is more than round-robin: it needs health checks, ban detection, cooldowns, and weighting.
  • Keep session affinity — some flows must stick to one IP — separate from stateless rotation.
  • Detect bans by response signals (403/429, challenge markers), then cool down the offending IP, don't discard it.
  • Track per-IP success rate and weight selection toward healthy IPs.
  • A managed rotating endpoint (rotation handled server-side) removes most of this plumbing — build your own only when you must.

On this page

What goes wrong when this scraper fails

When a scraper outgrows a single IP, teams reach for a rotation system. The common version — a list of proxies cycled round-robin — fails quietly: dead and banned IPs stay in rotation, multi-step flows break because the IP changes mid-session, and there's no visibility into which IPs are healthy. A real rotation system needs health checking, ban detection with cooldowns, success-weighted selection, and session affinity. The deeper question is whether to build it at all, since managed rotating endpoints handle rotation server-side.

Why this failure mode happens

Naive round-robin assumes every proxy is equally healthy and every request is independent — neither is true. IPs get temporarily blocked, vary in quality, and some workflows (login, cart, pagination) require IP continuity. Without state — health, cooldowns, affinity — rotation degrades into routing traffic through dead or burned IPs. The plumbing to do it well is non-trivial, which is exactly why provider-side rotation exists.

Four-stage rotation system: health check, ban detect, weight, session affinity.
Cool down a burned IP. Do not discard a residential exit forever.

Challenges that make this hard to automate

  • Removing dead/banned IPs instead of cycling through them
  • Detecting bans reliably from heterogeneous response signals
  • Maintaining session affinity for multi-step flows
  • Weighting selection toward consistently healthy IPs
  • Observability: knowing per-IP success rates and pool health

Approaches that usually fail

  • Round-robin over a static list — routes through dead and banned IPs
  • Random selection — same problem, no health awareness
  • Manual list curation — unsustainable as IPs churn
  • No affinity handling — multi-step flows break on rotation

When residential proxies fix this — and when they cannot

A managed residential endpoint performs rotation server-side: each request can exit from a fresh IP from a large, continuously-maintained pool, and a sticky session keeps one IP for a flow — all selected via the username, with no client-side pool to health-check. That removes the hardest parts (liveness, churn, scale). If you still build your own orchestration (e.g. across multiple sources), the same endpoint becomes one healthy, high-quality upstream behind your logic.

Form contrasting a self-built rotator with a managed rotating endpoint.
Keep session affinity in the username even when rotation is server-side.

How Aethyn residential proxies help here

Most rotation plumbing disappears when rotation is handled at the endpoint. Aethyn exposes both rotation and affinity through the username.

  • Server-side per-request rotation across a large residential pool
  • Sticky sessions via -session-<id>-lifetime-<min> for flow affinity
  • Country/city targeting so rotation stays within the right geography
  • Premium pricing that's efficient for high request volumes
  • Per-byte billing so a busy rotation workload stays predictable

How to implement this with residential proxies

  1. 1

    Decide whether to build at all

    If you need stateless rotation and sticky sessions over residential IPs, a managed rotating endpoint already does it — point your client at it and move on. Build your own orchestration only for genuine multi-source needs (combining providers, custom routing policies). Don't reimplement pool maintenance you can get for free.

    Python (managed rotation — no client pool)
    import requests
    
    # Rotation handled server-side: each call may exit a fresh residential IP.
    ROTATE = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
    # Sticky: same IP for a multi-step flow.
    STICKY = "http://aethyn-XXXXX-country-us-session-cart42-lifetime-10:PASSWORD@proxy.aethyn.io:2099"
    
    requests.get("https://example.com", proxies={"https": ROTATE}, timeout=30)

    Field note: The cheapest rotation system is the one you don't maintain. Reserve a custom layer for when you truly orchestrate multiple upstreams — otherwise endpoint-side rotation is less code and fewer failure modes.

  2. 2

    Model the pool with health state

    If you do build orchestration, represent each upstream with state: status (healthy/cooldown), success/failure counters, and a cooldown-until timestamp. Selection should consider health, not just order. This is what separates a real rotator from round-robin.

    Python (pool state)
    import time, random
    from dataclasses import dataclass, field
    
    @dataclass
    class Upstream:
        url: str
        ok: int = 0
        fail: int = 0
        cooldown_until: float = 0.0
        def healthy(self): return time.time() >= self.cooldown_until
        def score(self):  # success rate, smoothed
            n = self.ok + self.fail
            return (self.ok + 1) / (n + 2)

    Field note: Smooth the success rate (add-one/Laplace) so a single early failure doesn't permanently sink a good upstream. Raw ratios are too jumpy when sample counts are small.

  3. 3

    Detect bans and cool down, don't discard

    Treat 403/429 and challenge markers as ban signals: increment failure, put the upstream in cooldown for a backoff interval, and keep it for later — most blocks are temporary. Discarding IPs permanently shrinks your pool over a long run.

    Python (ban handling)
    def record(up: Upstream, resp):
        banned = resp.status_code in (403, 429) or "captcha" in resp.text.lower()
        if banned:
            up.fail += 1
            up.cooldown_until = time.time() + min(600, 30 * (up.fail))
        else:
            up.ok += 1
        return not banned

    Field note: Use escalating cooldowns per consecutive failure rather than a fixed one. An IP that fails repeatedly should rest longer, while a one-off failure shouldn't sideline a healthy IP for long.

  4. 4

    Select by weight, keep affinity for sessions

    For stateless work, pick among healthy upstreams weighted by score so traffic favors reliable IPs. For multi-step flows, pin a session to one upstream (or one sticky endpoint session) for its lifetime so cookies and login state survive — never rotate mid-flow.

    Python (weighted choice + affinity)
    def pick(pool):
        healthy = [u for u in pool if u.healthy()]
        if not healthy:
            return min(pool, key=lambda u: u.cooldown_until)  # least-bad
        return random.choices(healthy, weights=[u.score() for u in healthy])[0]
    
    sessions = {}  # session_id -> Upstream (affinity)
    def pick_for_session(pool, sid):
        if sid not in sessions or not sessions[sid].healthy():
            sessions[sid] = pick(pool)
        return sessions[sid]

    Field note: Keep stateless rotation and session affinity as two distinct code paths. Conflating them is how flows end up rotating mid-session — the single most common rotation bug.

  5. 5

    Make the pool observable

    Expose per-upstream success rate, cooldown count, and pool-wide healthy ratio. Alert when the healthy ratio drops, which usually means your global velocity is too high rather than the IPs being bad. Without metrics you're flying blind on a degrading pool.

Best practices that keep scrapers reliable

  • Prefer endpoint-side rotation; build custom only for multi-source needs
  • Model upstream health, not just rotation order
  • Cool down banned IPs with escalating backoff instead of discarding
  • Weight selection by smoothed success rate
  • Keep session affinity separate from stateless rotation
  • Expose pool health metrics and alert on drops

Common mistakes that burn proxy budget

  • Round-robin over a list with no health awareness
  • Permanently discarding IPs on a single block
  • Rotating IPs mid-session and breaking logged-in flows
  • Reimplementing pool maintenance a managed endpoint already provides
  • Using raw success ratios that overreact to one failure
  • No observability into which IPs are healthy

Common questions about this scraper problem

Should I build a proxy rotation system or use a managed endpoint?
If you need stateless rotation and sticky sessions over residential IPs, a managed rotating endpoint already handles rotation, liveness, and scale server-side — just point your client at it. Build your own orchestration only when you genuinely combine multiple upstreams or need custom routing policies.
Why is round-robin not enough?
It assumes every proxy is equally healthy and every request is independent. In reality IPs get temporarily banned and vary in quality, and some flows need IP continuity. Without health state, cooldowns, and affinity, round-robin routes traffic through dead or burned IPs.
How do I detect that an IP is banned?
Use response signals: 403/429 status codes and challenge markers (CAPTCHA/anti-bot strings) in the body. On a ban, increment the failure count and put the IP in an escalating cooldown rather than discarding it, since most blocks are temporary.
How do I handle multi-step flows during rotation?
Keep session affinity separate from stateless rotation. Pin a session to one IP (or one sticky endpoint session) for its whole lifetime so cookies and login state survive, and never rotate mid-flow — that's the most common rotation bug.
How should I choose among healthy IPs?
Weight selection by a smoothed success rate so traffic favors reliable IPs without overreacting to a single early failure. Use add-one smoothing on the ratio so small sample sizes don't unfairly sink a good IP.
Which tier should I use?
Premium fits high-volume rotation workloads cost-effectively, with country targeting and sticky sessions available through the username. Step up to Elite for specific targets where higher IP reputation improves success rates.
What metrics should a rotation system expose?
Per-IP success rate and cooldown counts, plus a pool-wide healthy ratio. Alert when the healthy ratio drops — that usually signals your global velocity is too high, not that the IPs are bad.
Does endpoint-side rotation remove all my work?
It removes the hardest parts — liveness, churn, and scale of the pool. You still control request pacing, retry/backoff logic, and session affinity choices in your client, but you no longer maintain a list of IPs or health-check them yourself.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions