ScrapingProxies

Scrapy Proxy Middleware for Rotating and Sticky Residential IPs (2026)

AE
Engineering desk
Published
November 7, 2026
Reviewed by
Read Time
5 min read
Downloader middleware priorities in Scrapy: the proxy middleware at 350, retry at 550 and Scrapy's own proxy handling at 750

A Scrapy proxy middleware is a downloader middleware that sets request.meta["proxy"] on every outgoing request, so a spider's traffic leaves through a proxy without each Request carrying the setting. With Aethyn it also builds the proxy username, which is where the country, the session id and the lifetime live. One small class can run a crawl on rotating exits, hold a single exit for a multi-page walk, and give a retried request a fresh exit. The Scrapy integration page shows the minimal meta['proxy'] setup. This guide covers the middleware that wraps it and the two ordering mistakes that quietly break it.

The code on this page was run against a local stand-in proxy on Scrapy 2.20 to check the client side: the login, the username format, the session handling and the retry behaviour. It was not run against live targets, so treat the exits it prints as yours to verify. The spider uses start(), which needs Scrapy 2.13 or newer. Older releases use start_requests().

The middleware

The class reads the account, password, host, port, country and lifetime from settings, so no credential sits in a spider. If a request carries meta["session"], the username gets session and lifetime suffixes and the request holds one exit. With no session, the username carries only the country and each request can draw a new exit. The password is percent-encoded before it goes into the proxy URL. Scrapy decodes the login in that URL when it builds the Proxy-Authorization header, so a password containing @, : or / arrives intact. A session id cannot contain a dash, because the dash separates username fields, so the middleware raises on one.

Five inputs (account, country, session, retry count and lifetime) combine into the proxy username for a sticky request, a retried request and a rotating request
How the username is built for each kind of request.
Code Snippet
from urllib.parse import quote


class AethynProxyMiddleware:
    """Put an Aethyn proxy URL on every request.

    meta["session"] -> sticky: the same id keeps the same exit for the lifetime.
    No session      -> rotating: each request can draw a new exit.
    """

    def __init__(self, user, password, host, port, country, lifetime):
        self.user, self.password = user, password
        self.host, self.port = host, port
        self.country, self.lifetime = country, lifetime

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(
            user=s["AETHYN_USER"],
            password=s["AETHYN_PASSWORD"],
            host=s.get("AETHYN_HOST", "proxy.aethyn.io"),
            port=s.getint("AETHYN_PORT", 2099),
            country=s.get("AETHYN_COUNTRY"),
            lifetime=s.getint("AETHYN_LIFETIME", 30),
        )

    def process_request(self, request, spider):
        # Leave a proxy the spider set itself alone, but re-sign our own on retries.
        if request.meta.get("proxy") and not request.meta.get("_aethyn"):
            return
        parts = [self.user]
        if self.country:
            parts += ["country", self.country]
        session = request.meta.get("session")
        if session:
            if "-" in session:
                raise ValueError("session id must not contain '-'")
            retries = request.meta.get("retry_times", 0)
            if retries:
                session = f"{session}r{retries}"  # a retry gets a fresh exit
            parts += ["session", session, "lifetime", str(self.lifetime)]
        user = "-".join(parts)
        request.meta["proxy"] = (
            f"http://{quote(user, safe='')}:{quote(self.password, safe='')}"
            f"@{self.host}:{self.port}"
        )
        request.meta["_aethyn"] = True

Settings, and why the number is 350

Code Snippet
# settings.py
import os

AETHYN_USER = os.environ["AETHYN_USER"]  # aethyn-XXXXX, from the dashboard
AETHYN_PASSWORD = os.environ["AETHYN_PASSWORD"]
AETHYN_COUNTRY = "de"
AETHYN_LIFETIME = 30

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.AethynProxyMiddleware": 350,  # must be below 750
}

RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]
CONCURRENT_REQUESTS = 16
DOWNLOAD_TIMEOUT = 30

Keep the middleware below 750. Scrapy's own HttpProxyMiddleware sits at 750 and turns the login inside meta["proxy"] into a Proxy-Authorization header. If your middleware runs after it, nothing signs the login. In a local test at priority 800, Scrapy raised ValueError: invalid hostname, because the user:password@ text was still sitting in the proxy URL when the downloader read it. At 350 the middleware runs before both the retry middleware (550) and the proxy handling (750). Keep the credentials in environment variables, as above, and out of the repository. The concurrency and timeout values match the Scrapy integration page. Raise concurrency only while the share of 2xx responses holds.

Rotating, sticky, and one session per walk

Request typemetaUsernameExit
Independent URLsnoneaethyn-XXXXX-country-deCan change on every request
Paginated walk or login{"session": "cart42"}aethyn-XXXXX-country-de-session-cart42-lifetime-30Same exit until the lifetime ends

Set the session on the first request of a walk and copy it onto the follow-ups with response.follow(href, meta={"session": response.meta["session"]}). Use one session id per walk, not one per spider. Every request that shares an id shares an exit, so a high CONCURRENT_REQUESTS on a single id puts the whole crawl on one IP. The longer explanation of the two modes is in how rotating residential proxies work and sticky proxies explained.

Rotating, sticky and retry requests, each with its username shape and the exits the site would see
Three kinds of request, three username shapes. IPs in the figure are documentation addresses.

Retries that change the exit

Scrapy's RetryMiddleware re-queues a 403, 429 or 5xx with the request's meta copied across. If a proxy middleware skips any request that already has meta["proxy"], the retry reuses the exact login that just failed. On a sticky session that is the same blocked exit, and the retry fails the same way.

The middleware above marks its own proxy with _aethyn and signs the request again on every pass. On a retry it appends r1, r2 to the session id, using retry_times, so the retry opens a new session and draws a new exit. In a local test where the stand-in proxy answered 429 for session page1, the guarded version finished on page1r1. The version that skipped requests with a proxy never produced a result. A spider that sets its own proxy on a request is left alone, because the flag is only on proxies this class set. For what a 403 or 429 means before you retry, see fixing 403 and 429 when scraping, and for a 407 see the proxy authentication guide.

Two lanes: a retry that keeps the old session and runs out of attempts, and a retry that opens a new session and returns 200
Re-sign the proxy on retry, or the retry hits the same exit.

Check the exit before the crawl

Code Snippet
import json

import scrapy


class ExitCheckSpider(scrapy.Spider):
    name = "exit_check"

    async def start(self):
        for _ in range(3):
            yield scrapy.Request(
                "https://api.ipify.org?format=json",
                meta={"session": "check1"},  # remove this line to rotate
                dont_filter=True,
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info("exit %s", json.loads(response.text)["ip"])
The exit check run once with the session line, printing one address three times, and once without it, printing three addresses
What to look for. Addresses in the figure are illustrative.

Run scrapy crawl exit_check. Three log lines with the same address mean the session holds. Remove the session line and the three addresses can differ. Confirm the country with an IP lookup through the same proxy URL before you point the crawl at a target, because a status code of 200 does not prove the exit landed where you asked. Targeting reference is in the docs. To run it on real traffic, see rotating residential proxies on Premium HTTP port 2099, the Elite pool on port 5499 when you need city or ISP, and pricing.

FAQ

Scrapy proxy questions

What is a Scrapy proxy middleware?
A Scrapy proxy middleware is a downloader middleware that sets request.meta['proxy'] on outgoing requests. It lets a whole spider use a proxy without each Request setting it, and it can also build the proxy username, including country, session id and lifetime.
Do I need a middleware, or is meta['proxy'] enough?
meta['proxy'] is enough for a small spider, and Scrapy's built-in HttpProxyMiddleware handles it. A middleware helps when every request should use the proxy, when the username changes per request (a session id per walk), or when retries should get a new exit.
How do I keep one IP across several Scrapy requests?
Put the same session id in meta['session'] on each request of the walk. The middleware adds -session-<id>-lifetime-<minutes> to the username, and the same exit is held until the lifetime ends (1 to 1440 minutes, default 30). Use one id per walk, not one for the whole spider.
Why does a retried request keep failing on the same IP?
RetryMiddleware copies the request's meta, including the proxy. If your middleware skips requests that already have a proxy, the retry reuses the same login and the same sticky exit. Re-sign the proxy on each pass and change the session id on retries so the retry opens a new session.
Share this article

Guides, integrations & docs

Continue reading

One middleware, rotating or sticky

Premium HTTP is port 2099. Country, session and lifetime go on the username. Free trial, no card.

Back to all articles