SC
Scraping Frameworks

Scrapy proxy integration

Scrapy reads the proxy from each request's meta, so you can route an entire spider through Aethyn by setting request.meta['proxy']. For large crawls, set it once in a middleware or in start_requests so every request inherits the residential exit.

Setup steps

  1. 1Set meta['proxy'] on requests (or in a downloader middleware).
  2. 2Use the full http://user:pass@host:port form.
  3. 3Add targeting suffixes to the username as needed.
  4. 4Tune CONCURRENT_REQUESTS and DOWNLOAD_TIMEOUT for residential latency.

Code

Per-request proxy in a spider
import scrapy

class IpSpider(scrapy.Spider):
    name = "ip"
    proxy = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"

    def start_requests(self):
        yield scrapy.Request(
            "https://httpbin.org/ip",
            meta={"proxy": self.proxy},
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info(response.text)
settings.py tuning for residential
# settings.py
CONCURRENT_REQUESTS = 16
DOWNLOAD_TIMEOUT = 30
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504]

Replace aethyn-XXXXX and PASSWORD with your dashboard credentials. See the full targeting reference and protocol/port list.

Tips

  • Rotation is automatic — each request gets a fresh IP unless you use a session.
  • Enable retries on 403/429 to ride out occasional blocks gracefully.
  • Keep concurrency reasonable; residential routes reward steady throughput.

Scrapy proxy FAQ

Do I need a third-party Scrapy proxy plugin?
No. Setting meta['proxy'] uses Scrapy's built-in HttpProxyMiddleware. A custom middleware is only useful if you want to set the proxy globally for every request.

Ready to wire up Scrapy?

Get your credentials and route your first Scrapy request through residential IPs in minutes.

Docs, solutions & blog

Related integrations

All integrations