CR
Scraping Frameworks

Crawlee proxy integration

Crawlee (by Apify) is a Node.js crawling framework with built-in proxy rotation via ProxyConfiguration. Provide your Aethyn endpoint as a proxy URL and Crawlee routes its requests through residential IPs.

Setup steps

  1. 1Install Crawlee: npm install crawlee.
  2. 2Create a ProxyConfiguration with your Aethyn proxy URL.
  3. 3Pass it to your crawler (Cheerio/Playwright/Puppeteer).
  4. 4Add targeting suffixes in the username.

Code

Crawlee with ProxyConfiguration
import { CheerioCrawler, ProxyConfiguration } from "crawlee";

const proxyConfiguration = new ProxyConfiguration({
  proxyUrls: ["http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"],
});

const crawler = new CheerioCrawler({
  proxyConfiguration,
  async requestHandler({ $, request }) {
    console.log(request.url, $("title").text());
  },
});

await crawler.run(["https://example.com"]);

Replace aethyn-XXXXX and PASSWORD with your dashboard credentials. See the full targeting reference and protocol/port list.

Tips

  • Crawlee retries failed requests and rotates proxies automatically.
  • Swap CheerioCrawler for PlaywrightCrawler when you need a real browser.
  • Use sticky sessions for flows that must hold one IP.

Crawlee proxy FAQ

Does Crawlee rotate proxies automatically?
Yes. ProxyConfiguration rotates across the provided proxy URLs, and Aethyn rotates exits per request, so you get rotation at both layers.

Ready to wire up Crawlee?

Get your credentials and route your first Crawlee request through residential IPs in minutes.

Docs, solutions & blog

Related integrations

All integrations