ScrapingProxies

curl_cffi vs Requests vs Playwright for Web Scraping (2026)

AE
Engineering desk
Published
September 19, 2026
Reviewed by
Read Time
9 min read
Same residential exit, two TLS stacks: Python requests OpenSSL versus curl_cffi Chrome impersonation

Use the least expensive client that reliably produces the required document. That is the whole job. Python requests is cheapest. curl_cffi costs a bit more and can make the transport resemble Chrome. Playwright is a real browser and the most expensive per page. Move up only when the cheaper client cannot see the document. Move down after a JavaScript page has already failed — that is cheaper than guessing.

A three-client scraping test holds the residential proxy username constant and changes only the HTTP client. You fetch the same URL three times: once with requests, once with curl_cffi and impersonate="chrome", once with Playwright. You record HTTP status and a body class. You do not publish a pass rate from this page. The residential exit is the experimental control, not a fourth client.

Anti-bot systems can evaluate signals beyond the IP address, including TLS characteristics, HTTP/2 behavior, JavaScript execution, cookies, and browser/session consistency. Buying more residential bandwidth does not execute JavaScript. Changing the User-Agent does not make requests use Chrome's TLS stack or reproduce Chrome's TLS fingerprint. Launching Chromium on a datacenter ASN does not make the handshake look like a household.

ClientJavaScriptTLS impersonationBrowserTypical use
requestsNoNoNoSimple HTTP / API
curl_cffiNoChrome-like TLSNoTLS-sensitive HTTP
PlaywrightYesBrowser TLSYesJS-rendered / challenge pages

The table is a starting map. The class you actually get still depends on the destination — a JSON API, a server-rendered article, a hydrated offer grid, a consent wall, or a challenge page.

Copy-paste for curl_cffi lives at the curl_cffi integration. Launch versus context, and credentials in fields, live in Playwright with residential proxies. What JA3 and JA4 actually measure lives in TLS fingerprinting for scrapers. This page does not duplicate those three jobs.

Note

[!NOTE] How to read this page. The three-client protocol, the body classes, and the decision table are the citable artifacts. Reuse them with attribution to Aethyn and a link back here. This article does not publish first-party pass rates. We did not measure a live target for this ship. HTTP 200 is not success.

When to use each client

Use requests when the payload is an API or JSON endpoint, simple server-rendered HTML, and there is no meaningful TLS or browser challenge.

Use curl_cffi when HTTP is sufficient, TLS characteristics matter, JavaScript is not required, and you want lower overhead than a browser.

Use Playwright when JavaScript renders the target content, browser cookies or storage matter, DOM interaction is required, or the target requires browser execution.

Pick the client for the layer
TLS-only jobs stay on curl_cffi. JS hydrate and challenge cookies need a browser. Buying more IPs does not execute JavaScript.

What each client actually is

Python requests is an HTTP library on OpenSSL (or whatever TLS stack your environment ships). It can send a Chrome User-Agent. Changing that header does not make it use Chrome's TLS stack or reproduce Chrome's TLS fingerprint. Against a destination that only checks the IP, a residential exit is often enough. Against a destination that scores library handshakes, it is still a named library. The encyclopedia for that handshake is TLS fingerprinting, not this page.

curl_cffi is a Python binding for curl-impersonate. impersonate="chrome" is a rolling alias for a current Chrome profile (the library default moves; do not freeze chrome124 in new copy). It aligns cipher order, extensions, and HTTP/2 frames with that profile, and injects matching default headers unless you set default_headers=False. That is the TLS and HTTP/2 layer. It is not a browser. It does not run JavaScript, does not paint a DOM, and does not clear a JS challenge. Chrome TLS impersonation can make the transport resemble a browser, but it does not reproduce browser execution, DOM behavior, Web APIs, or JavaScript-generated state.

Playwright is a real Chromium (or Firefox, or WebKit) whose every request — document, XHR, WebSocket — can exit through the same residential username. It executes scripts. It is also the most expensive client per page, and the easiest one to misconfigure: Chromium strips user:pass from a proxy URL and answers 407. Credentials go in username and password fields. Per-context proxies on Chromium need launch({ proxy: { server: "per-context" } }).

The residential exit is a fourth thing, not a fifth client. It rewrites the address and ASN. It does not pick the HTTP library, run JavaScript, or hide a library handshake. How one vendor encodes country, city, and sticky lifetime is later — How Aethyn fits the test.

Three-client test (run this, then cite your classes)

Hold these constants: URL, language, device class, proxy username, sticky session id, and collection window. Change only the client.

  1. Pin the exit. Hold the same proxy session for the three fetches. Verify with an IP lookup through that session before the first target request. Record exit_ip, resolved_city, resolved_isp.
  2. Fetch with requests. Same proxy URL, timeout 30. Store http_status, body length, and a class from the list below.
  3. Fetch with curl_cffi. Same proxy URL, impersonate="chrome". Store the same columns. Do not reuse the requests cookie jar.
  4. Fetch with Playwright. Same username and password in fields, not in the server string. New context, not a reused one. Store the same columns after the document is idle enough to classify.
  5. Classify the body before you insert a row. Status 200 with a challenge page is not a pass.

Body classes (fail-closed):

ClassMeaning
populatedThe document you came for is present (JSON keys, HTML article, offer cards).
empty200 or 204 with no payload, or a shell with no content nodes.
consentCookie or consent wall ("Before you continue", consent.google.com).
challengeInterstitial, unusual-traffic, /sorry/, JS challenge HTML.
wrong_geoResolved city or country is not what you asked for.
http_errorNon-200, including 407 (proxy auth) and 403/429.
Three clients on one residential username
Same proxy username. Record status and body class, not only the status code.

Do not average these into a “success rate.” A challenge on requests and a populated on curl_cffi is a TLS story. A challenge on both HTTP clients and a populated grid in Playwright is a JavaScript story. A challenge on all three is not a client bug — classify the wall (Cloudflare, DataDome, consent) before you buy more IPs.

Fictional example (not a measurement). Suppose requests returns 200 / challenge, curl_cffi returns 200 / challenge, Playwright returns 200 / populated. The layer that moved is JavaScript, not the handshake and not the ASN. Do not cite those three outcomes as Aethyn data. Replace them with your run and your collected_at.

Public web only. Respect robots.txt, rate limits, and Terms. This protocol samples publicly rendered pages. It is not a login, checkout, CAPTCHA, or Cloudflare-bypass recipe.

Decision table

Use the least expensive client that reliably produces the required document. Moving up costs CPU, bandwidth, and operational complexity. Moving down after a JS page has already failed is cheaper than guessing.

If you observe…Try nextDo not
http_error 407Rebuild username/password; credentials in fields for PlaywrightRotate IPs
requests challenge, curl_cffi populatedStay on curl_cffiLaunch Chromium “to be safe”
Both HTTP clients empty or challenge; Playwright populatedPlaywrightBuy more residential GB
All three challengeClassify the interstitial; sticky after a real cookie if one existsImpersonate harder on an HTTP client
wrong_geoNew session id; verify againLabel a country fallback as the city
consentHold sticky when session continuity is required; otherwise record consent as a separate document class and follow the site's permitted consent flowParse the wall as content
JSON or server-rendered HTML, all classes populated on curl_cffiStay on curl_cffiPay for a browser

When the destination is Cloudflare-shaped and the document never appears without a JS challenge, the collector playbook is scrape behind Cloudflare — still not a bypass. When the page loaded and the HTML is still a deny, that is Playwright silent 403.

How Aethyn fits the test

A browser download is larger than an HTTP GET. That is not a reason to avoid Playwright when the grid is JavaScript. It is a reason not to use Playwright for a JSON API. The thesis still holds on this network: pick the cheapest client that returns the document, then encode location and sticky on the username.

Identity has two layers and they are easy to glue together wrongly:

  • Session() / Playwright context is the cookie jar and storage.
  • Aethyn -session-<id>-lifetime-N is the exit IP on a rotating pool (1–1440 minutes, default 30).

A curl_cffi Session() on a rotating username will send cookies from identity A out of IP B. A Playwright context reused across session ids does the same. Hold sticky for the flow that minted the cookie; start a new session id for the next identity. Sticky is not a dedicated IP. When the window ends, the pool may hand you a different household in the same country — verify again if the job continues.

Premium versus Elite is location, not “stealth.” Country catalog and currency: Premium HTTP 2099 / SOCKS5 1099. City or ISP: Elite HTTP 5499 / SOCKS5 3499. A city suffix on Premium is refused, not silently ignored. Asking for a city is not landing there — if resolved_city does not match, do not label the row as that city. SOCKS5 does not hide TLS; it tunnels TCP. Copy-paste usernames are on the curl_cffi integration and the targeting docs.

Soft 200s

HTTP 200 with unusual-traffic, a consent wall, or an empty app shell will poison any comparison you later call a pass rate. Classify before you store. The longer treatment is HTTP 200 access denied.

curl_cffi can return 200 with Chrome-shaped TLS and still serve a challenge HTML page. Playwright can return 200 with a rendered deny. The status code is a transport fact. The body class is the measurement.

What this page is not

It is not a JA3 encyclopedia — TLS fingerprinting already is. It is not a Playwright setup guide — Playwright residential proxies already is. It is not a curl_cffi copy-paste card — the integration already is. It is not a claim that curl_cffi “bypasses Cloudflare,” clears CAPTCHAs, or beats Playwright on a published percentage.

If you need a number for a slide, run the three clients, store the schema, and put your collected_at on the slide.

FAQ

Common questions about this article

What is a three-client scraping test?
It is a protocol that holds the residential proxy username constant and changes only the HTTP client: Python requests, then curl_cffi with impersonate, then Playwright. You record HTTP status and a body class (populated, empty, consent, challenge, wrong_geo). It is not a published pass-rate table.
Does this page publish curl_cffi success rates?
No. We did not run a live target measurement for this article. HTTP 200 is not success. If you need a number, run the three-client protocol, store collected_at, and cite your own class counts. A percentage without a retrieval date is not reproducible.
When is curl_cffi enough instead of Playwright?
When the document you need is already in the first HTTP response: a JSON API, a server-rendered HTML page, or any payload that does not require JavaScript to appear. curl_cffi can make the transport resemble Chrome. It does not execute scripts, hydrate an offer grid, or clear a JS challenge.
When should I switch from curl_cffi to Playwright?
When requests and curl_cffi fail the same way through a clean residential exit, and the failure is an empty shell, a JS challenge, or a sensor that never runs in an HTTP client. Same proxy username, new Playwright context. Credentials go in username and password fields, not in the server URL.
Is impersonate the same as setting a Chrome User-Agent?
No. A User-Agent is a header. Changing it does not make requests use Chrome's TLS stack or reproduce Chrome's TLS fingerprint. impersonate on curl_cffi aligns the TLS Client Hello and HTTP/2 frames with a browser profile, and injects matching default headers unless you disable them. See the TLS fingerprinting guide for what JA3/JA4 measure.
Does a residential IP fix a library TLS fingerprint?
No. The proxy rewrites the exit address and ASN. The client still performs the TLS handshake to the destination. A household IP with a Python OpenSSL stack is still a library handshake. SOCKS5 does not hide it either — it tunnels TCP.
Share this article

Guides, integrations & docs

Continue reading

Same username. Pick the client for the layer.

Premium HTTP 2099 for country volume. Elite 5499 when the target scores the city. Sticky 1–1440 minutes on a rotating pool. Free trial, no card.

Back to all articles