AutomationPremium tier

How to Run Playwright at Scale

Playwright is easy for one page and brutal at a thousand — memory balloons, IPs get burned, and bandwidth costs explode on images you never needed. This guide covers scaling Playwright the right way: pooling browsers, isolating contexts with a proxy each, blocking heavy resources, and capping concurrency to what your machine can actually sustain.

DWWritten by David Wu·Reviewed by Elena Vance·Updated September 4, 2026
12 min read
Editorial board contrasting one Chromium per URL with a pooled browser and isolated contexts, each with its own proxy.

Key takeaways

  • Reuse a small pool of browsers and create a fresh context per job — don't launch a browser per page.
  • A browser context is the unit of isolation: its own cookies, storage, and proxy. Use one context = one identity.
  • Block images, fonts, media, and analytics via routing — it slashes proxy bandwidth and speeds pages dramatically.
  • Cap concurrency to what RAM allows (each context costs real memory); a queue beats an unbounded fan-out.
  • Assign proxies at the context level for per-job rotation; use sticky sessions for multi-step flows.

On this page

What goes wrong when this scraper fails

A scraper that works on one page with Playwright falls apart at scale. Launching a browser per URL exhausts RAM and CPU; running everything through one IP gets the whole job blocked; and loading full pages — images, fonts, video, trackers — burns proxy bandwidth and time on bytes you discard. The result is a slow, expensive, crash-prone crawler. Scaling Playwright is really about resource discipline: reusing browsers, isolating per-job state in contexts, trimming what loads, and bounding concurrency.

Why this failure mode happens

Each Chromium instance is heavy, so one-browser-per-page multiplies memory until the host thrashes or the OOM killer fires. Playwright's contexts are designed for isolation (separate cookies/storage/proxy), but if you don't use them deliberately, jobs leak state into each other or share one IP. And by default a browser loads every asset on a page; at scale those bytes dominate both runtime and proxy cost even though a scraper rarely needs the images or fonts.

Pipeline aborting images, fonts, and media while keeping HTML through Playwright routing.
Bytes you never parse still bill proxy bandwidth.

Challenges that make this hard to automate

  • Memory and CPU blowups from launching too many browsers
  • Sharing one IP across all jobs and getting the whole crawl blocked
  • Bandwidth wasted on images, fonts, media, and trackers
  • State leaking between jobs without proper context isolation
  • Unbounded concurrency overwhelming the host and the target

Approaches that usually fail

  • A browser per URL — simple but exhausts memory immediately
  • One shared context for everything — state bleed and a single IP
  • Loading full pages — heavy bandwidth and slow runs
  • Throwing more machines at it — costly without fixing the per-node waste

When residential proxies fix this — and when they cannot

Assigning a residential proxy per context gives each concurrent job its own identity, so the work spreads across many IPs and no single one accumulates the velocity that triggers blocks. Rotation per context (or a sticky session per multi-step flow) lets you push real concurrency through Playwright while staying under per-IP limits — turning a fragile single-IP crawler into a horizontally scalable one.

Job queue and semaphore feeding Playwright contexts, with sticky sessions only for multi-step flows.
Assign the proxy on the context. Sticky only for multi-step flows.

How Aethyn residential proxies help here

Scaling Playwright means many concurrent identities and tight bandwidth control. Aethyn supports both directly through the proxy username.

  • Per-request/per-context rotation so each Playwright job gets a fresh IP
  • Sticky sessions (30 min by default, up to 24 h) for multi-step flows in one context
  • Premium pool that's cost-efficient for high-volume browser traffic
  • Per-byte billing that rewards blocking images/fonts/media
  • Country/city targeting per context for geo-specific rendering

How to implement this with residential proxies

  1. 1

    Pool browsers, create a context per job

    Launch a small, fixed number of browsers and reuse them. For each job, open a fresh context (its own cookies, storage, and identity), do the work, and close the context — not the browser. This keeps memory bounded while isolating every job.

    Python (browser pool + per-job context)
    import asyncio
    from playwright.async_api import async_playwright
    
    async def worker(browser, queue, results):
        while not queue.empty():
            url = await queue.get()
            ctx = await browser.new_context(
                proxy={"server": "http://proxy.aethyn.io:2099",
                       "username": "aethyn-XXXXX-country-us",
                       "password": "PASSWORD"})
            try:
                page = await ctx.new_page()
                await page.goto(url, wait_until="domcontentloaded", timeout=30000)
                results.append((url, await page.title()))
            finally:
                await ctx.close()          # close context, keep the browser
            queue.task_done()
    
    async def run(urls, browsers=3, workers_per_browser=4):
        async with async_playwright() as p:
            q = asyncio.Queue()
            for u in urls: q.put_nowait(u)
            results = []
            pool = [await p.chromium.launch(headless=True) for _ in range(browsers)]
            tasks = [worker(b, q, results)
                     for b in pool for _ in range(workers_per_browser)]
            await asyncio.gather(*tasks)
            for b in pool: await b.close()
            return results

    Field note: Recycle browsers periodically (every few hundred contexts) by closing and relaunching them. Chromium accumulates memory over a long run, and a scheduled recycle prevents the slow leak that otherwise crashes multi-hour jobs.

  2. 2

    Assign a proxy per context for rotation

    Set the proxy on new_context, not on launch, so each job can use a different IP. Use a rotating username for per-job rotation, or a sticky-session username when a single context must keep one IP across a multi-step flow (login → navigate → extract).

    Node.js (per-context proxy + sticky)
    const { chromium } = require("playwright");
    
    async function job(browser, url, sessionId) {
      const ctx = await browser.newContext({
        proxy: {
          server: "http://proxy.aethyn.io:2099",
          // sticky: same IP for this context's whole flow
          username: `aethyn-XXXXX-country-us-session-${sessionId}-lifetime-10`,
          password: "PASSWORD",
        },
      });
      const page = await ctx.newPage();
      await page.goto(url, { waitUntil: "domcontentloaded" });
      const title = await page.title();
      await ctx.close();
      return title;
    }

    Field note: Match rotation to the task: per-context rotation for independent page fetches, a sticky session for any flow that sets cookies or logs in. Rotating IPs mid-session is the fastest way to get a flow flagged.

  3. 3

    Block heavy resources to cut bandwidth

    A scraper rarely needs images, fonts, media, or analytics. Intercept requests and abort those types — pages render their DOM far faster and you stop paying proxy bandwidth for bytes you throw away. This is often the single biggest cost and speed win at scale.

    Python (route + abort)
    BLOCK = {"image", "media", "font"}
    
    async def make_lean_context(browser, **kw):
        ctx = await browser.new_context(**kw)
        async def route(r):
            if r.request.resource_type in BLOCK:
                await r.abort()
            else:
                await r.continue_()
        await ctx.route("**/*", route)
        return ctx

    Field note: Blocking images/fonts/media commonly cuts page bytes by 70-90%. With per-byte proxy billing that's a direct, large cost reduction — verify the data you need still loads (some sites lazy-load content as images).

  4. 4

    Cap concurrency to your real limits

    Each context consumes meaningful RAM, so size concurrency to memory, not optimism. Use a bounded queue and a fixed worker count; measure peak RSS per context and set workers = (available RAM × safety factor) / per-context RAM. An overcommitted host is slower than a smaller, stable one.

    Field note: domcontentloaded plus an explicit wait for the selector you need is usually faster and more reliable than networkidle, which can hang on sites with long-polling or analytics beacons that never go quiet.

  5. 5

    Make jobs idempotent and observable

    Persist results per URL so a crash resumes instead of restarting, set timeouts on every navigation and action, and log per-job duration, bytes, and outcome. At scale you need to see which targets are slow, heavy, or blocking so you can tune workers and rotation.

Best practices that keep scrapers reliable

  • Reuse a small browser pool; open and close a context per job
  • Set the proxy on the context for per-job rotation
  • Use sticky sessions only for multi-step flows
  • Block images/fonts/media to slash bandwidth and time
  • Size concurrency to measured per-context RAM
  • Recycle browsers periodically and make jobs idempotent

Common mistakes that burn proxy budget

  • Launching a browser per URL and exhausting memory
  • Running every job through one shared context and IP
  • Loading full pages and paying for unused images/fonts
  • Waiting on networkidle and hanging on chatty sites
  • Unbounded concurrency that thrashes the host
  • No result persistence, so a crash means restarting the whole run

Common questions about this scraper problem

Should I launch a browser per page?
No — that exhausts memory fast. Launch a small pool of browsers and reuse them, opening a fresh context per job and closing the context (not the browser) when done. The context gives you isolation while the pool keeps memory bounded.
How do I rotate proxies in Playwright?
Set the proxy on new_context/newContext rather than at launch, so each job can use a different IP. Use a rotating username for independent fetches, or a sticky-session username (e.g. -session-<id>-lifetime-<min>) when one context must hold the same IP across a multi-step flow.
What's the biggest performance win at scale?
Blocking heavy resources. Routing requests and aborting images, fonts, and media typically cuts page bytes by 70-90%, which dramatically reduces both runtime and per-byte proxy cost while still loading the DOM you scrape.
How many contexts can I run concurrently?
It's bounded by RAM, since each context costs real memory. Measure peak memory per context under load and size workers to (available RAM × safety factor) / per-context RAM. An overcommitted host runs slower than a smaller, stable one.
Why does my long run slowly leak memory?
Chromium accumulates memory over time. Recycle browsers on a schedule (close and relaunch every few hundred contexts) so a multi-hour job doesn't drift into swap and crash. Closing contexts promptly also matters.
Which tier should I use?
Premium suits most high-volume browser automation cost-effectively. Use Elite for specific targets with heavy bot defenses where higher IP reputation materially improves success rates.
Should I wait for networkidle?
Usually not. networkidle can hang on sites with analytics beacons or long-polling. Prefer domcontentloaded plus an explicit wait for the specific selector you need — it's faster and more reliable.
How do I keep proxy costs down at scale?
Block images/fonts/media (per-byte billing rewards it), avoid re-fetching pages you've already processed, and cap concurrency so you don't spawn redundant work. Logging bytes per job shows you which targets to optimize first.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions