Web Scraping CorePremium tier

Your Scraper Framework Did Not Tell You It Was Blocked

The anti-bot page loaded. Your framework shrugged. Playwright’s page.goto hands back a Response with .status 403 and does not throw; Selenium’s driver.get likewise raises nothing while driver.title becomes Access Denied. Timeouts on #product-price are a late, misleading signal — this pipeline makes silence impossible.

JSWritten by Jordan Smith·Reviewed by Alex Rivera·Last verified September 4, 2026
12 min read
Editorial board showing a green page-loaded check beside HTML that is still an Access Denied document.

Key takeaways

  • Playwright navigation to a 403 block document does not raise — inspect response.status after goto.
  • Selenium driver.get is quieter still: no exception, title becomes the deny page title.
  • Node fetch also refuses to throw on 403 (ok=false); only your code can fail the job.
  • Selector timeouts are symptoms, not diagnoses — classify status/title before waiting on commerce nodes.
  • Proxy 407 without credentials can hang Playwright for the full goto timeout with no auth hint.
  • Wire an explicit classify → assert → attribute → remediate pipeline around every navigation.

On this page

Pipeline stages from request to verified data

  1. Navigate

    Capture Response/status (Playwright) or immediate title/URL (Selenium) — never assume throw-on-deny.

  2. Classify

    Map status + title + body markers to block vs document vs network failure.

  3. Assert

    Require commerce selectors only after classification says the document is real.

  4. Attribute

    Send the artifact tuple to the anti-bot hub before changing proxies.

  5. Remediate

    IP-shaped → residential; fingerprint/JS → client stack; never retry blindly on silence.

Playwright panes where page.goto finishes without throwing while page.title is Access Denied.
Check title and a selector after goto. Do not trust the lack of an error.

How to build each pipeline stage

  1. 1

    Treat navigation as data, not success

    In Playwright, always read the Response from page.goto (or wait_for_response). Log status, final URL, and title before any commerce locator. In Selenium, read driver.title and page_source length immediately after get().

    Python (Playwright)
    from playwright.sync_api import sync_playwright
    
    def goto_classified(page, url: str):
        resp = page.goto(url, wait_until="domcontentloaded", timeout=45_000)
        status = resp.status if resp else None
        title = page.title()
        return {"status": status, "title": title, "url": page.url}
    
    # Verified class of behaviour: status 403 → no exception from goto itself.
  2. 2

    Build the classify stage before waits

    If status >= 400, or title matches Access Denied / Just a Moment / px-captcha patterns, fail closed with a BlockedError. Only then call wait_for_selector on product nodes.

    Python (Selenium sketch)
    from selenium import webdriver
    
    def get_classified(driver, url: str):
        driver.get(url)  # raises nothing on HTTP 403 HTML
        title = driver.title or ""
        if "access denied" in title.lower():
            raise RuntimeError(f"blocked_document title={title!r}")
        return title
  3. 3

    Separate proxy failures from site blocks

    A dead proxy yields net::ERR_PROXY_CONNECTION_FAILED quickly in Playwright. A 407 without credentials can hang until the goto timeout with no status — configure proxy username/password first. That hang is not an anti-bot signal.

  4. 4

    Three-way curl outside the browser

    When the browser is silent, confirm the wire with plain curl, Chrome-UA curl, and residential curl so you know whether the framework merely hid a normal HTTP deny.

    bash
    URL=https://TARGET/
    UA='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
    echo plain; curl -sS -o /dev/null -w '%{http_code}\n' "$URL"
    echo ua; curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" "$URL"
    echo res; curl -sS -o /dev/null -w '%{http_code}\n' -A "$UA" \
      -x 'http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099' "$URL"
  5. 5

    Attribute with the hub, then remediate once

    Pass status, Set-Cookie names, and body prefix into /solutions/identify-antibot-blocking-your-scraper. Choose residential vs browser-grade client from that matrix — do not invent a bypass from a TimeoutError string.

What goes wrong when this scraper fails

Engineers search for “why is my locator timing out” when the real event was a block document that the automation API treated as a successful navigation. Under-served queries cluster around Playwright and Selenium because both stacks optimise for “page loaded,” not “page is the product HTML you wanted.”

Challenges that make this hard to automate

  • No exception on HTTP deny documents
  • Empty TimeoutException messages in Selenium
  • Confusing proxy auth hangs with site blocks
  • Scrapy ignoring 403 by default looks like “no items”
  • Teams rotating IPs while the framework never classified the page
Four-stage pipeline: capture, classify, remediate, then re-assert the document.
Rotation without classification just samples more deny pages.

How Aethyn residential proxies help here

Use Aethyn when the pipeline’s Attribute stage says reputation/velocity/geo. Keep credentials in the Playwright proxy object or Selenium options with the published ports.

  • Premium HTTP 2099 for routine residential A/B after classification
  • Elite HTTP 5499 when Bot Management–class targets are confirmed
  • SOCKS5 1099 / 3499 for clients that prefer SOCKS
  • Sticky -session-…-lifetime-N when a browser flow must reuse one exit
  • Clear 407 vs site-block separation once proxy auth is configured correctly

Common questions about this scraper problem

Does Playwright throw when a site returns HTTP 403?
No. page.goto returns a Response whose .status is 403 and raises nothing. You must check status or title yourself.
Does Selenium raise on an Access Denied page?
driver.get typically raises nothing; driver.title may show Access Denied. Failures appear later as NoSuchElementException or empty-message timeouts.
Why did Playwright hang 30s with a proxy?
A proxy returning 407 without configured credentials can consume the full goto timeout without surfacing 407. Fix proxy auth; compare with ERR_PROXY_CONNECTION_FAILED for dead proxies.
Where do residential proxies fit in this pipeline?
After Classify/Attribute show an IP-shaped problem. Use Premium 2099 or Elite 5499 — they do not replace status checks in your framework.
Is Scrapy silent too?
By default Scrapy logs and ignores many 403s (HTTP status not handled), which looks like zero items. Enable explicit errbacks or handle 403 in the spider.
Does this guide teach CAPTCHA solving?
No. It teaches failure detection for lawful public-data pipelines.

Best practices that keep scrapers reliable

  • Log status + title on every navigation
  • Fail closed on deny titles before commerce waits
  • Configure proxy auth explicitly in Playwright
  • Keep a non-browser three-way curl in the incident kit
  • Public-data scope only — no CAPTCHA solve automation

Common mistakes that burn proxy budget

  • Reading TimeoutError as “site slow”
  • Ignoring response.status after goto
  • Assuming Selenium throws on HTTP 403
  • Rotating IPs before classification
  • Equating proxy 407 hangs with vendor blocks
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions