Web Scraping CoreElite tier

HTTP 200 but the Body Is a Block Page

Your monitor says green because the status line says 200 — meanwhile the body is Access Denied, “page unavailable,” or an empty catalog shell. Soft blocks poison datasets without tripping raise_for_status(). This page shows how to catch them with title, length, and selector asserts, using verified public examples.

TMWritten by Thomas Müller·Reviewed by Elena Vance·Last verified September 4, 2026
11 min read
Editorial board contrasting HTTP 200 with an Access Denied title against HTTP 200 with a real catalog.

Key takeaways

  • HTTP 200 is not acceptance — assert title, body length, and required DOM tokens on every accepted response.
  • lowes.com has been observed returning 200 with an Access Denied document title and an entity-encoded Reference string (re-check live; edges drift).
  • autotrader.com has been observed returning 200 from Server: AkamaiNetStorage with title “Autotrader - page unavailable” and assets under /akamai-block/.
  • raise_for_status() and Playwright navigation both treat these as success unless you assert content.
  • A three-way curl matrix still matters: fingerprint-shaped soft blocks may ignore residential rotation.
  • Never publish or trust a scrape that only checks status codes.

On this page

Signal observed → what it means for your scraper

Match the signal you captured, then decide whether a residential proxy can change that layer before you rotate.

Signal observedWhat it indicatesWhich layerDoes a proxy change it
200 + Access Denied titleShort HTML; title or TITLE tag says Access Denied; optional entity-encoded ReferenceEdge soft deny — status intentionally healthy while content is a block shellFail the row in your validator; decode HTML entities before grepping Reference; then identify the vendor via the hub matrix.
200 + page unavailable / AkamaiNetStorageServer: AkamaiNetStorage; title like page unavailable; /akamai-block/ asset pathsAkamai-hosted unavailable/block template served as a normal documentTreat as block. Do not store. Follow /solutions/fix-akamai-access-denied-scraping for Reference-shaped siblings.
200 + thin body / missing price nodeLength far below baseline; expected selectors absentSubstitute shell, consent wall, or challenge document without a hard statusBaseline length and must-have tokens per URL class; alert on drift.
200 identical across plain/UA/residentialAll three curls return the same block shellClient fingerprint or always-on soft template — not an IP-only gateStop rotating exits; upgrade client stack or accept the path is JS-gated.
200 flips after curl_cffi / browserrequests gets deny shell; browser-grade TLS gets real HTML on same IPFingerprint decision with a soft statusIP rotation would not have fixed it — change the client, keep sticky residential only after content asserts pass.
Grep misses Reference #('Reference #' in text) is False on a real Akamai shellEntity encoding (Reference#18…) in the raw byteshtml.unescape() before pattern match.
Assertion rows for title, body length, and a required catalog marker before parsing.
A short HTML shell with a deny title is a block even when status is 200.

How to diagnose and fix this scraper failure

  1. 1

    Log status, Server, title, and byte length together

    For one failing URL, capture status, Server, the HTML title element text, Content-Length or len(body), and a 300-byte prefix. Soft blocks announce themselves in that tuple long before your parser notices missing JSON keys.

  2. 2

    Reproduce the known soft-block shapes (dated observations)

    On 2026-08-09 probes: www.lowes.com returned HTTP 200 with an Access Denied TITLE and entity-encoded Reference material; www.autotrader.com returned 200 from Server: AkamaiNetStorage with an unavailable title and /akamai-block/ assets. Re-run before you cite either in a ticket — live edges change.

    Field note: If lowes returns real HTML today, keep the assert logic anyway — the failure class is what matters.

  3. 3

    Three-way probe: plain, Chrome UA, residential

    Compare the title and length across three transports. If only residential flips a deny title to a real catalog, IP was implicated. If all three share Access Denied or page unavailable, fix the client or accept the soft template.

    bash
    UA='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
    URL='https://TARGET/'
    PX='http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099'
    
    summarize() {
      local f="$1"
      python - "$f" <<'PY'
    import sys,re,html
    from pathlib import Path
    b=Path(sys.argv[1]).read_text(errors="ignore")
    t=re.search(r"<tit" + r"le[^>]*>(.*?)</tit" + r"le>", b, re.I|re.S)
    title=html.unescape(t.group(1)).strip() if t else "(no title)"
    print(f"len={len(b)} title={title[:80]!r}")
    print("access_denied", "access denied" in title.lower())
    print("akamai_block_path", "/akamai-block/" in b.lower())
    PY
    }
    
    curl -sS -D /tmp/h1 -o /tmp/b1 "$URL" -w "plain %{http_code}\n"
    summarize /tmp/b1
    curl -sS -D /tmp/h2 -o /tmp/b2 "$URL" -A "$UA" -w "ua %{http_code}\n"
    summarize /tmp/b2
    curl -sS -D /tmp/h3 -o /tmp/b3 "$URL" -x "$PX" -A "$UA" -w "res %{http_code}\n"
    summarize /tmp/b3
    grep -iE '^(HTTP/|server:)' /tmp/h1 /tmp/h2 /tmp/h3
  4. 4

    Install a content gate before storage

    Reject Access Denied titles, px-captcha meta, Just a Moment, page unavailable, and below-baseline lengths. Unescape HTML before Reference matching.

    Python
    import html, re
    from bs4 import BeautifulSoup  # optional; regex enough for titles
    
    DENY_TITLE = re.compile(
        r"access denied|page unavailable|just a moment|attention required",
        re.I,
    )
    
    def content_ok(status: int, body: str, min_len: int = 2500) -> bool:
        if status != 200:
            return False
        text = html.unescape(body or "")
        if len(text) < min_len:
            return False
        m = re.search(r"<tit" + r"le[^>]*>(.*?)</tit" + r"le>", text, re.I | re.S)
        title = (m.group(1) if m else "").strip()
        if DENY_TITLE.search(title):
            return False
        if "/akamai-block/" in text.lower() or "px-captcha" in text.lower():
            return False
        return True
  5. 5

    Only then decide if residential helps

    When the soft shell disappears on Elite/Premium residential but not on direct egress, keep country-targeted exits. When it does not, move to curl_cffi or Playwright and keep asserting content on every navigation response.

What goes wrong when this scraper fails

Collectors and uptime checks that key only on HTTP status will green-light poison. Retail and classified edges increasingly return deny or unavailable documents with 200 so naive clients keep writing rows. The failure mode is quiet: dashboards fill with Access Denied titles, empty product grids, or AkamaiNetStorage “unavailable” pages that never raise.

When residential proxies fix this — and when they cannot

Residential exits help when the soft shell is reputation- or geo-shaped and flips under a clean ASN. They do nothing when the same Access Denied title appears on plain curl, Chrome-UA curl, and residential curl alike — that pattern is fingerprint or always-on substitution. Use proxies to isolate the IP variable, not as a content validator.

Two browser panes comparing a soft deny shell with a packed product catalog.
Store body hash with status so silent 200s show up in the same log.

How Aethyn residential proxies help here

Run the same URL through Premium or Elite residential while your assert layer rejects deny titles. Sticky sessions keep a passing client on one exit once content checks are green.

  • Premium HTTP 2099 / SOCKS5 1099 for volume diagnostics
  • Elite HTTP 5499 / SOCKS5 3499 when Bot Management–class soft shells punish commodity ASNs
  • Country and ISP suffixes so geo matches the catalog you expect
  • Documented username form for reproducible three-way tests
  • No implication that a proxy turns a deny shell into HTML by itself

Best practices that keep scrapers reliable

  • Pair every status check with title + length + selector asserts
  • Unescape HTML before vendor string greps
  • Date-stamp live example captures
  • Keep three-way diagnostics in the runbook
  • Stay in public-data / Terms-safe collection — no CAPTCHA solving

Common mistakes that burn proxy budget

  • Trusting 200 alone
  • Grepping Reference # without html.unescape
  • Burning exits on an identical soft shell
  • Skipping Server: AkamaiNetStorage as a tell
  • Celebrating flaky real HTML without asserts

Common questions about this scraper problem

Can HTTP 200 still mean my scraper is blocked?
Yes. Soft blocks return deny or unavailable HTML with a 200 status. Assert titles and required content; do not rely on status codes alone.
What public examples show 200 block bodies?
Documented 2026-08-09 observations include lowes.com (200 + Access Denied title) and autotrader.com (200 + AkamaiNetStorage + page unavailable). Re-verify before citing — live behaviour drifts.
Will a residential proxy fix a 200 Access Denied body?
Only when the three-way test shows the shell is IP-shaped. Fingerprint soft blocks stay denied across clean exits.
Why does raise_for_status() pass on a block page?
Because transport returned 2xx. raise_for_status() is not a content validator.
Which Aethyn ports for soft-block A/B tests?
Premium HTTP 2099 or Elite HTTP 5499 (SOCKS5 1099 / 3499). Example: http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:5499.
Is this about bypassing CAPTCHAs?
No. It is about detecting poisoned 200 responses during lawful public-data collection and respecting site Terms.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions