ScrapingProxies

HTTP 200 Access Denied: Soft Blocks That Poison Scrapers

JS
Founder
Published
August 10, 2026
Updated
August 30, 2026
Read Time
4 min read
The status line is 200; the body is Access Denied

Your monitor is green. The status line says 200. The body says Access Denied — or “page unavailable” under Server: AkamaiNetStorage. Soft blocks are the quiet failure mode of modern scraping: they never trip raise_for_status(), they fill warehouses with junk, and they make “success rate” dashboards look healthy while the business data is gone.

Playbook with validators: HTTP 200 but the body is a block page.

Why edges serve soft denies

A readable deny document with a healthy status keeps browsers calm and dumb uptime checks quiet. Akamai-shaped properties may serve Access Denied HTML or NetStorage block assets without flipping the status line. Your framework reports success because transport succeeded, not because the catalog document arrived.

Transport succeeded; the catalog document did not
A readable deny keeps browsers calm and uptime checks quiet.

Related silence problem: Playwright did not tell you it was blocked.

What breaks in production pipelines

Soft blocks fail differently depending on where you look:

LayerWhat you see
HTTP clientstatus_code == 200, raise_for_status() passes
HTML parserMissing price nodes; empty result lists; odd titles
Metrics“Success rate” near 100%; downstream null rates climb
HumansSpot-check in a browser “works fine” (different client stack)

The last row is especially cruel: a colleague opens Chrome on a residential ISP and sees the real page, while your Python job stores Access Denied shells from a library TLS fingerprint. Without content asserts, the debate becomes “is the proxy bad?” when the real question is “did we define success?”

Client, parser, metrics, and a colleague's Chrome disagree
Client, parser, metrics, and a colleague's Chrome disagree — without content asserts the debate is the proxy.

Verified shapes (re-check live)

Dated 2026-08-09 probes (edges change — re-run before tickets):

ObservationTells
lowes.comHTTP 200 + Access Denied document title + entity-encoded Reference material
autotrader.comHTTP 200 + Server: AkamaiNetStorage + title like page unavailable + /akamai-block/ assets

Same class as hard Reference #18 denials — different status cosmetics.

Grep trap: 'Reference #' in resp.text can be False until html.unescape().

Minimum content gate

Code Snippet
import html, re

DENY_TITLE = re.compile(
    r"access denied|page unavailable|just a moment|attention required",
    re.I,
)

def content_ok(status: int, body: str, min_len: int = 2500) -> bool:
    if status != 200:
        return False
    text = html.unescape(body or "")
    if len(text) < min_len:
        return False
    m = re.search(r"<tit" + r"le[^>]*>(.*?)</tit" + r"le>", text, re.I | re.S)
    title = (m.group(1) if m else "").strip()
    if DENY_TITLE.search(title):
        return False
    if "/akamai-block/" in text.lower() or "px-captcha" in text.lower():
        return False
    return True

Calibrate min_len per URL class. A 200 that fails the gate is a soft block, full stop — count it as failure in cost-per-success math.

A 200 that fails the gate is a soft block
Title, length, and deny phrases. Fail the gate, count it as failure in cost-per-success.

Also log a structured tuple on every accepted write: status, Server, title, body length, and a 300-byte prefix. Soft blocks announce themselves in that tuple long before your JSON schema notices missing keys.

Fingerprint soft blocks vs reputation soft blocks

The lowes.com-shaped pattern (same machine, same IP: library client gets Access Denied HTML on 200; browser-grade TLS gets real HTML) is a fingerprint lesson. Rotating residential exits would not have changed the library outcome. Conversely, if only residential curl flips a deny title to a real catalog while plain and UA curls stay denied, the IP layer is implicated — keep Premium or Elite in the path and re-run asserts.

Same IP, library deny, browser HTML — that is fingerprint
Residential only if three-way curl flips the deny title. Sticky 1–1440 min, default 30, after asserts.

Avoid relying on unverifiable “success rates” from marketing pages. Measure content-verified accepts on your own basket.

Three-way probe: is IP the lever?

  1. Plain curl — log status, Server, title, length
  2. Chrome User-Agent curl
  3. Same through residential

Aethyn Premium HTTP proxy.aethyn.io:2099 (SOCKS5 1099) or Elite HTTP 5499 / SOCKS5 3499:

Code Snippet
PX='http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099'
UA='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36'
curl -sS -o /tmp/b3 -w "res %{http_code}\n" -x "$PX" -A "$UA" "https://TARGET/"
ResultMeaning
Soft shell only on direct egressReputation/geo — keep residential
Identical Access Denied on all threeFingerprint or always-on template — fix client
requests soft-deny, curl_cffi real HTML, same IPFingerprint — rotation would not have helped

What not to do

  • Alert only on non-2xx
  • Store every 200 HTML blob
  • Rotate Elite exits against an identical soft shell
  • Skip Server: AkamaiNetStorage as a tell
  • Grep for Reference # without html.unescape
  • Treat a colleague’s browser screenshot as proof your library client is fine

After asserts fail, classify the vendor with Identify which anti-bot blocked you, then remediate once. Soft shells often sit next to hard Reference #18 denials on the same property — different status cosmetics, same edge family.

Full assert recipes and dated examples: HTTP 200 but the body is a block page.

FAQ

Common questions about this article

Can HTTP 200 mean my scraper is blocked?
Yes. Soft blocks return deny or unavailable HTML with a 200 status. Assert titles, length, and required selectors — do not rely on status codes alone.
What public examples show 200 block bodies?
Documented 2026-08-09 observations include lowes.com (200 + Access Denied title with entity-encoded Reference material) and autotrader.com (200 + Server: AkamaiNetStorage + page unavailable + /akamai-block/ assets). Re-verify before citing — live edges drift.
Why does raise_for_status() pass on a block page?
Because transport returned 2xx. raise_for_status() is not a content validator.
Will a residential proxy fix a 200 Access Denied body?
Only when a three-way test shows the shell is IP-shaped. Fingerprint soft blocks stay denied across clean exits.
Share this article

Guides, integrations & docs

Continue reading

Assert content, then test the IP layer

Reject deny titles before storage. A/B Premium 2099 vs Elite 5499 only after your validator fails closed on soft shells.

Back to all articles