SERP & SearchElite tier

How to Scrape Google Search Results Without Getting Blocked

Google has no official organic-results API, so rank trackers and SERP tools have to query the public pages directly — and Google is one of the most aggressive anti-automation surfaces on the web. Here is how to collect accurate, localized SERP data at volume without drowning in 'unusual traffic' pages.

MTWritten by Marcus Thorne·Reviewed by Sarah Chen·Updated November 21, 2026
14 min read
Editorial board contrasting a Google /sorry/ interstitial with a real SERP that contains #rso result rows.

Key takeaways

  • Google blocks on reputation + velocity + fingerprint, not just request count — fixing one signal alone rarely helps.
  • The reliable pattern is low per-IP velocity across a large residential pool, not a handful of fast IPs.
  • Match the proxy country to the gl/hl parameters, or you will collect SERPs that no real user actually sees.
  • &num=100 stopped working reliably in 2024 — paginate with start=0,10,20 and expect ~10 results per page.
  • Treat the /sorry/index 'unusual traffic' page and consent.google.com redirect as throttle signals: slow down and rotate, never retry harder.

On this page

What scraping Google search results at scale involves

A rank-tracking product might need to check 50,000 keywords across 12 countries every morning, and a competitor-research dashboard might fan out hundreds of related queries on demand. Both hit the same wall: Google returns the '/sorry/index' interstitial ('Our systems have detected unusual traffic from your computer network'), a reCAPTCHA, or — more insidiously — a 200 OK with a stripped-down or consent-walled page that your parser silently records as 'zero results.' The data quietly rots before anyone notices the dashboards are wrong.

Why Google blocks scrapers after a few queries

Google does not score requests on volume alone. It combines IP reputation (datacenter ASNs are pre-discounted and shared across thousands of scrapers), request velocity per IP and subnet, and client fingerprint — the TLS/JA3 handshake and HTTP/2 frame ordering of libraries like requests or Go's net/http look nothing like Chrome's. Add an inconsistent header set (a desktop User-Agent with no Accept-Language, no sec-ch-ua hints) and you have signed your request as a bot. Once an IP trips the threshold, Google throttles the whole exit, which is why one greedy worker can poison an entire datacenter range for everyone behind it.

Two query panes: pizza shop versus pizza shop Austin with a local pack.
A city on the query plus a matching exit is how the pack appears.

What breaks Google SERP scraping

  • The 'unusual traffic' interstitial and reCAPTCHA appear after surprisingly few queries from one IP
  • Soft blocks return HTTP 200 with consent walls or thin pages, corrupting data without raising an error
  • Results are localized by IP geography and personalized by gl/hl — a US IP gives a US SERP regardless of intent
  • Result containers use churned, generated class names, so selectors break every few weeks
  • &num=100 is no longer honored, so deep result sets require start= pagination and more requests

Why datacenter pools and fixed delays fail on Google

  • A small datacenter proxy pool — cheap, but the ASNs are flagged and the whole subnet throttles together
  • Fixed time.sleep() delays — cut throughput without changing the IP reputation that actually triggers blocks
  • Paid SERP APIs (SerpApi, DataForSEO) — fast to start, but pricey at six-figure query volumes and opaque about freshness/locale
  • Manual incognito checks — accurate for spot checks but biased by your own IP and impossible to scale or schedule

When residential exits fix Google blocks, and when they cannot

Residential IPs carry the trust of real consumer connections, so an individual query reads as an ordinary search rather than one of a thousand from a server farm. The leverage is not any single 'magic' IP — it is the pool size. Spreading 50,000 queries across tens of thousands of rotating residential exits keeps each IP's velocity near what a human would generate, which is the signal Google actually penalizes. Country targeting then guarantees the SERP you parse is the one a searcher in that market sees, not a US default.

Session rail rotating a new residential exit on every Google query.
Sticky sessions are for a single SERP walk. New query, new exit.

How Aethyn Elite fits Google SERP collection

Google rewards low velocity and correct geography, and punishes predictable fingerprints. Aethyn maps onto exactly those needs — everything below is controlled through the proxy username on a single endpoint, so a multi-country tracker is one credential, not twelve gateways.

  • A large, diverse residential pool so a 50k-keyword run spreads thin — the velocity-per-IP that keeps SERPs flowing
  • Country targeting (-country-XX) that aligns the exit IP with your gl/hl so localized rankings are accurate
  • Elite city/ISP targeting (-city-) for local-intent SERPs where rankings shift block by block
  • Per-request rotation by default so consecutive queries never stack on one IP
  • Sticky sessions (-session-…-lifetime-, 30 min by default, up to 24 h) for the rare case where you page a single keyword from one exit

How to scrape Google search results without getting blocked

  1. 1

    Pick the right tier and pin the locale

    Google is a high-security target, so start on the Elite residential pool (port 5499). Pin the country with -country-XX in the username and set the matching gl (country) and hl (language) query parameters. These two must agree: a -country-de exit with gl=us produces a contradictory SERP that reflects no real searcher and is easy for Google to flag as inconsistent.

    cURL
    curl -x "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499" \
      "https://www.google.com/search?q=residential+proxies&hl=de&gl=de"

    Field note: For local-intent keywords ("plumber near me", "coffee"), country alone is too coarse — rankings change by city. Use the Elite tier with -city- targeting so the SERP reflects the neighborhood you are actually reporting on.

  2. 2

    Send queries through rotating residential IPs (Python)

    Rotate on every request so consecutive queries never stack on one IP. Send a realistic, complete header set — a bare User-Agent with no Accept-Language is a classic bot tell. requests reads the same proxy URL for HTTP and HTTPS. Note the start parameter: since &num=100 stopped being honored, you collect deep results by stepping start in increments of 10.

    Python (requests)
    import requests
    
    PROXY = "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:5499"
    proxies = {"http": PROXY, "https": PROXY}
    
    HEADERS = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                      "(KHTML, like Gecko) Chrome/124.0 Safari/537.36",
        "Accept-Language": "de-DE,de;q=0.9,en;q=0.6",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    }
    
    def fetch_page(query, start=0, gl="de", hl="de"):
        params = {"q": query, "gl": gl, "hl": hl, "start": start}
        return requests.get("https://www.google.com/search",
                            params=params, headers=HEADERS, proxies=proxies, timeout=30)
    
    resp = fetch_page("residential proxies")
    print(resp.status_code, len(resp.text))

    Field note: Plain HTTP clients also leak a non-browser TLS/JA3 fingerprint. If you still see challenges after fixing headers and velocity, switch to a client that mimics a browser handshake (e.g. curl_cffi with impersonate="chrome") or move to the Playwright approach below — the residential IP fixes reputation, the browser fixes the fingerprint.

  3. 3

    Parse organic results defensively

    Google's result markup uses generated, frequently-rotated class names, so never anchor your parser on them. The stable structure is: the results live under div#search > div#rso, each organic result is an <a> whose href is a real destination and which contains an <h3> title. Anchoring on that survives most layout churn, and you should always guard against the soft-block case where #search is missing entirely.

    Python (BeautifulSoup)
    from bs4 import BeautifulSoup
    
    def parse_serp(html):
        soup = BeautifulSoup(html, "html.parser")
        rso = soup.select_one("div#search div#rso")
        if rso is None:
            raise BlockedError("No #rso container — likely a consent wall or soft block")
        results = []
        for h3 in rso.select("a > h3"):
            link = h3.find_parent("a")
            href = link.get("href", "")
            if href.startswith("http"):
                results.append({"title": h3.get_text(strip=True), "url": href})
        return results

    Field note: Always assert a non-empty result set before writing to your database. A 200 OK with zero parsed results is almost never a keyword with no rankings — it is a soft block, and silently storing it is how rank-tracking dashboards end up quietly wrong for weeks.

  4. 4

    Use a sticky session only when paging one keyword

    Per-request rotation is the right default. The exception is paging deep results for a single keyword (start=0,10,20…): switching IPs mid-keyword can yield slightly different result sets, so pin one exit for that short burst with a session id and lifetime, then drop back to rotation for the next keyword.

    Python (sticky)
    # Same exit IP for ~10 minutes while paging one keyword
    def sticky_proxy(keyword_id):
        user = f"aethyn-XXXXX-country-de-session-kw{keyword_id}-lifetime-10"
        url = f"http://{user}:PASSWORD@proxy.aethyn.io:5499"
        return {"http": url, "https": url}

    Field note: Keep the lifetime as short as the job needs (a couple of minutes is plenty for three pages). A long-lived sticky session is just a slow way to over-use one IP — the opposite of what keeps SERP collection healthy.

  5. 5

    Fall back to a real browser for consent walls (Playwright)

    In the EU, Google often redirects to consent.google.com before showing results. A headless browser through the same residential proxy can accept consent and render the SERP exactly as a real visit would — and it carries a genuine browser fingerprint, sidestepping the TLS/JA3 problem of plain HTTP clients. Reserve this for queries that get walled, since a browser costs far more per request than requests.

    Python (Playwright)
    from playwright.sync_api import sync_playwright
    
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy={
            "server": "http://proxy.aethyn.io:5499",
            "username": "aethyn-XXXXX-country-de",
            "password": "PASSWORD",
        })
        page = browser.new_page(locale="de-DE")
        page.goto("https://www.google.com/search?q=residential+proxies&gl=de&hl=de")
        # Accept the EU consent dialog if it appears
        btn = page.query_selector("button:has-text('Accept all'), button:has-text('Alle akzeptieren')")
        if btn:
            btn.click()
        page.wait_for_selector("div#search", timeout=15000)
        html = page.content()
        browser.close()

    Field note: Set the browser locale to match the proxy country (locale="de-DE" with a German exit). A German IP rendering a US-English browser is an obvious mismatch and a soft signal that you are automating.

  6. 6

    Treat blocks as a throttle signal, not a retry loop

    Detect the block states explicitly — HTTP 429, a redirect to /sorry/index, or a missing #rso — and respond by slowing down and rotating, never by retrying immediately on the same path. Exponential backoff with jitter plus per-request rotation lets the next attempt leave from a fresh IP after the pressure has eased. If 429s spike across many IPs at once, that is your system telling you global concurrency is too high.

    Python (block detection + backoff)
    import time, random
    
    class BlockedError(Exception):
        pass
    
    def is_blocked(resp):
        if resp.status_code == 429:
            return True
        if "/sorry/index" in resp.url or "consent.google.com" in resp.url:
            return True
        return "detected unusual traffic" in resp.text
    
    def fetch_with_retry(query, attempts=4):
        for i in range(attempts):
            resp = fetch_page(query)            # rotates IP each call
            if not is_blocked(resp):
                return parse_serp(resp.text)
            time.sleep(min(60, 2 ** i) + random.random())  # back off + jitter
        raise BlockedError(f"Still blocked after {attempts} attempts: {query}")

    Field note: Track your block rate as a first-class metric. A healthy SERP pipeline sits near zero; a slow climb usually means your concurrency crept up or a parser change is misreading good pages as blocks. Alert on it before the data silently degrades.

Best practices for Google SERP scrapers

  • Spread volume across the pool — a wide, slow fan-out beats a few fast IPs every time
  • Keep -country aligned with gl/hl, and the browser locale aligned with both
  • Send a full, consistent header set (UA + Accept-Language + Accept), not just a User-Agent
  • Anchor parsers on stable structure (#search > #rso, a > h3), never on generated class names
  • Validate a non-empty result set before persisting, and monitor block rate as a metric
  • Cache SERPs and re-check on a cadence (daily/weekly) instead of re-querying constantly
  • Stay within Google's Terms of Service and collect only public, non-personal data

Mistakes that trigger Google's unusual-traffic page

  • Storing a 200 OK with zero parsed results as 'no rankings' — it is almost always a soft block
  • Mismatching exit country and gl/hl, collecting a SERP no real user in that market sees
  • Sending a desktop User-Agent with no Accept-Language — an instant bot tell
  • Retrying on the same IP after a 429 instead of backing off and rotating
  • Assuming &num=100 still works; deep result sets need start= pagination
  • Hard-coding today's CSS class names and re-breaking every Google layout tweak

Google SERP scraping questions

Does Google have an official API for organic search results?
No. The Custom Search JSON API returns a restricted, often different result set scoped to a configured engine, not the live organic SERP. That gap is exactly why rank trackers query the public pages directly — and why anti-automation handling matters.
Is scraping Google search results legal?
Collecting publicly available search data is widely done for rank tracking and research, but it can conflict with Google's Terms of Service. Scrape only public, non-personal data, keep volume reasonable, and consult counsel for your specific use case and jurisdiction.
Why do I still get CAPTCHAs even through residential proxies?
A trustworthy IP fixes only the reputation signal. If velocity is still too high, headers are incomplete, or your HTTP client has a non-browser TLS/JA3 fingerprint, Google can still challenge you. Lower concurrency, send a full header set, and for stubborn cases use a browser engine or a TLS-impersonating client.
What is the '/sorry/index' page?
It is Google's 'Our systems have detected unusual traffic' interstitial, served when an IP or network trips a velocity/reputation threshold. Treat it as a hard signal to slow down and rotate — retrying immediately just deepens the throttle.
How many queries can one IP safely make?
There is no fixed number; it varies by locale, query type, and timing. Chasing a per-IP limit is the wrong mental model — keep per-IP velocity near human levels and scale throughput by widening the pool instead.
Rotating or sticky sessions for SERP scraping?
Rotate per request by default. Use a short sticky session only while paging a single keyword (start=0,10,20), where a consistent exit avoids subtle result-set drift between pages.
Why does &num=100 no longer return 100 results?
Google stopped reliably honoring the num parameter in 2024. Collect deep results by paginating with the start parameter in increments of 10, expecting roughly ten organic results per page.
How do I get accurate results for a specific city?
Use the Elite tier and target a city in the username (for example -city-newyork). City-level exits matter for local-intent queries where rankings differ block by block, which country targeting cannot capture.
Should I use raw HTTP requests or a headless browser?
Use requests for the bulk of queries — it is far cheaper. Fall back to Playwright/Puppeteer through the same proxy for queries that hit a consent wall or need JavaScript, since a real browser also carries a genuine fingerprint.
How do I keep my parser from breaking constantly?
Stop anchoring on Google's generated class names. Target the stable structure (div#search > div#rso, organic links as a > h3) and guard for a missing container so a layout change degrades gracefully instead of silently recording empty SERPs.

References

Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions