SERP & SearchElite tier

How to Scrape Google Shopping Results Without Getting Blocked

Google Shopping results are merchant offers: who sells a product, at what price, in which currency, with what shipping or store-pickup promise. They are not organic search rankings. A classic Google Search SERP is a ranked list of pages. Collectors usually read it from the #rso container. Google Shopping is a separate surface — historically tbm=shop, increasingly the udm=28 Shopping UI — that hydrates an offer grid after JavaScript runs. Price-intelligence and technical SEO teams often discover the difference the hard way. A Python requests GET to google.com/search can return HTTP 200 and still contain a consent wall, a /sorry/ challenge, an empty carousel, or a classic SERP with no offer cards. Dashboards stay green. The stored “lowest price” describes nobody. Reliable collection is not a promise that you will never be blocked. It is a pipeline: pin a geo-matched residential exit, render the public Shopping page in a real browser, classify the document before you write a price, and keep country-catalog rows separate from city-local inventory. This guide is that collector. It does not bypass Google, solve CAPTCHAs, or unlock a merchant feed. For why the same SKU shows different merchants across metros, use the city-price-divergence protocol after this pipeline is in place.

AEWritten by Aethyn Editorial·Reviewed by J Shirisha·Updated September 18, 2026
16 min read
Editorial board contrasting a classic Google blue-link SERP with a Shopping offer grid of merchant, price, and shipping chips.

Key takeaways

  • Shopping is a rendered offer grid (tbm=shop / udm=28). Use Playwright plus a residential exit — not requests against a classic SERP.
  • Store merchant, product, price, currency, shipping, and pickup as Shopping fields. Do not merge them with organic #rso rank.
  • Country owns catalog, currency, and merchant eligibility. City owns local inventory, pickup, shipping ETA, and some sponsored slots.
  • Match country, gl, hl, and browser locale, then verify the resolved city. Contradictory geo signals produce an unreliable dataset.
  • HTTP 200 is not successful Shopping collection. Classify populated, empty_grid, consent, challenge, and wrong_geo before insert.
  • Rotate per independent SKU query. Sticky (30 min by default on the rotating pool) only while pagination or a store cookie must survive.

On this page

Google Shopping offer pipeline

  1. Pin the market

    Do: encode country on the username, match gl, hl, and Playwright locale, add Elite city when pickup or shipping is the metric, then verify the exit with an IP lookup. Why: Shopping localizes catalog by country and local offers by approximate location. Wrong: a country-de exit with gl=us, or a requested city the tunnel did not land in. Store: requested_country, requested_city, resolved_city, and the IP lookup timestamp.

  2. Launch a browser context

    Do: launch Playwright Chromium with a per-context residential proxy and credentials in username and password fields. Why: Shopping hydrates after JavaScript; stock HTTP clients see a SERP shell or fail TLS checks. Wrong: user:pass in the proxy URL (407), one context reused across SKUs (cookie leak), or requests instead of a browser. Store: user-agent/locale on the context, not a password.

  3. Render the offer grid

    Do: open the public Shopping surface, wait for the grid to hydrate, and read visible merchant, product, price, currency, and shipping or pickup text. Why: generated class names are not a parser contract; entry chrome (tbm=shop / udm=28) drifts. Wrong: parsing #rso as offers, or treating a still-loading page as empty. Store: raw visible fields only after classification says populated.

  4. Classify the document

    Do: label the HTML as populated, empty_grid, consent, challenge, or wrong_geo before any insert. Why: HTTP 200 is not successful Shopping collection — those five documents can share a 200. Wrong: writing merchants=[] for a consent wall or a /sorry/ page. Store: status, and persist prices only when status is populated.

  5. Key the offers

    Do: write query, geo keys, merchant, product, price, currency, shipping, pickup, status, and collected_at. Why: country-catalog FX and city-local inventory are different series; mixing them poisons both. Wrong: one sticky IP for a huge SKU list, or merging Shopping rows into a SERP table. Store: country rows and city rows in separate series.

Four-stage pipeline: pin market, launch Playwright with user and pass fields, render offer cards, classify populated vs empty vs challenge.
Credentials go in username and password fields. Classify the document before you write a price.

How to scrape Google Shopping results with Playwright

  1. 1

    Pin the market and make locale agree

    Start by naming the object. Country targeting (-country-us) owns catalog, currency, and merchant eligibility. City targeting (-city-chicago, Elite only) owns local inventory, pickup, shipping ETA, and some sponsored slots. ISP targeting (-isp-, Elite only) is for carrier-specific research and is usually unnecessary for standard Shopping collection. Then make every geographic signal agree: exit country, gl, hl, and Playwright locale. A country-de exit with gl=us, or an en-US locale on a DE session, is a contradictory request and an unreliable dataset. Add Elite city when pickup or shipping is the metric, then verify the city you landed in with an IP lookup before the first Shopping request. Asking for a city is not landing there — city and ISP suffixes resolve to the closest available match and can fall back to country. Store requested_city next to resolved_city. Stay on Premium (HTTP 2099 / SOCKS5 1099) when the job is national catalog and FX only; Premium refuses a city suffix rather than ignore it. Do not mix those rows into a city series.

    text
    Requirement                    Premium (-country-)     Elite city (-city-)      Elite ISP (-isp-)
    ------------------------------  ----------------------  -----------------------  -------------------------
    National catalog / FX           Yes                     Yes                      Yes (usually overkill)
    City-specific Shopping results  No                      Yes                      Yes
    Local inventory                 No                      Yes                      Yes
    Store pickup                    No                      Yes                      Yes
    Shipping / location testing     No                      Yes                      Yes
    ISP / carrier-specific research No                      No                       Yes
    Standard Shopping collection    Yes                     Yes                      Usually unnecessary
    Sticky pagination / session     Available               Available                Available
    cURL
    curl -x "http://aethyn-XXXXX-country-us-city-chicago:PASSWORD@proxy.aethyn.io:5499" \
      "https://ipinfo.io/json"

    Field note: cURL here is only to prove the exit. Do not parse Shopping offers from this response — the offer grid will not be in it. Record resolved_city from the lookup, not from the username you sent.

  2. 2

    Run a small end-to-end pass: query, proxy, verify, classify, extract

    Put credentials in username and password fields. Chromium strips user:pass from a proxy URL and answers 407. Launch with Playwright’s per-context placeholder, then set the real proxy on each newContext. One context per independent SKU query so cookies never leak onto the next exit. The pass is: encode the query, open a geo-matched residential context, verify the exit, navigate to the public Shopping page, classify the document, extract visible offer fields only if the grid is populated, then write a structured record. Entry chrome drifts between tbm=shop and udm=28 — keep a small adapter around navigation. Generated class names are not a parser contract; prefer visible merchant, product, price, currency, and shipping or pickup text. This is public-page collection at a human-like rate, not a login, CAPTCHA, or checkout bypass.

    Python (Playwright)
    from datetime import datetime, timezone
    import json
    from playwright.sync_api import sync_playwright
    
    QUERY = "wireless earbuds"
    REQUESTED_CITY = "chicago"
    PROXY = {
        "server": "http://proxy.aethyn.io:5499",
        "username": "aethyn-XXXXX-country-us-city-chicago",
        "password": "PASSWORD",
    }
    CHALLENGE = ("unusual traffic", "sorry/index", "verify you are human")
    CONSENT = ("before you continue", "consent.google.com")
    
    def classify(html: str, n_offers: int, geo_ok: bool) -> str:
        if not geo_ok:
            return "wrong_geo"
        low = html.lower()
        if any(m in low for m in CHALLENGE):
            return "challenge"
        if any(m in low for m in CONSENT):
            return "consent"
        if n_offers:
            return "populated"
        return "empty_grid"
    
    def extract_offers(page):
        # Generated class names drift. Prefer visible price-like text, then cap the sample.
        rows = []
        for loc in page.get_by_text("$").all()[:12]:
            text = " ".join((loc.inner_text() or "").split())
            if text:
                rows.append({"raw": text[:240]})
        return rows
    
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy={"server": "per-context"}, headless=True)
        context = browser.new_context(proxy=PROXY, locale="en-US")
        page = context.new_page()
        page.goto("https://ipinfo.io/json", wait_until="domcontentloaded")
        geo = json.loads(page.inner_text("body"))
        resolved_city = (geo.get("city") or "").strip()
        geo_ok = REQUESTED_CITY.lower() in resolved_city.lower()
        page.goto(
            "https://www.google.com/search?q=wireless+earbuds&tbm=shop&gl=us&hl=en",
            wait_until="domcontentloaded",
        )
        page.wait_for_timeout(2500)  # grid hydrate; tbm=shop / udm=28 chrome drifts
        html = page.content()
        offers = extract_offers(page) if geo_ok else []
        status = classify(html, len(offers), geo_ok)
        record = {
            "query": QUERY,
            "requested_city": REQUESTED_CITY,
            "resolved_city": resolved_city,
            "status": status,
            "offers": offers if status == "populated" else [],
            "collected_at": datetime.now(timezone.utc).isoformat(),
        }
        print(json.dumps(record, indent=2))
        context.close()
        browser.close()

    Field note: If per-context proxies are ignored, you launched Chromium without the placeholder. Launch proxy: { server: 'per-context' }, then set the real gateway on newContext(). Adapt the extractor when Google’s chrome changes; keep classification and geo keys stable.

  3. 3

    Parse merchant, price, currency, shipping — not organic rank

    Wait for offer cards to settle before you extract. Stable enough signals are a product title, a merchant name, a visible price with a currency, and a shipping or pickup line. Do not store organic #rso links as offers. If the page is a classic SERP with a shopping module, that is a different object — collect it with the Google Search how-to and do not merge the rows. Fail closed: if you cannot name merchant and price on a populated grid, do not invent them from nearby SERP chrome.

    Python (record fields)
    offer = {
        "query": "wireless earbuds",
        "merchant": "Loop Audio",
        "product": "Loop Audio buds",
        "price": "79.00",
        "currency": "USD",
        "shipping": "Free delivery",
        "pickup": None,  # or a store-pickup chip when the card shows one
    }

    Field note: Entry URLs for Shopping drift (tbm=shop, udm=28, consent). Keep a small adapter around navigation; keep classification and offer keying stable.

  4. 4

    Classify empty grids before you store them

    A 200 with zero offer cards is not automatically “this SKU has no merchants.” Distinguish five documents: a populated offer grid; a genuine empty grid (the UI loaded and attached no cards for this query and IP); a consent wall; a challenge or /sorry/ interstitial; and an invalid or wrong geographic exit. Those five outcomes must not share one empty array. Fail closed: if you cannot name the document, do not persist a price. HTTP 200 is not successful Shopping collection.

    Python (record shape)
    record = {
        "query": "wireless earbuds",
        "requested_country": "us",
        "requested_city": "chicago",
        "resolved_city": "Chicago",  # from IP lookup at session start
        "collected_at": "2026-09-18T12:00:00Z",
        "status": "populated",  # populated | empty_grid | consent | challenge | wrong_geo
        "merchant": "Loop Audio",
        "price": "79.00",
        "currency": "USD",
        "shipping": "pickup",
    }
    
    # Never write empty_grid, consent, challenge, or wrong_geo as merchants=[].
    # Write status=... and omit prices.

    Field note: Assert resolved_city (and country) before the first Shopping request. A mis-typed username is a silent wrong market, not a block. IP geolocation is a sanity check, not GPS.

  5. 5

    Rotate per SKU; sticky only for pagination and store cookies

    Rotate: close the Playwright context when the independent SKU query is done so the next query gets a new household exit. Use a sticky session only while you page the same grid, while a store-selection cookie must stay on the IP that earned it, or while a consent cookie must survive the next render — typically well inside the default 30-minute window. Terminate the sticky session when that flow ends: drop the session id, close the context, and rotate again. Do not pin one sticky IP for a 5,000-SKU morning run. Sticky holds a rotating-pool residential IP for 1–1440 minutes (default 30). It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

    Python (sticky username)
    # Same rotating-pool IP for 30 minutes (lifetime-30) while a store cookie lives
    user = "aethyn-XXXXX-country-us-city-chicago-session-shop42-lifetime-30"
    proxy = {
        "server": "http://proxy.aethyn.io:5499",
        "username": user,
        "password": "PASSWORD",
    }
    # When pagination / store selection ends: drop -session-…-lifetime-N and close the context.

    Field note: Sticky holds a rotating-pool residential IP. It is not a static, dedicated, or ISP product. Elite ISP suffixes are targeting, not a different SKU.

What are you actually scraping?

A Google Search SERP is a ranked list of organic results. Collectors usually parse the #rso container for title, URL, and snippet. Google Shopping is a different product. The unit of a Shopping page is an offer card: a merchant, a product title, a price, a currency, and often a shipping line or a store-pickup chip. Those fields describe commerce, not rank. Mixing them with organic SERP rows is how price dashboards start reporting a blue link as a seller. Store Shopping offers in their own table. Keep country-catalog rows — currency, eligible merchants, national price — separate from city-local rows — pickup, shipping ETA, local inventory. The same HTTP 200 can carry a populated grid, a genuine empty carousel, a consent wall, a /sorry/ challenge, or a session that resolved in the wrong city. Classify the document before you persist a price. HTTP 200 is not successful Shopping collection.

What breaks Google Shopping collection

  • Shopping is a rendered offer grid — there is no stable public JSON feed to poll
  • Entry URLs drift between tbm=shop and udm=28; generated class names are not a parser contract
  • Country and city are different variables; mixing them in one series poisons FX and local inventory alike
  • gl, hl, browser locale, requested city, and resolved city can disagree; contradictory geo signals produce an unreliable dataset
  • Soft 200s (empty carousel, challenge, consent, wrong geo) look like “no merchants” if you do not classify them
  • Consent walls and store cookies bind to the IP and browser that earned them
Session rail holding one rotating-pool residential IP for query, store cookie, and page two, then rotating for the next SKU.
Sticky holds a rotating-pool IP for 30 minutes by default, up to 24 hours. It is not a dedicated address.

How Aethyn Elite fits Google Shopping collection

Use Premium when the job is national catalog, currency, and merchant eligibility. Use Elite when pickup, shipping, local inventory, or a sponsored mix that changes by metro is the object — city targeting lives on Elite, and Premium refuses a city suffix rather than silently ignore it. ISP targeting (-isp-) is Elite-only and usually unnecessary for Shopping unless the carrier is the research variable. Sticky sessions help only while pagination or a store cookie must survive; rotate again when that flow ends. HTTP CONNECT is the usual Playwright path; SOCKS5 if the client requires it. Same residential pool either way. A proxy does not skip Google’s terms, robots.txt, or rate limits.

  • Premium HTTP 2099 / SOCKS5 1099 for country catalog and FX — encode -country-XX; do not put a city on this username
  • Elite HTTP 5499 / SOCKS5 3499 with -country-XX-city- when pickup, shipping, local inventory, or metro-varying offers are the metric
  • Elite -isp- only when the carrier or network is the variable; skip it for standard Shopping collection
  • Per-request rotation by omitting a session id so each independent SKU query gets a fresh household exit
  • Sticky 1–1440 minutes (default 30) as -session-…-lifetime-N only while pagination or a store cookie lives, then drop the session id

Google Shopping scraping FAQ

Is Google Shopping the same as Google Search?
No. Google Search returns organic blue links (and sometimes a shopping module) that collectors usually read from #rso. Google Shopping is a separate offer grid — historically tbm=shop, often udm=28 — whose unit is merchant, product, price, currency, and shipping or pickup. Store them as different objects. Rank tracking belongs on the Google Search how-to; this page is Shopping offers.
Can I scrape Google Shopping with Python requests?
Not usefully. Shopping is a rendered UI. requests sees a SERP shell, a consent wall, or a challenge, fails TLS fingerprint checks on defended queries, and will not hydrate offer cards. Use Playwright through a residential proxy so the browser supplies a real TLS stack and can wait for the grid. Pace requests; a proxy does not replace rate limits.
Why Elite instead of Premium?
Premium (HTTP 2099 / SOCKS5 1099) is the right pool for national catalog, currency, and merchant eligibility. Elite (HTTP 5499 / SOCKS5 3499) is the default when pickup, shipping, local inventory, or a sponsored mix that changes by metro is the object — city targeting lives there, and Premium refuses a city suffix rather than ignore it. ISP targeting is a separate Elite suffix and is usually unnecessary for Shopping.
Do I always need city targeting?
No. Country targeting is enough when you can name only catalog, FX, or merchant eligibility as the mechanism. Add Elite city when you can name pickup, shipping ETA, or local inventory. Asking for a city is not landing there — city suffixes can fall back to country. Verify via an IP lookup and record requested_city next to resolved_city.
What should I store when the offer grid is empty?
Store a status, not a fake empty merchant list. Use challenge or consent when you see those documents, wrong_geo when the exit missed the requested city, and empty_grid when the UI loaded but attached no cards for this query and IP. Do not write merchants=[] unless you actually rendered a populated-shape grid that listed none. Fail closed on anything you cannot name.
Should I rotate or use a sticky session?
Rotate: one household exit per independent SKU query. Sticky (1–1440 minutes, default 30) only while you page the same grid or while a store or consent cookie must stay on the IP that earned it. Terminate the sticky session when that flow ends — drop the session id and close the Playwright context. Do not pin one IP for a large SKU list. This is not a static IP product.
Is this a merchant-feed unlocker API?
No. This is a proxy-layer collector for the public Shopping UI. It does not claim to bypass Google, solve challenges, reproduce a vendor Shopping API, or replace Google’s terms. Pair it with the city-price-divergence protocol when you need a citable method for why offers differ between cities, rather than this how-to.
HTTP or SOCKS5?
Same residential pool. HTTP CONNECT on Premium 2099 or Elite 5499 is the usual Playwright path. SOCKS5 on Premium 1099 or Elite 3499 if the client requires it. There is no second residential SKU. Put credentials in username and password fields; do not embed user:pass in the proxy URL.

Best practices for Shopping offer pipelines

  • Collect public Shopping offer cards only. Do not log in, do not solve CAPTCHAs, and do not treat a proxy as a license to ignore Google’s terms, robots.txt, or rate limits.
  • Match exit country, gl, hl, and Playwright locale on every context. Contradictory locale is both a bad dataset and a bot tell.
  • Verify the exit with an IP lookup at the start of the context and store resolved_city next to requested_city and the price.
  • Keep country-catalog series and city-local series in different tables. Mixing them is how national FX pretends to be store inventory.
  • Classify populated vs empty_grid vs consent vs challenge vs wrong_geo before insert. Empty-as-absent is how price dashboards lie.
  • One Playwright context per independent SKU query. Reusing a context across queries leaks cookies onto a new identity.
  • Reserve sticky (1–1440 minutes, default 30) for pagination and the identity that holds a store or consent cookie; terminate the session when that flow ends.
  • Keep classic SERP collection on its own pipeline. Merging #rso HTML into a Shopping table poisons both products.

Common mistakes when scraping Google Shopping

  • Using requests or httpx and parsing a SERP as if it were Shopping offers.
  • Reporting a US datacenter crawl as a DE (or any other) market’s catalog.
  • Sending gl/hl/locale that contradict the exit country, then treating the mix as a valid national catalog.
  • Storing a 200 with zero cards as merchants=[] instead of status=empty_grid, consent, challenge, or wrong_geo.
  • Embedding user:pass in the Playwright proxy server URL and debugging the resulting 407 as a Google block.
  • Rotating the IP while a store cookie is still required, then calling the next empty page “out of stock.”
  • Pinning one sticky IP for the entire SKU list and recreating a per-IP velocity problem.
  • Mixing city-local pickup rows into a national FX series.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions