E-commercePremium tier

How to Scrape E-commerce Product Listings

Pulling a whole catalog — every title, price, variant, image, and stock status across thousands of products — is a different job from grabbing one price. You have to traverse category trees, survive three kinds of pagination, model variants correctly, and not get throttled halfway through. This guide covers doing it reliably and completely.

PKWritten by Priya Kapoor·Reviewed by Elena Vance·Updated September 4, 2026
12 min read
Editorial board contrasting a catalog truncated at page 5 with a complete crawl that reached the end sentinel.

Key takeaways

  • Completeness is the real metric: a crawl that silently stops at page 5 is worse than no crawl.
  • Prefer the embedded schema.org/Product JSON-LD over visual markup — it's structured and far more stable.
  • Identify which pagination style a store uses (numbered, load-more, infinite scroll) and detect the true end of each.
  • Model variants explicitly: size/color often carry their own price, SKU, and availability within one product page.
  • Refresh volatile fields (price, stock) far more often than static ones (title, description, images).

On this page

What goes wrong when this scraper fails

A comparison site, a merchandising analytics tool, or a brand monitoring its distribution all need the same thing: a complete, structured snapshot of many stores' catalogs — titles, prices, currencies, images, variants, and availability. E-commerce sites make this hard on purpose. They paginate deeply and inconsistently, localize price and stock by region, structure variants differently on every platform, and throttle repetitive automated access. The result for a naive crawler is partial catalogs (pagination quietly truncated), wrong-region data, and variant fields mashed together — data that looks fine until someone checks it against the live site.

Why this failure mode happens

Stores protect catalogs with rate limits and bot detection, and they localize price and availability by the visitor's region, so crawling from a few datacenter IPs both gets throttled and returns the wrong locale. On top of that, the front-end is built on platforms (Shopify, Magento, custom SPAs) that each express pagination and variants differently, so a parser tuned to one store breaks on the next — and a crawler that doesn't detect the genuine end of a category silently stops collecting.

Three pagination styles: numbered pages, load more, and infinite scroll, each with an end detector.
Prefer schema.org Product JSON-LD over painted markup when it exists.

Challenges that make this hard to automate

  • Traversing deep category trees and three different pagination styles
  • Detecting the true end of a listing rather than truncating early
  • Localized price, currency, and availability by region
  • Variant/option structures that differ on every platform
  • Anti-bot defenses triggered by repetitive, high-volume catalog access

Approaches that usually fail

  • Datacenter proxies — throttled mid-catalog and serving the wrong geography
  • A single-IP crawler — blocked partway through a large category
  • Platform APIs — not always available, complete, or covering competitors' stores
  • Manual exports — impossible to keep current across many stores

When residential proxies fix this — and when they cannot

Residential proxies make catalog crawls look like ordinary local shoppers and return the correct regional price and availability, while rotation across a large pool lets you traverse deep pagination and many categories without any single IP tripping a rate limit. That's what turns 'we crawled most of it' into a reliably complete catalog snapshot.

Parent product form beside size and color variant pills that each carry price and SKU.
Refresh price and stock more often than title and images.

How Aethyn residential proxies help here

Catalog crawling is high-volume, recurring, and geo-sensitive — the Premium tier's sweet spot, with the same endpoint scaling across many stores and regions.

  • Premium pool sized for cost-efficient, high-volume catalog crawls
  • Country targeting so localized price and availability are captured correctly
  • Per-request rotation to traverse deep pagination without throttling
  • Sticky sessions for cart- or zip-gated availability checks
  • Per-byte billing that keeps large, repeated catalog pulls predictable

How to implement this with residential proxies

  1. 1

    Map the category tree, then collect product URLs

    Start from the category structure (often in the nav or a sitemap.xml) so you crawl the catalog systematically rather than guessing URLs. For each category, collect product links with IPs rotating per request. Many stores expose a sitemap that lists every product URL directly — always check for it first; it's the cleanest, most complete source of URLs.

    Python (requests + parsing)
    import requests
    from bs4 import BeautifulSoup
    
    PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
    proxies = {"http": PROXY, "https": PROXY}
    
    def product_links(category_url):
        html = requests.get(category_url, proxies=proxies, timeout=30).text
        soup = BeautifulSoup(html, "html.parser")
        return [a["href"] for a in soup.select("a.product-card, a[href*='/product/']")]

    Field note: Check /sitemap.xml and /robots.txt before writing a crawler. A product sitemap gives you every URL without traversing pagination at all, and robots tells you what the site asks you not to crawl.

  2. 2

    Handle the store's pagination style — and detect the real end

    There are three common styles: numbered pages (?page=N), a 'load more' button (a background API call), and infinite scroll. Identify which the store uses. The critical part is detecting the genuine end: stop when a page yields no new products, not after a fixed count, or you'll silently truncate large categories.

    Python (pagination)
    def crawl_category(base, max_pages=200):
        seen, empty_streak = set(), 0
        for page in range(1, max_pages + 1):
            found = set(product_links(f"{base}?page={page}"))
            new = found - seen
            if not new:
                empty_streak += 1
                if empty_streak >= 2:      # two consecutive no-new-items pages = end
                    break
            else:
                empty_streak = 0
            seen |= new
        return seen

    Field note: For 'load more' and infinite scroll, open the Network tab — the button or scroll almost always calls a paginated JSON endpoint you can hit directly (with cursor/offset), which is faster and more reliable than driving a browser to click.

  3. 3

    Extract from schema.org/Product JSON-LD

    Most modern storefronts embed a schema.org Product block in JSON-LD for SEO. It gives you name, price, currency, availability, and often SKU in a structured form that survives visual redesigns. Parse that first and fall back to DOM selectors only for fields it omits.

    Python (JSON-LD product)
    import json
    from bs4 import BeautifulSoup
    
    def parse_product(html):
        soup = BeautifulSoup(html, "html.parser")
        for tag in soup.find_all("script", {"type": "application/ld+json"}):
            try:
                data = json.loads(tag.string)
            except (TypeError, json.JSONDecodeError):
                continue
            if isinstance(data, dict) and data.get("@type") == "Product":
                offers = data.get("offers", {})
                return {
                    "sku": data.get("sku"),
                    "name": data.get("name"),
                    "price": offers.get("price"),
                    "currency": offers.get("priceCurrency"),
                    "availability": offers.get("availability"),  # e.g. .../InStock
                }
        return None

    Field note: When offers is a list rather than a single object, that's the variant signal — each entry is usually a size/color with its own price and availability. Don't collapse it to one number; model the variants.

  4. 4

    Model variants explicitly and dedupe on SKU

    A single product page often represents many buyable variants (size, color), each with its own SKU, price, and stock. Store them as child records under the product, and dedupe everything on SKU/product id rather than title — names repeat and titles drift, but identifiers are stable.

    Field note: Capture the variant axes (size, color) as structured attributes, not just a concatenated label. 'Red / XL' as one string is unsearchable; {color: red, size: XL} lets you analyze price-by-size or stockouts-by-color later.

  5. 5

    Refresh incrementally by field volatility

    Re-crawl volatile fields (price, availability) on a frequent cadence and static fields (description, images, specs) rarely. Store deltas with timestamps so analytics can see stockouts and price moves over time, and so you're not re-downloading unchanged descriptions every run.

Best practices that keep scrapers reliable

  • Check sitemap.xml/robots.txt first — a product sitemap beats crawling pagination
  • Detect the true end of a listing; stop on no-new-items, not a fixed count
  • Parse schema.org/Product JSON-LD before falling back to DOM selectors
  • Model variants as structured child records and dedupe on SKU
  • Set country so localized price and availability are correct
  • Refresh volatile fields often, static fields rarely, and store deltas

Common mistakes that burn proxy budget

  • Stopping pagination after a fixed page count and truncating large categories
  • Scraping brittle visual markup instead of the embedded Product JSON-LD
  • Collapsing variants into one price/stock instead of modeling each SKU
  • Deduping on product name, merging distinct items that share a title
  • Ignoring localized currency and availability from a single vantage point
  • Crawling an entire catalog from one IP until it's throttled

Common questions about this scraper problem

How do I scrape an entire product catalog completely?
Start from the category tree or a product sitemap, collect product URLs, paginate until no new items appear (not a fixed count), then fetch and parse each product — all through rotating residential IPs. Completeness, not raw speed, is the metric: a crawl that silently truncates is the failure mode to design against.
What's the most reliable way to extract product fields?
Parse the schema.org/Product JSON-LD most storefronts embed for SEO — it gives name, price, currency, availability, and often SKU in a structured form that survives visual redesigns. Use DOM selectors only as a fallback for fields it omits.
How do I handle 'load more' and infinite scroll?
They almost always call a paginated JSON endpoint in the background (visible in the Network tab). Calling that endpoint directly with its cursor/offset is faster and far more robust than driving a headless browser to click or scroll.
How do I handle product variants?
Model them explicitly as child records of the product, each with its own SKU, price, and availability, and store the variant axes (size, color) as structured attributes rather than one concatenated label. When a product's JSON-LD offers is a list, that's your variant signal.
Do I need geo-targeting for e-commerce?
Yes wherever the store localizes price, currency, or availability. Set the country in the proxy username so you capture the region's actual data instead of a default locale.
How often should I re-crawl?
Refresh volatile fields (price, stock) frequently and static fields (descriptions, images, specs) rarely. Storing deltas with timestamps keeps freshness high and cost low.
Which tier should I use?
Premium is the cost-efficient default for most storefronts. Move specific high-security sites (large marketplaces) to Elite if they block Premium.
Is e-commerce scraping legal?
Collecting public product data is common practice, but respect each site's Terms of Service, avoid personal data, and consult counsel for your jurisdiction and use case.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions