E-commercePremium tier

How to Monitor Shopify Stores

Most people scrape Shopify stores the hard way — rendering pages and parsing HTML — when almost every Shopify store exposes a clean, paginated JSON feed of its entire catalog. This guide shows how to use it (and what to do when a store locks it down), all through residential proxies so recurring monitoring doesn't get throttled.

SLWritten by Sophie Laurent·Reviewed by Alex Rivera·Updated September 4, 2026
12 min read
Editorial board contrasting brittle Shopify theme HTML with the paginated /products.json catalog feed.

Key takeaways

  • Nearly every Shopify store serves its full catalog at /products.json — paginated, structured, no HTML parsing.
  • Prices in the JSON are strings in the store's currency; variants carry their own price, SKU, and availability.
  • Use ?limit=250&page=N and stop when a page returns an empty products array.
  • Some stores disable /products.json — fall back to /collections/<handle>/products.json or the product .js endpoint.
  • It's light infrastructure, but recurring polling from one IP still gets rate-limited; rotate residential IPs.

On this page

What goes wrong when this scraper fails

A brand wants to watch competitors' Shopify catalogs — new products, price changes, what's selling out — across dozens of stores, refreshed daily. The instinct is to build a browser-based scraper per store and fight each theme's markup. That's slow, brittle, and overkill, because Shopify ships a standardized, public JSON API on every store. The real task is using that feed efficiently, handling the minority of stores that restrict it, and polling reliably without getting rate-limited from a single IP.

Why this failure mode happens

Shopify is a hosted platform, so every store shares the same underlying endpoints — most usefully /products.json, a paginated feed of the catalog intended for legitimate integrations. Because it's standardized, one client works across every Shopify store with no per-theme parsing. The friction is operational: recurring polling from one IP triggers Shopify's platform rate limits, and a subset of merchants disable the endpoint, so you need fallbacks and IP rotation rather than clever HTML parsing.

Pagination rail for /products.json until an empty products array stops the crawl.
Stop when a page returns an empty products array. That is the real end.

Challenges that make this hard to automate

  • Knowing the /products.json feed exists instead of scraping rendered HTML
  • Paginating the feed correctly and detecting the true end
  • Stores that disable /products.json and need a fallback
  • Platform rate limits on recurring polling from one IP
  • Modeling variants, which carry their own price and availability

Approaches that usually fail

  • Headless-browser scraping per store — slow, brittle, and unnecessary on Shopify
  • Theme-specific HTML parsers — break whenever a merchant changes themes
  • Polling the JSON from one server IP — quickly rate-limited
  • Manual catalog checks — impossible across dozens of stores daily

When residential proxies fix this — and when they cannot

The data access is easy on Shopify; reliability at frequency is the real need. Routing the /products.json polling through rotating residential IPs keeps per-IP request rates under Shopify's platform limits, so you can refresh many stores on a schedule without throttling. Country targeting also surfaces the correct currency and market where stores localize.

Fallback pipeline from products.json to collections feed, product.js, then IP rotation.
Light JSON still rate-limits from one IP. Rotate on a recurring monitor.

How Aethyn residential proxies help here

Shopify monitoring is high-frequency but lightweight per request, which is exactly the Premium tier's profile. Aethyn handles rotation and geography through the username.

  • Premium residential pool — cost-efficient for frequent, light JSON polling
  • Per-request rotation so recurring catalog refreshes stay under rate limits
  • Country targeting for stores that localize currency and availability
  • Sticky sessions for cart/checkout-based stock checks when needed
  • Per-byte billing that keeps daily multi-store monitoring predictable

How to implement this with residential proxies

  1. 1

    Pull the catalog from /products.json

    Append /products.json to any Shopify store's domain to get a paginated JSON feed of its products — title, handle, vendor, product type, tags, images, and a variants array with prices and SKUs. No rendering, no theme parsing. Route it through a residential IP so recurring polling isn't rate-limited.

    Python (requests)
    import requests
    
    PROXY = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
    proxies = {"http": PROXY, "https": PROXY}
    
    def fetch_page(store, page, limit=250):
        url = f"https://{store}/products.json"
        r = requests.get(url, params={"limit": limit, "page": page},
                         proxies=proxies, timeout=30)
        r.raise_for_status()
        return r.json().get("products", [])
    
    print(len(fetch_page("examplestore.com", 1)))

    Field note: limit=250 is the maximum page size Shopify honors. Using it cuts the number of requests (and proxy bandwidth) for a large catalog by up to 25x versus the default of 10.

  2. 2

    Paginate until the feed is empty

    Walk pages with ?page=N until a page returns an empty products array — that's the true end. Don't guess a page count; catalogs grow and a fixed cap silently truncates large stores.

    Python (pagination)
    def fetch_catalog(store, max_pages=500):
        products, page = [], 1
        while page <= max_pages:
            batch = fetch_page(store, page)
            if not batch:
                break
            products.extend(batch)
            page += 1
        return products

    Field note: Each product's updated_at lets you do incremental syncs: after the first full pull, you only care about products whose timestamp changed, which keeps daily refreshes cheap.

  3. 3

    Model variants — that's where price and stock live

    On Shopify, the buyable unit is the variant (size, color), and each variant has its own price, SKU, and an available flag. Flatten products into variant rows so price tracking and stockout detection are accurate, and key on variant id for stability.

    Python (flatten variants)
    def variant_rows(product):
        rows = []
        for v in product.get("variants", []):
            rows.append({
                "product_id": product["id"],
                "variant_id": v["id"],
                "title": f'{product["title"]} - {v["title"]}',
                "sku": v.get("sku"),
                "price": v.get("price"),          # string, store currency
                "available": v.get("available"),
            })
        return rows

    Field note: products.json doesn't always include inventory counts, but it does include the available boolean per variant — enough to detect sellouts and restocks over time without hitting cart endpoints.

  4. 4

    Fall back when /products.json is disabled

    A minority of merchants disable the global feed. You can usually still read per-collection feeds at /collections/<handle>/products.json, or a single product's JSON at /products/<handle>.js. Only fall back to a headless browser through the proxy if all JSON routes are closed.

    Python (fallbacks)
    def product_json(store, handle):
        # Single-product JSON when the global feed is off
        url = f"https://{store}/products/{handle}.js"
        return requests.get(url, proxies=proxies, timeout=30).json()
    
    # Collection-scoped feed:
    # https://{store}/collections/{collection_handle}/products.json?limit=250&page=1

    Field note: Discover collection handles from the store's sitemap.xml (Shopify auto-generates /sitemap_products_1.xml and collection sitemaps). It's the cleanest way to enumerate what to poll when the global feed is restricted.

  5. 5

    Schedule, rotate, and diff

    Poll each store on a cadence through rotating residential IPs, store variant-level price and availability with timestamps, and diff against the last pull to detect new products, price changes, and sellouts. Alert only on meaningful changes to keep the signal clean.

Best practices that keep scrapers reliable

  • Use /products.json with limit=250 instead of scraping HTML
  • Paginate until an empty array; never hard-code a page count
  • Flatten to variant rows and key on variant id
  • Use updated_at for cheap incremental syncs
  • Fall back to collection/product JSON before a headless browser
  • Rotate residential IPs so recurring polling avoids platform rate limits

Common mistakes that burn proxy budget

  • Building a headless-browser scraper when /products.json exists
  • Writing theme-specific HTML parsers that break on theme changes
  • Collapsing variants into one price and missing per-variant stock
  • Stopping pagination at a fixed page and truncating large catalogs
  • Polling many stores from one IP and getting rate-limited
  • Ignoring the available flag and missing sellouts/restocks

Common questions about this scraper problem

Do all Shopify stores expose /products.json?
The large majority do — it's a standard endpoint on the platform. A minority of merchants disable it, in which case you can usually fall back to per-collection feeds (/collections/<handle>/products.json) or a single product's .js endpoint before resorting to a headless browser.
Why use /products.json instead of scraping the pages?
It returns the whole catalog as structured JSON — titles, variants, prices, SKUs, images — with no theme-specific HTML parsing, and it works identically across every Shopify store. It's faster, far more robust, and cheaper in bandwidth than rendering pages.
Why do I still need proxies if the data is in an open API?
Because the constraint is frequency, not access. Recurring polling of many stores from a single IP hits Shopify's platform rate limits. Rotating residential IPs keeps per-IP request rates low so scheduled monitoring stays reliable, and country targeting surfaces correct localized currency.
How do I track prices and stock accurately?
Work at the variant level. Each variant has its own price string and an available boolean, so flatten products into variant rows keyed on variant id, then store them with timestamps and diff over time to catch price changes and sellouts.
How do I paginate the feed correctly?
Request ?limit=250&page=N and increment page until a page returns an empty products array — that empty response is the true end. Don't assume a page count, since catalogs change size.
How do I do efficient incremental updates?
Use each product's updated_at timestamp. After an initial full pull, only reprocess products whose updated_at changed, which keeps daily refreshes cheap in both compute and bandwidth.
Which tier should I use?
Premium. Shopify monitoring is high-frequency but lightweight per request, so the cost-efficient Premium pool with per-request rotation is the right fit; reserve Elite for any specific store fronted by heavier protection.
Is monitoring Shopify stores legal?
Reading a store's public product feed is common for competitive research, but respect each store's Terms, avoid personal/customer data, keep request rates reasonable, and consult counsel for your jurisdiction and use case.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions