Price IntelligenceElite tier

How to Build a Price Intelligence System

A price intelligence system is the difference between 'we scraped some prices' and 'pricing can trust this number at 9am every day.' That gap is architecture: decoupled stages, durable storage of raw and parsed data, product matching, and data-quality monitoring. This guide covers the whole pipeline, not just the scraper.

SCWritten by Sarah Chen·Reviewed by Liam O'Connor·Updated November 21, 2026
14 min read
Editorial board contrasting a scraper wired straight to a dashboard with queued collect, parse, store, and serve stages.

Key takeaways

  • Decouple collection, parsing, normalization, storage, and serving with a queue — never wire a scraper straight to a dashboard.
  • Store raw HTML separately from parsed records so you can re-parse history when a layout (or a parser bug) changes.
  • Product matching is a first-class service with confidence scores, not an afterthought in the parser.
  • Monitor data quality — coverage, freshness, parse-success — as explicitly as you monitor uptime.
  • The collection layer's reliability caps the whole system's reliability, which is why geo-accurate residential proxies sit at the base.

On this page

What a price intelligence system has to deliver

A company that competes on price wants a system that, every morning, tells pricing exactly where they stand against competitors across marketplaces and regions — with history, alerts, and enough trust to drive real decisions. A single scraper script cannot deliver that. It gets blocked, breaks on a layout change with no way to recover the lost days, mixes currencies, keeps no history, and silently degrades until someone notices the dashboard is wrong. The actual problem is designing a durable pipeline where each concern is isolated and observable.

Why one-off price scrapers fail in production

One-off scrapers don't survive production because they couple everything: fetching, parsing, currency handling, and presentation all live in one fragile script with no buffer between stages. When a target blocks the crawler, parsing stops; when a layout changes, you lose data permanently because the raw HTML was never kept; when one currency is misparsed, it pollutes the dashboard directly. Without decoupling, durable raw storage, and quality monitoring, the data is incomplete, stale, and untrustworthy precisely when pricing needs to rely on it.

Four-stage pipeline: collect, parse, store, then serve.
Collection reliability is the ceiling for every stage after it.

What makes price intelligence hard to build

  • Reliable, geo-aware collection across many heterogeneous sources
  • Recovering history when a parser or layout changes after the fact
  • Normalizing currency and units and matching equivalent products across sites
  • Storing time-series history that supports trends and audits
  • Detecting meaningful changes and surfacing data-quality regressions without alert fatigue

Why spreadsheets and monolithic scrapers fall short

  • Spreadsheets and manual checks — stale the moment they're saved
  • A single monolithic scraper — fragile, unrecoverable, and blocked at scale
  • Bought price feeds — limited coverage, unknown freshness, and not your exact SKUs
  • A BI dashboard with no reliable, monitored ingestion underneath it

Where residential proxies fit in a price intelligence stack

Every stage downstream depends on the collection layer actually returning correct, geo-accurate pages on schedule — so that layer's reliability is the ceiling for the whole system. Residential proxies with rotation and country targeting keep ingestion working across many sources and regions, giving normalization, matching, and serving trustworthy input instead of a stream of blocks and wrong-region prices.

Two panes showing stored raw HTML beside a re-parsed record after a layout change.
If you only stored the parsed row, a parser bug deletes history.

How Aethyn Elite fits price collection

In a price intelligence system the proxy network is the foundation of the collection stage. Aethyn is built to be that backbone: one endpoint, per-source tier selection, rotation, and per-byte cost control for continuous crawling.

  • Elite reputation for the high-security marketplaces usually in the source mix
  • Premium for general retail sources, switchable per source by port
  • Country targeting so regional pricing is captured accurately at ingestion
  • Per-request rotation for durable, scheduled, high-volume collection
  • Per-byte metering so continuous crawling stays cost-predictable

How to build a price intelligence system

  1. 1

    Design decoupled stages around a queue

    Lay out five stages connected by a message queue (SQS, Kafka, or Redis Streams): collect → store raw → parse → normalize/match → serve. Decoupling means a blocked crawler doesn't stop parsing of already-fetched pages, a parser bug doesn't lose data, and you can scale or redeploy any stage independently. Add a dead-letter queue so poison records are quarantined instead of stalling the line.

    Field note: The single most valuable design decision is persisting raw HTML (e.g. to S3) before parsing. When a site changes layout — or you find a parser bug — you re-run parsing over stored raw pages and recover history, instead of staring at a permanent gap in your charts.

  2. 2

    Build the proxied collection layer

    Crawl each source from the correct country through rotating residential IPs, selecting the tier per source by port (Elite 5499 for marketplaces, Premium 2099 for general retail). Emit the raw payload to storage/queue with enough metadata to trace and re-parse it later — never parse inline here.

    Python (collector)
    import requests, time, hashlib
    
    def collect(url, country, tier_port=5499):
        proxy = f"http://aethyn-XXXXX-country-{country}:PASSWORD@proxy.aethyn.io:{tier_port}"
        proxies = {"http": proxy, "https": proxy}
        r = requests.get(url, proxies=proxies, timeout=30)
        return {
            "url": url,
            "country": country,
            "tier_port": tier_port,
            "status": r.status_code,
            "fetched_at": time.time(),
            "html_key": store_raw(r.text),          # persist raw -> object store
            "content_hash": hashlib.sha256(r.content).hexdigest(),
        }

    Field note: Store the content_hash and skip re-emitting downstream when a page is byte-identical to last crawl. Most product pages don't change between crawls; deduping on hash cuts parsing load and proxy bandwidth dramatically.

  3. 3

    Parse and normalize as separate concerns

    Parse raw HTML into typed fields, then normalize in its own step: parse the localized price to a (amount, currency) pair and convert to a base currency with a pinned daily FX snapshot. Keeping parse and normalize separate means an FX or rounding change never requires re-fetching pages.

    Python (normalize)
    import re
    
    def parse_price(text, decimal=","):
        num = re.sub(r"[^\d.,]", "", text)
        num = num.replace(".", "").replace(",", ".") if decimal == "," else num.replace(",", "")
        return round(float(num), 2)
    
    def to_base(amount, currency, rates):
        # rates = pinned daily snapshot, relative to base currency
        return round(amount / rates[currency], 2)

    Field note: Pin FX to a daily snapshot and store both local and base price. Converting with live rates makes every product look like it changed price on every crawl — currency drift masquerading as competitor moves.

  4. 4

    Match products as a dedicated service

    Treat product matching as its own component, not a line in the parser. Match on stable identifiers (GTIN/UPC/MPN) first, fall back to attribute + fuzzy-title matching, and attach a confidence score. Route low-confidence matches to human review so one bad mapping can't quietly drive a pricing decision.

    Python (matching)
    def match(record, catalog):
        if record.get("gtin") and record["gtin"] in catalog.by_gtin:
            return catalog.by_gtin[record["gtin"]], 1.0          # exact
        cand, score = catalog.fuzzy(record["title"], record.get("brand"))
        return cand, score                                       # score in [0,1]
    
    # downstream: auto-accept >= 0.9, queue 0.6-0.9 for review, drop < 0.6

    Field note: Persist the match decision and its score, not just the final link. When pricing disputes a comparison, being able to show 'matched on GTIN, confidence 1.0' versus 'fuzzy title, 0.7' is the difference between trust and a credibility problem.

  5. 5

    Store time-series history and monitor data quality

    Append every observation to a time-series store (TimescaleDB/Postgres) keyed by (product, source, region, timestamp). Then monitor the pipeline like a product: track coverage (% of SKUs successfully collected), freshness (age of the newest data per source), and parse-success rate. A drop in any of these is a data-quality incident, even when nothing has 'errored'.

    Field note: Alert on coverage and freshness, not just exceptions. The dangerous failure mode isn't a crash — it's a source quietly slipping from 98% to 60% coverage while every individual job reports success.

  6. 6

    Serve dashboards and threshold alerts

    Expose the normalized, matched, historical data to BI dashboards and push change alerts — using the same threshold-plus-confirmation logic that suppresses false positives — to where pricing teams actually work. The serving layer reads from storage only; it never touches the scrapers, so a collection hiccup never takes the dashboard down.

    Python (change detection)
    def is_real_change(history, new, threshold=0.01):
        # history: recent base prices, oldest -> newest
        if not history:
            return False                      # first observation: record, don't alert
        last = history[-1]
        delta = abs(new - last) / last
        if delta < threshold:
            return False                      # within noise band
        # confirm: the previous crawl must already agree with 'new'
        return len(history) >= 2 and abs(history[-1] - new) / new < threshold

Best practices for price intelligence pipelines

  • Decouple collection, parsing, normalization, matching, and serving with a queue
  • Persist raw HTML before parsing so you can re-parse and recover history
  • Make product matching a service with confidence scores and human review
  • Pin FX daily and store both local and base price
  • Monitor coverage, freshness, and parse-success as core SLOs
  • Geo-target collection per source and select tier by port
  • Capture promo badges, availability, and seller alongside price, not price in isolation
  • Alert with hysteresis — a threshold plus confirm-on-next-crawl — and record the first observation silently so a new SKU never fires a phantom change

Mistakes that corrupt price history

  • Wiring scrapers directly to dashboards with no buffer between stages
  • Parsing inline and never keeping raw HTML, so layout changes lose history
  • Burying product matching in the parser with no confidence or review
  • Converting with live FX, turning currency drift into phantom changes
  • Storing only the latest price with no time-series history
  • Watching only for crashes while coverage silently decays
  • Setting the exit country but leaving Accept-Language mismatched, so a German exit sending en-US quietly returns the English/EUR variant instead of the local price

Price intelligence questions

What are the components of a price intelligence system?
Five decoupled stages connected by a queue: a proxied collection layer, durable raw storage, a parsing layer, a normalization + product-matching layer, and a serving layer for dashboards and alerts. Decoupling is what makes it survive blocks, layout changes, and scale.
Why store raw HTML separately from parsed data?
Because parsers and layouts change. If you only keep parsed records, a layout change or a parser bug means permanently lost history. Keeping the raw HTML lets you re-run parsing over the past and backfill, which is the single biggest reliability win in the design.
Why are proxies central to this system?
Collection is the foundation, and the whole system is only as reliable as its input. If the crawler is blocked or returns the wrong region's prices, normalization and dashboards are confidently wrong. Residential proxies with rotation and geo-targeting keep ingestion accurate and durable.
How do I match the same product across sites?
Make matching its own service: prefer stable identifiers (GTIN/UPC/MPN), fall back to attribute and fuzzy-title matching, attach a confidence score, and route low-confidence matches to human review before they affect pricing.
How do I monitor that the data is actually good?
Track data-quality SLOs — coverage (% of SKUs collected), freshness (age of newest data per source), and parse-success rate — and alert on them. The common silent failure is a source drifting from full coverage to partial while every job still reports success.
Why not just collect prices from datacenter IPs?
Retail sites frequently block datacenter ranges outright or serve them a different, non-representative price — the classic IP-based price discrimination problem. Residential IPs in the target country return the price and promotions a real local shopper sees, which is the only input a price intelligence system can safely act on.
Can the system capture promotions and stock, not just price?
Yes, and a serious pipeline should. A headline 'price drop' is often a coupon, a flash promo, or a third-party seller — capturing promo badges, availability, and seller alongside price is what gives you the real competitive picture instead of a misleading number.
How often should the pipeline re-collect each source?
Match cadence to volatility: hourly or sub-hourly for fast-moving or high-value SKUs, daily for the stable long tail. Scheduling collection by priority keeps both proxy cost and block risk down.
Is competitor price monitoring legal?
Collecting publicly displayed prices is common practice, but respect each site's Terms of Service, avoid personal data, and consult counsel for your jurisdiction and specific use case.
Which tier should I use?
Mix per source: Elite for high-security marketplaces and Premium for general retail, switching by endpoint port. The architecture treats tier as a per-source collection parameter.
How do I avoid alert fatigue?
Normalize to a base currency, ignore sub-threshold deltas, and require a change to persist across crawls before alerting — the same hysteresis used in standalone monitoring, applied at the serving layer.
How do I keep continuous crawl costs predictable?
Aethyn meters by the byte, so dedupe on content hash, cache, crawl by SKU priority, and skip unchanged pages. Persisting raw HTML also means re-parsing is free — you don't re-download to fix a parser.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions