AI & DataPremium tier

How to Build LLM Training Datasets From the Web

Raw web text is not a training set — it's mostly boilerplate, duplicates, and noise, and shipping it to a model wastes compute and inflates benchmarks. This guide covers turning broad web collection into an LLM-ready corpus: quality filtering, aggressive dedup, benchmark decontamination, PII handling, and sharded storage — with residential proxies under the collection stage.

SCWritten by Sarah Chen·Reviewed by Elena Vance·Updated September 4, 2026
13 min read
Editorial board contrasting a raw crawl dump of boilerplate and spam with a sharded JSONL training corpus.

Key takeaways

  • Collection is the easy part; the value is in filtering, dedup, decontamination, and formatting.
  • Quality-filter hard: drop boilerplate, low-text pages, and machine-generated spam before training.
  • Dedup exact first (hashes), then near-duplicates (MinHash/LSH) — duplication causes memorization and wastes tokens.
  • Decontaminate against known benchmarks or your eval scores are inflated and meaningless.
  • Redact PII, record provenance/licensing, and honor AI opt-out signals; store as sharded JSONL/Parquet.

On this page

What goes wrong when this scraper fails

Teams building pretraining or domain corpora often equate 'more web text' with 'better data'. In practice raw crawls are dominated by navigation boilerplate, near-duplicate copies of the same content, machine-generated spam, and personal data — and they frequently contain the very benchmarks you'll evaluate on. Train on that and you waste compute, encourage memorization, leak PII, and report inflated scores. Building an LLM dataset is a data-engineering pipeline: collect broadly, then filter, dedup, decontaminate, redact, and format into shards.

Why this failure mode happens

The web is built for humans and SEO, not for training: pages repeat headers/footers, syndicate the same articles across domains, and include large volumes of low-information or auto-generated text. Benchmarks are themselves published on the web, so crawls ingest them. And personal data is everywhere. None of this is removed by simply downloading more pages — it requires explicit processing stages, which is where dataset quality is actually won or lost.

Four-stage pipeline for quality filter, exact hashes, near-copy MinHash, and benchmark decontamination.
If the crawl contains your eval set, the scores are inflated. Filter, hash, then decontaminate.

Challenges that make this hard to automate

  • Separating substantive text from boilerplate and spam
  • Removing exact and near-duplicate documents at corpus scale
  • Decontaminating against evaluation benchmarks
  • Redacting PII and respecting licensing/AI opt-out signals
  • Storing huge corpora in an efficient, shardable format

Approaches that usually fail

  • Dumping raw crawl text into training — boilerplate, dupes, and PII included
  • Exact-dedup only — misses reworded near-duplicates
  • No decontamination — benchmark leakage inflates eval scores
  • Flat single-file storage — unwieldy and slow to stream at scale

When residential proxies fix this — and when they cannot

Building a representative corpus means broad, geographically- and linguistically-diverse collection — exactly what a large residential pool with country targeting enables, while rotation keeps the crawl reliable at volume. Residential IPs also reach the regionally-varied content that datacenter ranges often can't, so the corpus isn't skewed toward a single locale. The heavy lifting after collection is pure data engineering.

Two-pane view of raw HTML pages versus sharded JSONL and Parquet with provenance fields.
Store shards, not a pile of HTML. Redact PII and honor AI opt-out signals before you ship.

How Aethyn residential proxies help here

The collection stage of a dataset pipeline needs scale and geographic/linguistic breadth. Aethyn provides both through the username.

  • Country/city targeting to deliberately sample many regions and languages
  • A large rotating pool for broad, high-volume crawling
  • Premium pricing that's cost-efficient for big collection workloads
  • Sticky sessions for multi-step source flows when needed
  • Per-byte billing so a large crawl's cost stays transparent

How to implement this with residential proxies

  1. 1

    Collect broadly and diversely, store raw

    Crawl your target sources through rotating residential IPs, deliberately sampling regions and languages so the corpus isn't skewed. Write raw documents with provenance (source URL, fetch time, region, content hash) to immutable storage — never filter in the collector, so you can reprocess as your filters improve.

    Python (collect with provenance)
    import time, hashlib, requests
    
    PROXY = "http://aethyn-XXXXX-country-de:PASSWORD@proxy.aethyn.io:2099"
    
    def fetch(url, sink):
        r = requests.get(url, proxies={"https": PROXY}, timeout=30)
        if r.ok:
            sink.write({"url": url, "ts": time.time(), "region": "de",
                        "hash": hashlib.sha256(r.content).hexdigest(),
                        "html": r.text})

    Field note: Record per-document provenance and licensing/robots/AI-opt-out signals at collection time. You cannot reconstruct lawful basis or honor opt-outs after the fact — capture it when you fetch or you'll have an unauditable corpus.

  2. 2

    Extract main text and quality-filter

    Strip boilerplate (nav, footers, ads) to main content, then filter hard: drop documents below a length threshold, with low text-to-markup ratios, garbled encoding, or signals of machine-generated spam. A smaller, cleaner corpus trains better than a huge noisy one.

    Python (quality filters)
    def quality_ok(text):
        if len(text) < 200:                  # too short to be useful
            return False
        words = text.split()
        if len(words) < 50:
            return False
        # crude symbol-ratio guard against menus/garbled pages
        alpha = sum(c.isalpha() or c.isspace() for c in text) / max(1, len(text))
        return alpha > 0.7

    Field note: Use a lightweight quality classifier (e.g. trained to distinguish reference text from web spam) for the borderline middle. Hard heuristics catch the obvious junk; a classifier recovers the gray-area documents that simple rules wrongly drop.

  3. 3

    Dedup exact, then near-duplicate

    Duplication causes memorization and wastes training tokens. Remove exact duplicates by content hash, then collapse near-duplicates (reworded syndications, templated pages) with MinHash + LSH. At corpus scale this is one of the highest-impact quality steps.

    Python (MinHash near-dedup)
    from datasketch import MinHash, MinHashLSH
    
    def minhash(text, num_perm=128):
        m = MinHash(num_perm=num_perm)
        for tok in set(text.lower().split()):
            m.update(tok.encode())
        return m
    
    def dedup(docs, threshold=0.8):
        lsh = MinHashLSH(threshold=threshold, num_perm=128)
        keep = []
        for i, d in enumerate(docs):
            mh = minhash(d["text"])
            if not lsh.query(mh):          # no near-duplicate already kept
                lsh.insert(str(i), mh)
                keep.append(d)
        return keep

    Field note: Dedup across the whole corpus, not per-source. The same article syndicated across many domains is the dominant duplicate pattern, and per-source dedup leaves all the cross-domain copies in.

  4. 4

    Decontaminate against benchmarks

    If evaluation data leaks into training, your scores are inflated and meaningless. Build a set of known benchmark items and remove training documents that contain them (exact and n-gram overlap). Treat decontamination as mandatory, not optional, for any model you'll evaluate.

    Field note: Decontaminate against the specific benchmarks you report on, using n-gram overlap rather than only exact match — paraphrased or reformatted test items still leak capability signal if left in the training mix.

  5. 5

    Redact PII, then shard the output

    Redact personal data (emails, phones, and named entities where required) before training, not after. Then write the cleaned corpus as compressed, sharded JSONL or Parquet so it streams efficiently during training, and publish a dataset card documenting sources, filtering, language mix, and known limitations.

    Python (redact + shard)
    import re, json, gzip
    
    EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.-]+")
    PHONE = re.compile(r"\+?\d[\d\s().-]{7,}\d")
    
    def redact(t):
        return PHONE.sub("[PHONE]", EMAIL.sub("[EMAIL]", t))
    
    def write_shards(docs, shard_size=100_000, prefix="corpus"):
        for s in range(0, len(docs), shard_size):
            with gzip.open(f"{prefix}-{s//shard_size:05d}.jsonl.gz", "wt") as f:
                for d in docs[s:s+shard_size]:
                    f.write(json.dumps({"text": redact(d["text"]),
                                        "url": d["url"]}) + "\n")

    Field note: Shard by a fixed document count and keep shards independently streamable. Training pipelines parallelize over shards, so uniform, self-contained shards make data loading simple and let you mix data sources by sampling shards.

Best practices that keep scrapers reliable

  • Collect broadly across regions/languages and store raw with provenance
  • Quality-filter hard, using a classifier for borderline documents
  • Dedup exact then near-duplicate across the whole corpus
  • Decontaminate against the benchmarks you evaluate on
  • Redact PII before training and honor AI opt-out/licensing
  • Store as compressed, sharded JSONL/Parquet with a dataset card

Common mistakes that burn proxy budget

  • Equating more raw web text with a better dataset
  • Training on boilerplate, spam, and unfiltered low-quality pages
  • Exact-dedup only, leaving reworded near-duplicates in
  • Skipping decontamination and inflating eval scores
  • Per-source dedup that misses cross-domain syndication
  • Ignoring PII, licensing, and AI opt-out signals

Common questions about this scraper problem

Isn't more web data always better for training?
No. Raw web text is mostly boilerplate, duplicates, and spam, and often contains PII and benchmark data. Training on it wastes compute, drives memorization, and inflates eval scores. A smaller, well-filtered and deduplicated corpus consistently trains better than a huge noisy one.
Why does deduplication matter so much?
Duplicated text causes models to memorize rather than generalize and wastes training tokens. Remove exact duplicates by content hash, then collapse near-duplicates with MinHash/LSH across the whole corpus — cross-domain syndication is the dominant duplicate pattern, so per-source dedup isn't enough.
What is benchmark decontamination and why is it required?
It's removing training documents that contain your evaluation benchmark items. Benchmarks are published on the web, so crawls ingest them; if test data leaks into training, your scores are inflated and meaningless. Decontaminate against the benchmarks you report on using n-gram overlap, not just exact match.
How should I handle PII and licensing?
Capture licensing, robots, and AI opt-out signals at collection time (you can't reconstruct them later), redact PII like emails and phones before training, and document everything. Honor opt-out signals and consult counsel on licensing for your jurisdiction and intended use.
Where do residential proxies fit in?
In the collection stage. Building a representative corpus needs broad, region- and language-diverse crawling, which a large residential pool with country targeting enables, while rotation keeps high-volume collection reliable. Everything after collection is data engineering.
What format should I store the corpus in?
Compressed, sharded JSONL or Parquet. Shard by a fixed document count and keep shards independently streamable so training pipelines can parallelize over them and mix sources by sampling shards. Include a dataset card describing sources, filtering, language mix, and limitations.
How is this different from general AI data collection?
Collecting AI training data is the upstream act of gathering web data responsibly; building an LLM dataset is the downstream pipeline that turns that raw data into a training-ready corpus through quality filtering, dedup, decontamination, PII redaction, and sharded formatting. This guide focuses on that processing.
Which tier should I use?
Premium. Corpus collection is high-volume, broad crawling rather than a heavily-defended single target, so the cost-efficient Premium pool with country targeting for linguistic/geographic diversity is the right fit.
Share this article

Solve this with Aethyn residential proxies

Route your first request through rotating residential IPs in minutes, with transparent per-byte pricing.

All solutions