ScrapingProxies

What Is Web Scraping? (2026)

AE
Engineering desk
Published
October 25, 2026
Reviewed by
Read Time
3 min read
Four stages: request a public page, receive HTML, pick the fields, save a row

Web scraping is reading a public page and saving the fields you need. A program requests a URL, receives HTML or JSON the site already sends to a browser, and writes a few of those fields to a row: a title, a price, a name. The glossary definition is that sentence. The rest of this page is the pipeline, what it is not, and where a residential address fits.

People scrape to compare a public price across shops, to record a search result from a given country, or to notice that a public page changed. Those jobs are price monitoring and SERP tracking when the field is a price or a ranking. The tool is ordinary HTTP. The judgment is whether you are allowed to store what you read. That judgment is the legal and ethical note, and it is not optional once the run leaves a single example URL.

The request and the row

Start with a page you are allowed to read. https://example.com is a public placeholder. Replace it with your own public URL. The pattern is the same: one GET, then the <title> pulled out of the body. A real job swaps the title for the fields you actually store.

Code Snippet
import re
import requests

url = "https://example.com"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.48 Safari/537.36",
}
r = requests.get(url, headers=headers, timeout=30)
match = re.search(r"<title>(.*?)</title>", r.text, re.I | re.S)
print(r.status_code, match.group(1).strip() if match else "")

The User-Agent above is a Chrome 156 example from the stable channel on 20 October 2026. What that header does, and what it does not change, is the User-Agent note. Check the status and the body. A 200 whose HTML is a block page is not a successful scrape. Save a row only when the fields are the ones you asked for.

A browser page with a title and a price, beside a table that keeps those two fields
The row is a copy of public text. It is not a second copy of the whole page unless you meant to store that.

When the address matters

Many public pages answer the snippet above from your own network. A residential proxy is the step you add when the site treats the caller’s IP as part of the visitor: a local price, a country catalog, or a refusal of a hosting address. The proxy is not the scraper. It is the path. What a residential proxy is is the product definition. It is not a VPN, which is a device tunnel and a different job, covered in proxy vs VPN.

Premium HTTP is proxy.aethyn.io port 2099. The username carries the country. The password is the dashboard secret. Quote both only because they sit inside a proxy URL. requests decodes them before it builds the header.

Code Snippet
from urllib.parse import quote

user = "aethyn-XXXXX-country-us"
password = "PASSWORD"
proxy = f"http://{quote(user, safe='')}:{quote(password, safe='')}@proxy.aethyn.io:2099"
r = requests.get(url, headers=headers, proxies={"http": proxy, "https": proxy}, timeout=30)

Pause between requests. A new exit does not replace a polite pace. Stay on pages the public can open without an account. This pipeline does not log in, does not solve a challenge, and does not call a private API the site did not give you. If the gateway refuses the login, the status is 407 and the body says whether the header was missing or the secret was wrong. That is a credential problem, not a parsing problem.

Three checks before a large run: a public page, a pause between requests, and a household exit when the site sorts visitors by address
Public page, a pause, and a household exit only when the address is part of the answer.
FAQ

Web scraping questions

What is web scraping?
Web scraping is requesting a page and saving specific fields from the response, such as a title or a price. The short glossary line is Web scraping. It is a copy of text the page already sent, not a login and not a private API.
Is web scraping the same as a proxy?
No. Scraping is the read and the save. A proxy is an optional path for that request. Many public pages answer a normal client. A residential proxy matters when the site uses the caller’s address to decide what to show or whether to answer.
Do I have to solve a challenge to scrape?
No. This page does not cover challenge solving, CAPTCHA bypass, or account logins. Stay on public pages, pause between requests, and read the legal note before you scale a run.
What should I save from the page?
Save the fields you can point at in the HTML or in JSON the page itself returned. A title, a price, and a URL are enough to prove the pipeline. Do not store a page you were not allowed to read.
Share this article

Guides, integrations & docs

Continue reading

Read the page from a household address

When the site sorts visitors by IP, Premium HTTP on port 2099 sends the request through a residential exit. Free trial, no card.

Back to all articles