ScrapingProxies

What a User-Agent Is for Web Scraping (2026)

AE
Engineering desk
Published
October 25, 2026
Reviewed by
Read Time
3 min read
Three request rows: User-Agent, Accept-Language, and a separate TLS handshake row that the header does not change

A User-Agent is the HTTP header that names the client. The value is one string: a browser family, a version, and an operating system. A site reads it to pick a layout, a language hint, or a simpler response for an old client. Web scraping uses the same header a browser would send, so the response is the document people get, not a stripped page reserved for unknown clients. The short glossary line is User-Agent.

The header is not the browser. Python requests sends its own default, which names the library. That default is honest and, on some sites, enough to get a different document than Chrome gets. Setting the header replaces that label. It does not turn the HTTP client into Chrome, and it does not change the TLS handshake. The handshake is a separate layer, covered in TLS fingerprinting for scrapers. A residential exit is a third layer: the address the site sees. Header, handshake, and address are three settings.

One string for the whole run

Copy a current desktop Chrome string and keep it stable. Chrome 156 was the stable desktop channel on 20 October 2026 (156.0.8078.48 on Windows and macOS). This is an example, not a score:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.48 Safari/537.36

Send that exact value on every request in the run. Pair it with one Accept-Language. If the username asks for country-de and the header still says en-US, the page can come back in a language you did not intend to parse. Match the language to the country you asked for, then leave both alone. Rotating the User-Agent per request does not imitate a person. A person keeps one browser open.

There is no ranked list of agents on this page. A string does not have a success rate. When Chrome ships the next stable version, replace the example with the string from a browser you control, or from the current stable release notes. An outdated major version is a stale label. A fresh label still leaves the TLS handshake and the exit address untouched.

The client sets User-Agent on the request, and the page is built from that label
The header is a label on the request. The page you parse is the response to that label.

Where to set it

In Requests, pass the string in headers. The library will not invent a Chrome handshake because the header says Chrome. It will send the header you gave it.

Code Snippet
import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.48 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}
r = requests.get("https://example.com", headers=headers, timeout=30)
print(r.status_code, r.headers.get("content-type"))

In Playwright, set user_agent on the browser context so every navigation in that context shares it. Put the proxy login in username and password, not inside the server string. The full browser setup is Playwright and residential proxies. A manual Chrome window is a different path: the browser already has a User-Agent, and the proxy host is Chrome’s proxy setting.

Code Snippet
context = browser.new_context(
    user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.8078.48 Safari/537.36",
    locale="en-US",
    proxy={
        "server": "http://proxy.aethyn.io:2099",
        "username": "aethyn-XXXXX-country-us",
        "password": "PASSWORD",
    },
)
Three places the same User-Agent string is set: Requests, a Playwright context, and the browser’s own header
One string, three clients. The handshake is still the client’s own.

Check the body you received, not the header you sent. A 200 with a challenge page is still a challenge page. The header did its job if the document is the public page you meant to read. What to do with that document is what web scraping is.

FAQ

User-Agent questions

What is a User-Agent in web scraping?
It is the User-Agent HTTP header. The value is a string that names a browser, a version, and an operating system. Sites read it when they decide how to answer. The glossary entry is the one-sentence form.
Does a User-Agent change my TLS fingerprint?
No. The header is sent after the handshake. JA3 and JA4 describe the handshake, which the header does not rewrite. That split is the TLS fingerprinting note.
Should I rotate User-Agent on every request?
No. Pick one current browser string and send it for the whole run, with the same Accept-Language. A new string on every request is a different client each time, which is the opposite of a normal browser.
Which User-Agent should I copy?
Use the string from a current desktop Chrome you control, or the Chrome 156 example below from the stable channel on 20 October 2026. Replace it when Chrome’s version moves. It is an example, not a ranking.
Share this article

Guides, integrations & docs

Continue reading

Send the header from a household exit

Premium HTTP is port 2099 with a country on the username. The header and the exit are separate settings. Free trial, no card.

Back to all articles