BS
Scraping Frameworks

BeautifulSoup proxy integration

BeautifulSoup parses HTML but doesn't fetch it — you pair it with Requests (or HTTPX) to download pages. Route the fetch through Aethyn residential proxies, then parse the response with BeautifulSoup.

Setup steps

  1. 1Install both: pip install requests beautifulsoup4.
  2. 2Fetch the page through the Aethyn proxy with Requests.
  3. 3Parse response.text with BeautifulSoup.
  4. 4Add targeting suffixes for localized content.

Code

Requests + BeautifulSoup
import requests
from bs4 import BeautifulSoup

proxy = "http://aethyn-XXXXX-country-us:PASSWORD@proxy.aethyn.io:2099"
proxies = {"http": proxy, "https": proxy}

resp = requests.get("https://example.com", proxies=proxies, timeout=30)
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.string)

Replace aethyn-XXXXX and PASSWORD with your dashboard credentials. See the full targeting reference and protocol/port list.

Tips

  • BeautifulSoup only parses — the proxy lives on the Requests/HTTPX call.
  • Use lxml as the parser for speed on large pages.
  • Rotate IPs per request for broad crawls; the default mode handles this.

BeautifulSoup proxy FAQ

Does BeautifulSoup need its own proxy config?
No. BeautifulSoup parses HTML you already fetched. Configure the proxy on the HTTP client (Requests or HTTPX) that downloads the page.

Ready to wire up BeautifulSoup?

Get your credentials and route your first BeautifulSoup request through residential IPs in minutes.

Docs, solutions & blog

Related integrations

All integrations