Web Scraping With Python and Proxies: A Practical Guide

Web Scraping With Python and Proxies

Short answer: in Python you scrape through a proxy by passing the proxy URL to your HTTP client, adding authentication, and rotating IPs across requests so no single address makes too many calls. This guide shows the core pattern with the popular requests library and how to scale it.

Why use proxies in Python scraping

A scraper that sends every request from one IP gets rate limited or blocked fast. Proxies spread requests across many IPs, and residential proxies look like real users so protected sites are less likely to block you.

The basic pattern with requests

Pass a proxies dictionary to each request. A proxy URL with credentials looks like http://user:pass@host:port.

import requests

proxy = "http://USER:PASS@HOST:PORT"
proxies = {"http": proxy, "https": proxy}

resp = requests.get("https://example.com", proxies=proxies, timeout=20)
print(resp.status_code)

Send realistic headers too, especially a normal user agent, so requests look like a browser.

Rotating proxies

For scale you rotate IPs. With a rotating endpoint from your provider, the IP changes automatically on each request, so you keep using the same proxy URL. With a list of proxies, pick a different one per request.

import random

PROXIES = ["http://USER:PASS@host1:port", "http://USER:PASS@host2:port"]

def get(url):
    proxy = random.choice(PROXIES)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

ShiftProxies provides rotating residential and mobile endpoints, so a single proxy URL gives you a fresh IP per request without managing a list, available at dashboard.shiftproxies.com.

Handling errors and retries

• Set timeouts so a slow proxy does not hang your crawler
• Catch connection errors and retry on a different IP
• Back off after blocks rather than retrying instantly
• Log failed URLs to rerun later

Pacing and politeness

Add small randomized delays, keep per IP volume modest, and respect each site. Steady crawling finishes more often than aggressive bursts.

Sticky sessions for stateful flows

For logins or multi step actions, use a sticky session so the IP stays the same across the flow. Switching IPs mid session triggers security checks.

Summary

Pass a proxy URL with credentials to requests, send realistic headers, rotate IPs, handle errors with retries and backoff, and pace your crawl. That core pattern scales from a small script to a large scraper.