How to Rotate Proxies in Python
Short answer: you rotate proxies in Python in one of three ways. Use a provider rotating endpoint that changes IP automatically, pick a random proxy from a list per request, or assign a sticky session per worker. Rotation spreads requests across many IPs so no single address gets flagged.
Why rotate at all
Sites rate limit and block IPs that send too many requests. Rotation makes your traffic look like many separate users rather than one bot, which is the single most effective way to keep a crawler running.
Option 1: Use a rotating endpoint
The simplest method. Your provider gives one proxy URL that returns a new IP on each request, so your code never changes.
import requests
proxy = "http://USER:PASS@ROTATING_HOST:PORT"
proxies = {"http": proxy, "https": proxy}
for url in urls:
r = requests.get(url, proxies=proxies, timeout=20)
ShiftProxies rotating residential and mobile endpoints work this way, so you get fresh IPs without managing a list at dashboard.shiftproxies.com.
Option 2: Rotate from a list
If you have many proxy URLs, choose one per request.
import random, requests
PROXIES = ["http://USER:PASS@h1:port", "http://USER:PASS@h2:port"]
def fetch(url):
p = random.choice(PROXIES)
return requests.get(url, proxies={"http": p, "https": p}, timeout=20)
Track which proxies fail and skip them for a while so you do not keep hitting a bad one.
Option 3: Sticky session per worker
When each worker runs a multi step flow, give each its own sticky IP so the session stays consistent. Rotate between workers, not within a single session.
Tips for reliable rotation
- Send a realistic user agent and headers
- Add randomized delays between requests
- Set timeouts and retry on a different IP after failures
- Back off after blocks instead of retrying instantly
- Prefer residential or mobile IPs for protected sites
Summary
For most scrapers a rotating endpoint is the easiest and most reliable choice. Use a list when you manage your own pool, and use sticky sessions for stateful flows. Combine rotation with good headers and pacing to avoid blocks.