How to Rotate a List of Static Proxies in Python

Rotating static IPs vs a rotating proxy service

A rotating proxy service gives you one gateway and changes the exit IP behind it per request or on a timer. You do not know which IP comes next and you cannot keep one you like. That suits very high-volume crawling, but it makes it hard to know exactly which IP sent which request. For the full comparison, read static vs rotating proxies.

When you rotate your own list of static IPs, every decision lives in your code. Each IP is yours alone, so its reputation depends on how you use it. You can log the IP behind every request, bench an IP when a site starts returning errors, and keep one IP for a logged-in session. The trade-off is that you write the selection, retry and throttling logic yourself. The rest of this guide is the minimal code to do that properly.

Step 1: prepare the proxy list

Proxies are delivered as ip:port:user:pass, one per line. Save them to proxies.txt. The requests library expects URLs, so convert once when you load the file. If a password contains characters such as @, : or #, URL-encode it; otherwise requests parses the URL wrongly and you get a 407 Proxy Authentication Required error. You can also convert a whole list with the proxy format converter.

• from urllib.parse import quote
• def to_url(line):
• ip, port, user, pw = line.strip().split(':', 3)
• return 'http://' + quote(user, safe='') + ':' + quote(pw, safe='') + '@' + ip + ':' + port
• PROXIES = [to_url(l) for l in open('proxies.txt') if l.strip()]
• Use the http:// scheme for both the http and https keys of the proxies dict. For HTTPS sites, requests opens a CONNECT tunnel through the HTTP proxy, so the proxy itself does not need to speak TLS.

Step 2: three ways to rotate

No single strategy fits every job. Public pages without a login can be spread evenly across IPs, while anything with a session, cart or login should keep one IP for the whole session.

• Round-robin example: for url in urls: p = next(pool); r = requests.get(url, proxies={'http': p, 'https': p}, timeout=(5, 20))
• Sticky example: set key to an account name or domain and the same key always maps to the same IP, even after a restart. Use zlib.crc32 rather than hash(), because Python randomises hash() between runs.

Step 3: one Session per proxy

Calling requests.get() for every URL opens a fresh connection and TLS handshake each time. For a group of requests on the same IP, create one requests.Session() per proxy and reuse it. A session keeps connections alive and stores cookies, which pairs naturally with sticky rotation: each session is a fixed IP plus its own cookies.

A simple pattern is a sessions = {} dict, then s = sessions.setdefault(p, requests.Session()) and s.proxies = {'http': p, 'https': p}. Avoid sharing one session while swapping proxies between requests, because cookies earned on one IP get sent from another, and the site sees a session jumping between IPs. More configuration detail is on the Python requests proxy setup page.

Step 4: retry on another IP and bench failing ones

In a list of a few dozen IPs, one will occasionally time out or receive 403 or 429 from a particular site. Good code does not stop the whole run; it moves the request to another IP and rests the one that failed. The simplest way is a cooldown = {} dict holding the time each IP may be used again.

• Catch connection errors with except (requests.exceptions.ProxyError, requests.exceptions.ConnectTimeout, requests.exceptions.ReadTimeout), record the failure and try the next IP.
• On 429 or 403, set cooldown[p] = time.time() + 300 to rest the IP for five minutes, and skip any proxy where cooldown.get(p, 0) > time.time().
• Cap attempts, for example three IPs per URL, so one broken page cannot burn through the whole list.
• Log URL, IP, status code and response time for every request. When something goes wrong, the log tells you whether the problem is one IP, the whole pool or the target site.
• For health checks, load balancing and logging at a larger scale, see proxy pool management.

Step 5: run concurrently without overloading one IP

With concurrent.futures.ThreadPoolExecutor, scale the number of threads to the number of IPs rather than cranking it up. A safe starting rule is no more threads than IPs, with each IP handling one request at a time. A queue.Queue of idle proxies does this neatly: a thread takes a proxy, sends the request, waits briefly and puts the proxy back.

A small random delay between requests on the same IP, such as time.sleep(random.uniform(1, 3)), makes the pattern less mechanical and is kinder to the target server. The right pace depends on the site; web scraping rate limits explains how to estimate requests per IP and how many IPs a workload needs.

Checklist before a real run

• Check the whole list first with the proxy checker or a test call to an IP echo endpoint such as requests.get('https://httpbin.org/ip', proxies=..., timeout=10).
• Always set a timeout; without one, a hung IP can block a thread forever.
• Send a sensible User-Agent and keep it consistent within a session.
• Respect robots.txt and the site's terms, and collect only public data you are allowed to use.
• Keep proxy credentials out of code you push to Git; read them from a file or environment variables.
• When the rental period ends, renew from the dashboard to keep the same IPs, so your list and per-IP logs stay valid.

Frequently asked questions