Proxy Pool Management: From a List of IPs to a Stable System

The inventory: what to record

First, turn the ip:port:user:pass list into a structured inventory. A spreadsheet, JSON file or database table all work as long as every IP has the same fields. For bulk format changes, use the proxy format converter.

Periodic health checks

An IP is alive when you can connect to the proxy, authenticate, exit from the expected IP and load a page in reasonable time. Checking each step separately shows where a fault is: no connection usually means your network or firewall, a 407 means wrong credentials, and an unexpected exit IP means a configuration mistake.

• Send a light request through the proxy to an IP echo endpoint, such as curl -x http://user:pass@ip:port -s -m 10 https://httpbin.org/ip, and compare the returned IP with the inventory.
• Record response time to spot unusually slow IPs; testing proxy speed and latency explains how to measure properly.
• Add a target-specific check if needed: a live IP is not guaranteed to reach the site you scrape.
• Run checks on a schedule, for example hourly for pools running jobs, and before every large job.
• Check a whole list by hand with the proxy checker; other methods are in checking whether a proxy is alive.
• Do not mark an IP dead after one failure. Transient network errors happen; change status only after several consecutive failures.

States and the life of an IP

Four states cover most systems. active: ready for work. cooling: recently got 429, 403 or a CAPTCHA from a target and rests until a set time. dead: repeated connection or authentication failures that need checking or a support ticket. reserved: pinned to one account and kept out of general rotation.

Keeping cooling separate from dead matters. An IP throttled by a site is healthy and just needs a rest; marking it dead and reporting a fault confuses a pacing problem with a proxy problem. If an IP really is dead because of a provider-side fault, open a ticket with the IP, time and error message so it can be handled quickly.

Allocate IPs by task

• Accounts that keep sessions (social, shops, ads): one reserved IP per account, never in rotation. Changing the IP of a healthy account adds an unusual signal.
• Public-data scraping: a shared pool of active IPs, round-robin or random, each at its own safe pace. Code for this is in rotating a proxy list in Python.
• Several markets: split the pool by country so each job only takes IPs from the country it needs.
• Several people and tools: record who owns which IP. Two tools unknowingly sharing an IP is a common cause of puzzling 429s. See sharing proxies with a team safely.
• Spares: keep a few idle IPs to swap in when one needs a long rest, rather than loading the rest more heavily.

A retry policy

Uncontrolled retries are the fastest way to turn a small error into a request storm. Write a clear policy and apply it across tools: connection errors and timeouts retry on another IP a few times; 429 and 503 honour Retry-After, move the IP to cooling and lower the overall pace; repeated 403s across many IPs stop the job for a review of how requests are sent, because the IP is not the issue.

Each retry should wait longer than the last, with jitter. Cap total attempts per URL and push failed URLs to a later queue. More on 429 is in HTTP 429 Too Many Requests.

Logs and metrics to watch

• Log one line per request: time, job, URL or domain, IP, status code, response time, attempt number.
• Track success rate per IP and per domain. An IP well below the rest needs attention; a domain whose rate drops across the board has changed how it blocks.
• Watch median and slow-tail latency per IP, not just the average.
• Count IPs in each state daily; a steady rise in cooling IPs means you are running too fast.
• Never write proxy passwords to logs. Record IP and port only.
• Keep logs long enough to compare before and after configuration changes.

Track expiry and renewals

A dedicated IP is yours only for the rental period. Let it lapse and it may go to someone else, taking your log history and any accounts tied to it. Set reminders before expiry in the inventory and renew from the dashboard to keep the same IP; renewing a proxy to keep the same IP covers the details.

Review the pool periodically: IPs unused for weeks may not need renewing, and IPs that are always busy may signal you need more. Decide from the logs, not from a hunch.

Frequently asked questions