Web Scraping Rate Limits: How Fast Is Too Fast

What sites count when they limit you

Many people assume rate limits are per IP only, so more IPs means faster scraping. In practice a site can count requests by several keys at once, and more IPs only fix one of them. Knowing which key you are hitting lets you fix the right thing instead of buying proxies you do not need.

Read robots.txt, the terms and response headers

Open https://example.com/robots.txt to see which paths the site does not want bots to visit. Some sites also declare Crawl-delay, the number of seconds to wait between requests. Not every tool honours it, but it is the clearest hint about the pace the owner accepts. Read the terms of use too: some sites forbid automated collection, others offer an official API or data export that is faster and more stable than parsing HTML.

Once you are running, watch response headers. Retry-After on a 429 or 503 tells you when to try again. Some APIs also return headers such as X-RateLimit-Remaining or RateLimit-Remaining showing how many requests are left in the current window. Good code reads these and slows down before it gets blocked.

Pace requests in code

Limit at two levels: concurrent requests per domain, and the minimum gap between two requests on the same IP. Start slow, watch the error rate, then raise the pace gradually.

• Scrapy: set CONCURRENT_REQUESTS_PER_DOMAIN and DOWNLOAD_DELAY, and enable AUTOTHROTTLE_ENABLED = True so Scrapy adapts to response latency. Proxy wiring is covered in the Scrapy proxy setup guide.
• Python requests: after each request on an IP, wait time.sleep(base + random.uniform(0, jitter)). The jitter avoids a machine-like rhythm.
• Token bucket: give each IP a bucket of N tokens refilled at a fixed rate and only send when a token is available. It allows short bursts while holding a steady average.
• Concurrency caps: use an asyncio.Semaphore or a queue of idle proxies so each IP handles one request at a time.
• Per-domain budgets: if you scrape several sites, keep a separate limiter per domain; a small site tolerates far less than a large marketplace.

Backoff when you are refused

After a 429, a 503 or a challenge page, do not retry immediately. Exponential backoff means the first retry waits one interval, the second twice as long, the third four times, plus a random component so threads do not retry in lockstep. If Retry-After is present, use it. After several consecutive failures on one IP, rest that IP for a while and lower the overall pace of the job.

A common mistake is moving a 429'd request to another IP without slowing down, which simply walks the whole pool into the limit one IP at a time. The status code itself is covered in HTTP 429 Too Many Requests.

Work out how many IPs you need

There is no universal number, so measure. Run one IP at a slow pace and increase until you start seeing 429s or CAPTCHAs, then take roughly half of that as the safe pace per IP. IPs needed equals total requests per hour divided by safe requests per IP per hour, rounded up, plus a few spares.

A hypothetical example: you need 6,000 pages in 5 hours, which is 1,200 pages an hour. If one IP runs cleanly at around 200 pages an hour, you need 6 IPs, plus 1 or 2 spares for when one is resting. If the result is a large number, question the scope: do you really need to refetch every page each run? How many proxies do I need covers estimates for other tasks too.

Cut requests before adding IPs

• Use an official API or the sitemap when available instead of paging through category listings.
• Cache responses and send If-Modified-Since or ETag so you only refetch pages that changed.
• Fetch only what you need; skip images, CSS and fonts when you only want HTML.
• Schedule runs during the target site's quiet hours.
• Deduplicate jobs so the same URL is not fetched several times a day.

Proxies for controlled scraping

For public data, dedicated datacenter IPv4 proxies are usually the sensible choice: fast, unlimited bandwidth, and used only by you, so nobody else degrades their reputation. Pick the IP's country to match the content you need, since many sites show different prices, stock or results by location. See proxies for web scraping and the web scraping use case for setups.

A proxy is not a licence to scrape anything. Respect site terms, do not collect personal data unlawfully, and do not overload other people's servers.

Frequently asked questions