How to set up and rotate proxies in Python Scrapy
Updated:
Scrapy's asynchronous Twisted engine includes a built-in HttpProxyMiddleware (at priority 750) that parses http://user:pass@ip:port strings assigned to request.meta['proxy']. Writing a 10-line custom Downloader Middleware at priority 740 lets your spiders rotate across a pool of unmetered VinaProxy static IPs automatically.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
Scrapy proxy middleware order & concurrency settings
Because HttpProxyMiddleware runs at order 750 to strip user:pass from the proxy URL and encode the Proxy-Authorization header, any custom middleware setting request.meta['proxy'] must run at a lower number (e.g., 740):
| Scrapy Setting | Recommended Value | Purpose |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Routes the request through the specified VinaProxy proxy |
| Middleware Priority | 740 (before 750) | Ensures HttpProxyMiddleware encodes the user:pass header |
CONCURRENT_REQUESTS_PER_IP | 2 to 4 | Caps simultaneous sockets per proxy IP to avoid HTTP 429 |
AUTOTHROTTLE_ENABLED | True | Dynamically adapts crawl delay based on target latency |
Best practices for high-volume Scrapy crawls
- Load your VinaProxy proxy list in
from_crawler()and assignrequest.meta['proxy'] = random.choice(self.proxies)insideprocess_request() - Enable
RANDOMIZE_DOWNLOAD_DELAY = TruealongsideDOWNLOAD_DELAY = 1.0so request intervals per IP never look robotic - On retry (
429or503), pick a different proxy from your pool before re-queueing the request
Step by step
- 1
Format your proxy pool as http://user:pass@ip:port
Paste your VinaProxy list into our Proxy Format Converter to get URL strings.
- 2
Create a ProxyPoolMiddleware in middlewares.py
Define
process_request(self, request, spider)to setrequest.meta['proxy'] = random.choice(self.proxies). - 3
Register at priority 740 in settings.py
Add
'myproject.middlewares.ProxyPoolMiddleware': 740underDOWNLOADER_MIDDLEWARESand setCONCURRENT_REQUESTS_PER_IP = 2. - 4
Test with scrapy shell or scrapy crawl
Run a test request against
https://vinaproxy.com/api/my-ipto verify rotation and authentication.
Frequently asked questions
Why did my custom Scrapy proxy middleware fail when set to priority 800?
Scrapy's built-in HttpProxyMiddleware runs at priority 750. If your middleware runs at 800 (after 750), the inline user:pass credentials are never converted into a Proxy-Authorization header. Always set your middleware priority below 750 (such as 740).
Does Scrapy keep persistent connections open to each proxy?
Yes. Twisted's connection pool maintains persistent keep-alive connections per proxy endpoint.
Are VinaProxy proxies unmetered for multi-million-page Scrapy jobs?
Yes. All Dedicated IPv4 and Static ISP plans include unlimited monthly bandwidth.
Scale your Scrapy spiders with unmetered static IP pools
Instant bulk Dedicated IPv4 and Static ISP proxies across 7 countries.
Buy Scrapy Proxies