Bí a ṣe lè ṣètò àti yí àwọn proxy padà nínú Python Scrapy
Updated:
Ẹnjini asynchronous Twisted ti Scrapy pẹlu HttpProxyMiddleware ti a ṣe sinu rẹ (ni ipo pataki 750) ti o n ṣatupọ awọn ọrọ http://user:pass@ip:port ti a pin si request.meta['proxy']. Kikọ Downloader Middleware ti adani ti ila 10 ni ipo pataki 740 gba awọn spider rẹ laaye lati yipada laarin àkójọ awọn static IP VinaProxy ti ko ni opin laifọwọyi.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
Eto aṣẹ middleware proxy Scrapy & awọn eto concurrency
Nitori pe HttpProxyMiddleware n ṣiṣẹ ni aṣẹ 750 lati yọ user:pass kuro ninu URL proxy ati ki o fi koodu si akọsori Proxy-Authorization, eyikeyi middleware aṣa ti n ṣeto request.meta['proxy'] gbọdọ ṣiṣẹ ni nọmba kekere (fun apẹẹrẹ, 740):
| Eto Scrapy | Iye Ti A Daba | Idi |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | N dari ibeere naa nipasẹ proxy VinaProxy ti a sọ ni pato |
| Ipo Pataki Middleware | 740 (ṣaaju 750) | N ṣe idaniloju pe HttpProxyMiddleware n fi koodu si akọsori user:pass |
CONCURRENT_REQUESTS_PER_IP | 2 si 4 | N fi opin si awọn socket igbakanna fun IP proxy kọọkan lati yago fun HTTP 429 |
AUTOTHROTTLE_ENABLED | True | N ṣatunṣe idaduro crawl laifọwọyi da lori latency ibi-afẹde |
Awọn iṣe ti o dara julọ fun awọn crawl Scrapy iwọn nla
- Gbe atokọ proxy VinaProxy rẹ sinu
from_crawler()ki o si firequest.meta['proxy'] = random.choice(self.proxies)si inuprocess_request() - Muu
RANDOMIZE_DOWNLOAD_DELAY = Trueṣiṣẹ pẹluDOWNLOAD_DELAY = 1.0ki awọn aarin ibeere fun IP kọọkan ma ba dabi ti roboti - Nigbati o ba tun gbiyanju (
429tabi503), mu proxy ti o yatọ lati inu àkójọ rẹ ṣaaju ki o to tun fi ibeere naa sinu laini
Ìgbésẹ̀ ní ìgbésẹ̀
- 1
Ṣeto àkójọ proxy rẹ gẹgẹbi http://user:pass@ip:port
Lẹ mọ atokọ VinaProxy rẹ sinu Proxy Format Converter wa lati gba awọn okun URL.
- 2
Ṣẹda ProxyPoolMiddleware ninu middlewares.py
Ṣalaye
process_request(self, request, spider)lati ṣetorequest.meta['proxy'] = random.choice(self.proxies). - 3
Forukọsilẹ ni ipo pataki 740 ninu settings.py
Fi
'myproject.middlewares.ProxyPoolMiddleware': 740kun labẹDOWNLOADER_MIDDLEWARESki o si ṣetoCONCURRENT_REQUESTS_PER_IP = 2. - 4
Ṣe idanwo pẹlu scrapy shell tabi scrapy crawl
Ṣiṣe ibeere idanwo si
https://vinaproxy.com/api/my-iplati rii daju yiyipada ati ijẹrisi.
Àwọn ìbéèrè tí a ń béèrè lemọ́lemọ́
Kini idi ti middleware proxy Scrapy aṣa mi fi kuna nigbati a ṣeto si ipo pataki 800?
HttpProxyMiddleware ti a ṣe sinu Scrapy n ṣiṣẹ ni ipo pataki 750. Ti middleware rẹ ba n ṣiṣẹ ni 800 (lẹhin 750), awọn iwe-ẹri user:pass ti o wa ninu ila kii yoo yipada si akọsori Proxy-Authorization. Ṣeto ipo pataki middleware rẹ ni isalẹ 750 nigbagbogbo (bii 740).
Ṣe Scrapy n tọju awọn asopọ ti o tẹsiwaju ni ṣiṣi si proxy kọọkan?
Bẹẹni. Àkójọ asopọ Twisted n ṣetọju awọn asopọ keep-alive ti o tẹsiwaju fun opin proxy kọọkan.
Ṣe awọn proxy VinaProxy ko ni opin fun awọn iṣẹ Scrapy ti awọn oju-iwe miliọnu pupọ?
Bẹẹni. Gbogbo awọn eto Dedicated IPv4 ati Static ISP pẹlu iwọn bandiwidi oṣooṣu ailopin.
Fẹ awọn spider Scrapy rẹ pọ pẹlu awọn àkójọ static IP ti ko ni opin
Awọn proxy Dedicated IPv4 ati Static ISP olopobobo lẹsẹkẹsẹ kọja awọn orilẹ-ede 7.
Ra Awọn Proxy Scrapy