Uburyo bwo gushyiraho no guhinduranya porogisi muri Python Scrapy
Updated:
Moteri ya Scrapy idategereza ya Twisted ikubiyemo HttpProxyMiddleware isanzwemo (ku cyiciro cya 750) isoma imirongo ya http://user:pass@ip:port yashyizwe muri request.meta['proxy']. Kwandika Downloader Middleware yawe bwite y'imirongo 10 ku cyiciro cya 740 bituma spiders zawe zihinduranya mu buryo bwikora mu matsinda ya IP zidahinduka zidapimirwa umurongo za VinaProxy.
request.meta['proxy'] = 'http://user:pass@ip:port'- IP 1₫ 40.000
- IP 5+₫ 35.000
- IP 10+₫ 25.000
- IP 20+₫ 25.000
- IP 50+₫ 25.000
- IP 100+₫ 25.000
- IP 200+₫ 25.000
- IP 500+₫ 25.000
- IP 1000+₫ 22.000
- IP 2000+₫ 20.000
- IP 1₫ 40.000
- IP 5+₫ 35.000
- IP 10+₫ 25.000
- IP 20+₫ 25.000
- IP 50+₫ 25.000
- IP 100+₫ 25.000
- IP 200+₫ 25.000
- IP 500+₫ 25.000
- IP 1000+₫ 22.000
- IP 2000+₫ 20.000
Urukurikirane rwa middleware ya porogisi muri Scrapy n'igenamiterere rya concurrency
Kubera ko HttpProxyMiddleware ikorera ku murongo wa 750 kugira ngo ikureho user:pass kuri URL ya porogisi kandi ikode umutwe wa Proxy-Authorization, middleware iyo ari yo yose yihariye ishyiraho request.meta['proxy'] igomba gukorera ku mubare muto (urugero, 740):
| Igenamiterere rya Scrapy | Agaciro gasabwa | Intego |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Yohereza icyifuzo binyuze muri porogisi ya VinaProxy yagaragajwe |
| Priority ya Middleware | 740 (mbere ya 750) | Iremeza ko HttpProxyMiddleware ikoda umutwe wa user:pass |
CONCURRENT_REQUESTS_PER_IP | 2 kugeza kuri 4 | Igarukira ku masoketi y'icyarimwe kuri buri IP ya porogisi kugira ngo hirindwe HTTP 429 |
AUTOTHROTTLE_ENABLED | True | Ihindura mu buryo bwikora igihe cyo gutegereza cyo gushakisha hashingiwe ku gutinda kw'intego |
Inama nziza zo gukoresha Scrapy ku bwinshi bw'amakuru
- Shyira urutonde rwawe rwa porogisi za VinaProxy muri
from_crawler()hanyuma ushyirehorequest.meta['proxy'] = random.choice(self.proxies)imbere muriprocess_request() - Fungura
RANDOMIZE_DOWNLOAD_DELAY = Truehamwe naDOWNLOAD_DELAY = 1.0kugira ngo intera z'ubusabe kuri buri IP zitagaragara nk'izakozwe na robo - Mu kongera kugerageza (
429cyangwa503), hitamo porogisi itandukanye mu itsinda ryawe mbere yo kongera gushyira icyifuzo ku murongo
Intambwe ku yindi
- 1
Tunganya itsinda ryawe rya porogisi nka http://user:pass@ip:port
Koporora ushyire urutonde rwawe rwa VinaProxy muri Proxy Format Converter yacu kugira ngo ubone imirongo ya URL.
- 2
Kora ProxyPoolMiddleware muri middlewares.py
Sobanura
process_request(self, request, spider)kugira ngo ushyirehorequest.meta['proxy'] = random.choice(self.proxies). - 3
Iyandikishe kuri priority 740 muri settings.py
Ongeraho
'myproject.middlewares.ProxyPoolMiddleware': 740munsi yaDOWNLOADER_MIDDLEWAREShanyuma ushyirehoCONCURRENT_REQUESTS_PER_IP = 2. - 4
Gerageza ukoresheje scrapy shell cyangwa scrapy crawl
Kora icyifuzo cy'igerageza kuri
https://vinaproxy.com/api/my-ipkugira ngo ugenzure ihinduranya n'iyemeza ry'umwirondoro.
Ibibazo bikunze kubazwa
Kuki middleware yanjye yihariye ya porogisi muri Scrapy yananiwe igihe yashyirwaga kuri priority 800?
HttpProxyMiddleware isanzwemo muri Scrapy ikorera kuri priority 750. Niba middleware yawe ikorera kuri 800 (nyuma ya 750), imyirondoro ya user:pass iri ku murongo ntiyigera ihindurwamo umutwe wa Proxy-Authorization. Buri gihe shyira priority ya middleware yawe munsi ya 750 (nka 740).
Ese Scrapy igumana imiyoboro ihoraho ifunguye kuri buri porogisi?
Yego. Itsinda ry'imiyoboro rya Twisted rigumana imiyoboro ihoraho ya keep-alive kuri buri mpera ya porogisi.
Ese porogisi za VinaProxy ntizipimirwa umurongo ku mirimo ya Scrapy y'amapaji miriyoni nyinshi?
Yego. Gahunda zose za Dedicated IPv4 na Static ISP zikubiyemo umurongo wa interineti utagira umupaka wa buri kwezi.
Agura spiders zawe za Scrapy ukoresheje amatsinda ya IP zidahinduka zidapimirwa umurongo
Porogisi z'akazi kenshi z'ako kanya za Dedicated IPv4 na Static ISP mu bihugu 7.
Gura Porogisi za Scrapy