Yadda ake saitawa da juya proxies a Python Scrapy
Updated:
Injin Twisted na Scrapy mai aiki a lokaci guda yana ɗauke da HttpProxyMiddleware na asali (a matsayin fifiko 750) wanda ke karanta rubutun http://user:pass@ip:port da aka sanya wa request.meta['proxy']. Rubuta Downloader Middleware na musamman mai layuka 10 a matsayin fifiko 740 zai ba spiders ɗinka damar juyawa ta atomatik a cikin rukunin IP na static marasa iyaka na VinaProxy.
request.meta['proxy'] = 'http://user:pass@ip:port'- IP 1₫ 40,000
- IPs 5+₫ 35,000
- IPs 10+₫ 25,000
- IPs 20+₫ 25,000
- IPs 50+₫ 25,000
- IPs 100+₫ 25,000
- IPs 200+₫ 25,000
- IPs 500+₫ 25,000
- IPs 1000+₫ 22,000
- IPs 2000+₫ 20,000
- IP 1₫ 40,000
- IPs 5+₫ 35,000
- IPs 10+₫ 25,000
- IPs 20+₫ 25,000
- IPs 50+₫ 25,000
- IPs 100+₫ 25,000
- IPs 200+₫ 25,000
- IPs 500+₫ 25,000
- IPs 1000+₫ 22,000
- IPs 2000+₫ 20,000
Tsarin fifikon middleware da saitunan concurrency na Scrapy proxy
Saboda HttpProxyMiddleware yana aiki ne a lambar fifiko 750 domin cire user:pass daga URL na proxy tare da sanya bayanan Proxy-Authorization a header, duk wani middleware na musamman da ke saita request.meta['proxy'] dole ne ya yi aiki a ƙaramar lamba (misali, 740):
| Saitin Scrapy | Ƙimar da Aka Ba da Shawara | Manufa |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Yana tura buƙatar ta cikin takamaiman proxy na VinaProxy |
| Matsayin Fifiko na Middleware | 740 (kafin 750) | Yana tabbatar da cewa HttpProxyMiddleware ya sanya bayanan user:pass a header |
CONCURRENT_REQUESTS_PER_IP | 2 zuwa 4 | Yana iyakance haɗin kai tsaye a kowane IP na proxy don kauce wa kuskuren HTTP 429 |
AUTOTHROTTLE_ENABLED | True | Yana daidaita jinkirin crawl ta atomatik bisa ga saurin amsa na shafin |
Hanyoyi mafi kyau don gudanar da manyan ayyukan crawl a Scrapy
- Shigar da jerin proxy na VinaProxy ɗinka a cikin
from_crawler()sannan ka sanyarequest.meta['proxy'] = random.choice(self.proxies)a cikinprocess_request() - Kunna
RANDOMIZE_DOWNLOAD_DELAY = Truetare daDOWNLOAD_DELAY = 1.0domin tazara tsakanin buƙatu a kowace IP ba zai yi kama da na na'ura ba - Idan ana sake gwadawa (
429ko503), zaɓi wani proxy daban daga rukuninka kafin sake tura buƙatar a jerin gwano
Mataki-mataki
- 1
Sanya rukunin proxy ɗinka a tsarin http://user:pass@ip:port
Manna jerin VinaProxy ɗinka a cikin Proxy Format Converter don samun tsarin URL.
- 2
Ƙirƙiri ProxyPoolMiddleware a cikin middlewares.py
Ƙayyade
process_request(self, request, spider)domin saitarequest.meta['proxy'] = random.choice(self.proxies). - 3
Yi rajista a matsayin fifiko 740 a settings.py
Ƙara
'myproject.middlewares.ProxyPoolMiddleware': 740a ƙarƙashinDOWNLOADER_MIDDLEWARESkuma saitaCONCURRENT_REQUESTS_PER_IP = 2. - 4
Gwada da scrapy shell ko scrapy crawl
Gudanar da buƙatar gwaji zuwa
https://vinaproxy.com/api/my-ipdomin tabbatar da juyawa da kuma shiga cikin nasara.
Tambayoyin da aka fi yi
Me ya sa middleware na proxy na Scrapy na musamman ya gaza lokacin da aka saita shi a fifiko 800?
HttpProxyMiddleware na asali na Scrapy yana aiki ne a matsayin fifiko 750. Idan middleware ɗinka yana aiki a 800 (bayan 750), bayanan user:pass ba za a taɓa canza su zuwa header na Proxy-Authorization ba. Koyaushe saita fifikon middleware ɗinka ƙasa da 750 (kamar 740).
Shin Scrapy yana ci gaba da buɗe haɗin kai tsaye zuwa kowane proxy?
E. Rukunin haɗin Twisted yana kula da haɗin keep-alive mai dorewa ga kowane adireshin proxy.
Shin proxies na VinaProxy ba su da iyaka don manyan ayyukan Scrapy na miliyoyin shafuka?
E. Dukkan tsare-tsaren Dedicated IPv4 da Static ISP sun haɗa da bandwidth na wata-wata marar iyaka.
Faɗaɗa spiders ɗinka na Scrapy tare da rukunin IP na static marasa iyaka
Sami Dedicated IPv4 da Static ISP proxies masu yawa nan take a cikin ƙasashe 7.
Sayi Proxies na Scrapy