Indlela yokusetha nokushintshanisa ama-proxy ku-Python Scrapy
Updated:
Injini ye-Twisted ye-asynchronous ye-Scrapy ifaka i-HttpProxyMiddleware eyakhelwe ngaphakathi (ku-priority 750) ehlaziya izintambo ze-http://user:pass@ip:port ezinikezwe ku-request.meta['proxy']. Ukubhala i-Downloader Middleware yangokwezifiso yemigqa eyi-10 ku-priority 740 kuvumela ama-spider akho ukuthi ashintshanise ngokuzenzakalelayo phakathi kweqoqo lama-static IP angenamkhawulo e-VinaProxy.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP₫40,000
- 5+ ama-IP₫35,000
- 10+ ama-IP₫25,000
- 20+ ama-IP₫25,000
- 50+ ama-IP₫25,000
- 100+ ama-IP₫25,000
- 200+ ama-IP₫25,000
- 500+ ama-IP₫25,000
- 1000+ ama-IP₫22,000
- 2000+ ama-IP₫20,000
- 1 IP₫40,000
- 5+ ama-IP₫35,000
- 10+ ama-IP₫25,000
- 20+ ama-IP₫25,000
- 50+ ama-IP₫25,000
- 100+ ama-IP₫25,000
- 200+ ama-IP₫25,000
- 500+ ama-IP₫25,000
- 1000+ ama-IP₫22,000
- 2000+ ama-IP₫20,000
Ukuhleleka kwe-proxy middleware nezilungiselelo ze-concurrency ku-Scrapy
Ngenxa yokuthi i-HttpProxyMiddleware isebenza ku-oda 750 ukuze isuse i-user:pass ku-URL ye-proxy futhi ibethele i-header ye-Proxy-Authorization, noma iyiphi i-middleware yangokwezifiso esetha i-request.meta['proxy'] kumele isebenze enambeni ephansi (isib., 740):
| Isilungiselelo se-Scrapy | Inani Elinconywayo | Inhloso |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Iqondisa isicelo nge-proxy ebekiwe ye-VinaProxy |
| I-Priority ye-Middleware | 740 (ngaphambi kuka-750) | Iqinisekisa ukuthi i-HttpProxyMiddleware ibethela i-header ye-user:pass |
CONCURRENT_REQUESTS_PER_IP | 2 kuya ku-4 | Imisa umkhawulo wama-socket ngasikhathi sinye nge-IP ye-proxy ngayinye ukugwema i-HTTP 429 |
AUTOTHROTTLE_ENABLED | True | Ilungisa ngokuzenzakalelayo ukubambezeleka kokukhasa kususelwa ku-latency eqondiwe |
Izindlela ezinhle kakhulu zokukhasa kwe-Scrapy kwevolumu ephezulu
- Layisha uhlu lwakho lwama-proxy e-VinaProxy ku-
from_crawler()bese unikezarequest.meta['proxy'] = random.choice(self.proxies)ngaphakathi kwe-process_request() - Vula
RANDOMIZE_DOWNLOAD_DELAY = Trueeceleni kuka-DOWNLOAD_DELAY = 1.0ukuze izikhawu zesicelo nge-IP ngayinye zingabonakali njengezerobhothi - Lapho uzama futhi (
429noma503), khetha i-proxy ehlukile eqoqweni lakho ngaphambi kokubuyisela isicelo kulayini
Isinyathelo ngesinyathelo
- 1
Fometha iqoqo lakho lama-proxy njenge-http://user:pass@ip:port
Namathisela uhlu lwakho lwe-VinaProxy ku-Proxy Format Converter yethu ukuze uthole izintambo ze-URL.
- 2
Dala i-ProxyPoolMiddleware ku-middlewares.py
Chaza i-
process_request(self, request, spider)ukuze usethe i-request.meta['proxy'] = random.choice(self.proxies). - 3
Bhalisa ku-priority 740 ku-settings.py
Faka
'myproject.middlewares.ProxyPoolMiddleware': 740ngaphansi kwe-DOWNLOADER_MIDDLEWARESbese usetha i-CONCURRENT_REQUESTS_PER_IP = 2. - 4
Hlola nge-scrapy shell noma nge-scrapy crawl
Qalisa isicelo sokuhlola ku-
https://vinaproxy.com/api/my-ipukuze uqinisekise ukushintshana nokugunyazwa.
Imibuzo evame ukubuzwa
Kungani i-proxy middleware yangokwezifiso yami ye-Scrapy yehlulekile lapho isethelwe ku-priority 800?
I-HttpProxyMiddleware eyakhelwe ngaphakathi ku-Scrapy isebenza ku-priority 750. Uma i-middleware yakho isebenza ku-800 (ngemuva kuka-750), imininingwane ye-user:pass yangaphakathi ayiguqulwa ibe yi-header ye-Proxy-Authorization. Hlala usetha i-priority ye-middleware yakho ngaphansi kuka-750 (njengokuthi 740).
Ingabe i-Scrapy igcina ukuxhumana okuqhubekayo kuvuliwe ku-proxy ngayinye?
Yebo. Iqoqo lokuxhumana le-Twisted ligcina ukuxhumana okuqhubekayo kwe-keep-alive endaweni ngayinye yokugcina ye-proxy.
Ingabe ama-proxy e-VinaProxy awanamkhawulo emisebenzini ye-Scrapy yamakhasi ezigidi eziningi?
Yebo. Zonke izinhlelo ze-Dedicated IPv4 neze-Static ISP zifaka i-bandwidth yanyanga zonke engenamkhawulo.
Khulisa ama-spider akho e-Scrapy ngamaqoqo e-static IP angenamkhawulo
Ama-proxy amaningi asheshayo e-Dedicated IPv4 nawama-Static ISP emazweni ayi-7.
Thenga Ama-Proxy e-Scrapy