Sida loo habeeyo loona wareejiyo proxy-yada gudaha Python Scrapy
Updated:
Matoorka Twisted ee Scrapy ee asynchronous-ka ah waxa uu leeyahay HttpProxyMiddleware ku dhex jira (oo leh mudnaanta 750) kaas oo kala furfura qoraallada http://user:pass@ip:port ee loo qoondeeyay request.meta['proxy']. Qorista Downloader Middleware gaar ah oo 10 xariiq ah oo leh mudnaanta 740 waxay u oggolaanaysaa spiders-kaaga inay si toos ah ugu dhex wareegaan kaydka IP-yada ma guurtada ah ee aan xaddidnayn ee VinaProxy.
request.meta['proxy'] = 'http://user:pass@ip:port'- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
- 1 IP₫40,000
- 5+ IPs₫35,000
- 10+ IPs₫25,000
- 20+ IPs₫25,000
- 50+ IPs₫25,000
- 100+ IPs₫25,000
- 200+ IPs₫25,000
- 500+ IPs₫25,000
- 1000+ IPs₫22,000
- 2000+ IPs₫20,000
Kala-horreynta middleware-ka proxy-ga Scrapy & habeynta isku-mar-socodka
Maaddaama HttpProxyMiddleware uu ku shaqeeyo nidaamka 750 si uu uga saaro user:pass URL-ka proxy-ga ugana dhigo madaxa Proxy-Authorization mid encode ah, middleware kasta oo gaar ah oo dejinaya request.meta['proxy'] waa inuu ku shaqeeyaa lambar ka hooseeya (tusaale, 740):
| Habeynta Scrapy | Qiimaha Lagu Talinayo | Ujeeddada |
|---|---|---|
request.meta['proxy'] | 'http://user:pass@ip:port' | Waxay codsiga u marisaa proxy-ga VinaProxy ee la cayimay |
| Mudnaanta Middleware | 740 (ka hor 750) | Waxay hubisaa in HttpProxyMiddleware uu encode ku sameeyo madaxa user:pass |
CONCURRENT_REQUESTS_PER_IP | 2 ilaa 4 | Waxay xaddidaysaa isku-xirnaanta isku-mar-socodka ah ee halkii IP proxy si looga fogaado HTTP 429 |
AUTOTHROTTLE_ENABLED | True | Waxay si toos ah ula qabsataa dib-u-dhaca baarista iyadoo ku saleysan daahitaanka bartilmaameedka |
Hababka ugu wanaagsan ee baarista heerka sare ah ee Scrapy
- Ku shub liiskaaga proxy-yada VinaProxy gudaha
from_crawler()oo ku qoondeerequest.meta['proxy'] = random.choice(self.proxies)gudahaprocess_request() - Daar
RANDOMIZE_DOWNLOAD_DELAY = Trueoo ay wehelisoDOWNLOAD_DELAY = 1.0si waqtiyada u dhexeeya codsiyada halkii IP aysan marnaba ugu ekaan kuwo robotik ah - Marka dib la isku dayayo (
429ama503), ka dooro proxy kale kaydkaaga ka hor inta aanad codsiga dib safka u gelin
Tallaabo-tallaabo
- 1
U qaabee kaydkaaga proxy sida http://user:pass@ip:port
Ku dheji liiskaaga VinaProxy qalabkayaga Proxy Format Converter si aad u hesho qoraallada URL-ka.
- 2
Ku samee ProxyPoolMiddleware gudaha middlewares.py
Qeex
process_request(self, request, spider)si aad u dejisorequest.meta['proxy'] = random.choice(self.proxies). - 3
Ka diiwaangeli mudnaanta 740 gudaha settings.py
Ku dar
'myproject.middlewares.ProxyPoolMiddleware': 740qaybtaDOWNLOADER_MIDDLEWARESoo dejiCONCURRENT_REQUESTS_PER_IP = 2. - 4
Ku tijaabi scrapy shell ama scrapy crawl
Ku dhex wad codsi tijaabo ah
https://vinaproxy.com/api/my-ipsi aad u xaqiijiso wareegga iyo aqoonsiga.
Su'aalaha inta badan la isweydiiyo
Maxay u fashilantay middleware-ka gaarka ah ee Scrapy proxy markii la dhigay mudnaanta 800?
HttpProxyMiddleware ee ku dhex jira Scrapy waxa uu ku shaqeeyaa mudnaanta 750. Haddii middleware-kaagu uu ku shaqeeyo 800 (ka dib 750), xogta aqoonsiga ee user:pass marnaba looma beddelo madaxa Proxy-Authorization. Had iyo jeer deji mudnaanta middleware-kaaga mid ka hooseysa 750 (sida 740).
Scrapy ma sii hayaa isku-xirnaan joogto ah oo furan oo ku socota proxy kasta?
Haa. Kaydka isku-xirka ee Twisted waxa uu ilaaliyaa isku-xirnaan joogto ah oo keep-alive ah barta proxy kasta.
Proxy-yada VinaProxy ma yihiin kuwo aan xaddidnayn hawlaha Scrapy ee malaayiinta bog ah?
Haa. Dhammaan qorshayaasha Dedicated IPv4 iyo Static ISP waxaa ku jira xaddiga xogta bil kasta oo aan xaddidnayn.
Balaadhi spiders-kaaga Scrapy adoo adeegsanaya kaydka IP-yada ma guurtada ah ee aan xaddidnayn
Proxy-yo badan oo Dedicated IPv4 iyo Static ISP ah oo degdeg ah kana kala tirsan 7 waddan.
Iibso Proxy-yada Scrapy