Competitor Price Monitoring with Proxies: Frequency, IPs and Ethics
Updated:
Monitoring competitor prices on marketplaces sustainably comes down to three decisions: how often to check (based on how fast prices move in your category, not as often as possible), how many IPs to use (worked out from the pages you need and the rate you allow each IP), and what level of collection is acceptable (public data only, respecting the site's terms, never overloading it). Private IPv4 proxies are the usual choice for this because they are fast, stable and country-specific. This guide goes step by step with a concrete way to do the maths.
When you need a proxy for price monitoring
If you track a few dozen products by hand, you do not need a proxy. You need one once you automate: a script loading hundreds or thousands of product pages a day from one IP quickly hits the site's rate limits, gets 429 Too Many Requests errors or is blocked for a while.
The second reason is regional pricing. Many marketplaces show different prices, promotions and shipping by country. To know what a competitor charges in the US, you need to look from a US IP. VinaProxy offers private IPv4 in 7 countries, including Vietnam and the US.
Choosing a check frequency
Frequency should follow how quickly prices change in your category, not what your script can handle. Checking more often than necessary wastes your resources and the target site's without adding information.
| Category or situation | Suggested frequency | Note |
|---|---|---|
| Stable goods, few promotions | Once a day | Enough to see trends |
| Electronics, cosmetics, highly competitive goods | A few times a day | Focus on your key products |
| Shopping festivals, flash sales | More often during the event window | Only for a short product list |
| Stock and availability tracking | As the business needs | Prefer an official API or data feed if one exists |
Sizing your IP pool
The simple formula: total pages to load per day divided by the pages you let each IP load per day at a polite pace. As an illustration, tracking 1,000 products four times a day means 4,000 page loads. If you cap each IP at one page every 15 seconds and only run for 8 hours, each IP handles roughly 1,900 pages. So 3 IPs are enough, plus one spare.
The per-IP cap is yours to set, so start conservatively and adjust based on what you observe. If 429s or verification pages start appearing, slow down before adding IPs. Web scraping rate limits and how many proxies do I need go deeper into estimation.
- Separate IPs by country: one group per market.
- Spread requests evenly across IPs rather than piling onto one.
- Log response codes per IP to spot the ones being limited.
- ISP proxies are unnecessary here; private IPv4 is faster and more economical.
Configuring the proxy in code
VinaProxy proxies are HTTP/HTTPS with username and password in ip:port:user:pass format. In Python with requests, pass the proxy as http://user:pass@ip:port for both the http and https keys. Full guides are on the Python requests setup and Scrapy setup pages. To cycle through a list of static IPs, see rotating static proxies in Python.
- Add a random delay between requests instead of firing continuously.
- Set sensible timeouts and limited retries with increasing back-off.
- Cache results: do not reload pages that are not due for a check.
- Extract only the fields you need, such as name, price, promotion and stock status.
- Check the whole IP list periodically with the proxy checker.
Ethical and legal limits
Monitoring public prices is common business practice, but how you collect still matters. Proxies spread load; they are not for defeating barriers a site deliberately puts up.
- Public data only: no logging in to reach data behind a login wall, no collecting personal data about buyers or sellers.
- Read the terms and robots.txt; if the marketplace offers an API or official data programme, use it first.
- Do not overload: keep the rate low and run outside peak hours where possible.
- Do not bypass CAPTCHAs: a verification challenge is a signal to slow down.
- Use the data appropriately: analyse prices for your own decisions, do not copy competitors' content or images.
Turning price data into decisions
Data only has value when you act on it. Store price history over time rather than just the latest price, so you can see when competitors discount and how they react to promotions. Set alerts for when a competitor drops below you by a threshold you choose, instead of reading tables every day.
Combine price data with your own costs and margins before changing prices. Chasing every competitor discount usually costs more profit than it saves in customers. For the wider picture on data collection, see proxies for web scraping and the web scraping use case.
Frequently asked questions
Do I need a proxy to monitor competitor prices?
Not for checking a few products by hand. Once you automate many pages a day or need country-specific prices, proxies spread the load and show the right market.
Private IPv4 or ISP for price monitoring?
Private IPv4 is enough and more economical. Price monitoring does not log into accounts, so you only need fast, stable IPs in the right country.
How many IPs do I need for 1,000 products?
It depends on frequency and the rate you allow each IP. At a few checks a day with a cautious rate, a handful of IPs is usually enough. Divide total page loads by loads per IP.
What should I do about 429 errors?
Slow down first, add pauses between requests and keep retries limited. Add IPs only if a reasonable rate still is not enough.
Is price monitoring legal?
Collecting public prices at a reasonable rate is common practice, but rules vary by jurisdiction and site terms. Read the terms and get legal advice if you operate at scale.
Private IPv4 for price monitoring
Private IPv4 in 7 countries, fast and with unlimited bandwidth, in ip:port:user:pass format ready to drop into your code.
View pricing