Scraping Google SERPs for Rank Tracking: Doing It Sensibly
Updated:
If you want to scrape Google results to track rankings, start with two facts: Google's terms restrict automated queries, and its systems for blocking automated traffic are strong. The durable route is official data first: Google Search Console for your own site, licensed search APIs, or a dedicated rank tracking tool. Country proxies are most useful for manual and small-scale checks at a low frequency, so you see the same results as users in Vietnam, the US or Japan. This guide helps you choose the data source, frequency, country and setup.
What do you actually need SERP data for?
Before writing any code, define the question. Tracking your own site for a few hundred keywords is one problem; researching competitors across markets is another; checking how ads or featured snippets appear in a given country is a third. Each fits a different data source.
Many people jump into scraping Google daily for thousands of keywords and end up fighting CAPTCHAs and blocks, when most of their questions could be answered from official data or periodic manual checks.
Data sources, in order of preference
Read each service's terms and limits before relying on it; features and costs of these platforms change over time.
| Source | What it tells you | Use it when |
|---|---|---|
| Google Search Console | Queries, average position, impressions and clicks for your site, filterable by country | Tracking your own site; always the first source |
| Licensed search APIs | Search results per query through an official channel or data provider | You need structured data at volume, within the terms |
| Rank tracking tools | Rankings by keyword, country and device, with history | Hundreds of keywords across several projects |
| Manual checks through a country proxy | The real results page users in that country see | Periodic checks of SERP features, ads and local results |
Choose country and language correctly
Google results vary by location, language and device. Searching an English keyword from Vietnam does not show what a US user sees. A country proxy fixes the location part: the query leaves from an IP in that country.
Beyond the IP, the search URL accepts hl (interface language) and gl (preferred country), for example hl=en&gl=us or hl=vi&gl=vn. Use a matching IP and parameters in a clean browser that is not signed in to Google, so your own history does not personalise results. Checking Google results by country goes deeper into manual checks.
- Vietnamese market: a Vietnam IP with
hl=vi&gl=vn. - US market: a US IP with
hl=en&gl=us. - Other markets: IPv4 is available in Japan, South Korea, Singapore and Hong Kong, plus the UK.
- City-level results do not depend on the IP alone, and VinaProxy proxies do not offer city or state selection.
How often is often enough?
Organic rankings rarely change meaningfully hour by hour. For most projects, weekly checks show the trend, with daily checks for the most important keywords or right after a content update. Checking more often mostly produces noise and raises the risk of being blocked.
If you run small checks through a proxy, keep the pace low: a few queries, well spaced, no parallel threads on one IP, and stop when you hit a verification page instead of pushing on. Web scraping rate limits and proxies and CAPTCHA explain why speed backfires.
Set up manual SERP checks through a proxy
- Create one browser profile per country, not signed in to Google, with that country's proxy. Setup is covered in the Chrome proxy guide or SwitchyOmega guide.
- Set browser language and timezone to match the IP's country.
- Open what is my IP in the profile to confirm the IP and country.
- Search with the right
hlandgl, and record position, SERP features (featured snippets, People Also Ask, maps, shopping) and ads. - Save dated screenshots to compare week to week.
- When the keyword list grows, move to a rank tracker or licensed API rather than speeding up manual checks.
Which proxies suit SERP checks
SERP checks do not involve logging in, so a dedicated datacenter IPv4 is usually enough: fast, unlimited bandwidth, the right country and used only by you. Dedicated matters, because shared or free proxies are often already flagged by Google after other people's automated queries, and you get a verification page on the first search.
If your rank tracker supports custom proxies, fixed IPv4 addresses keep results consistent between checks. See proxies for SEO rank tracking and the SERP tracking use case.
Frequently asked questions
Does scraping Google break its terms?
Google's terms restrict automated queries. For your own site's data, use Search Console. For large-scale data, use a licensed API or data provider and read their terms carefully.
Why do my results differ from my rank tracker?
Location, language, device and account personalisation all differ. Check with a clean, signed-out profile using a proxy and hl and gl for the right country.
Can I pick a city proxy for local results?
VinaProxy proxies are chosen by country, not by city or state. For city-level local results, use a rank tracker that supports detailed locations.
IPv4 or ISP proxies for SERP checks?
Private IPv4 is usually enough since you are not logging in. Consider ISP proxies only if you keep hitting verification despite a slow pace and a dedicated IP.
How often should I check rankings?
Weekly for most keywords, daily for a priority group or after content updates. More frequent checks mostly add noise.
See Google as users in 7 countries do
Private IPv4 proxies in Vietnam, the US, Japan, South Korea, Singapore, the UK and Hong Kong. Fixed IPs, unlimited bandwidth, automatic activation.
See pricing