Scrapy vs Playwright: Pick the Right Tool for Each Kind of Page

Updated: 2026-11-12

If the data you need is already in the HTML the server returns, use Scrapy: it is fast, light and handles thousands of pages with few resources. If the data only appears after JavaScript runs, or you need to scroll, click or sign in through the interface, use Playwright, because it drives a real browser. Many projects use both: Scrapy for most pages, Playwright for the few hard ones. This guide shows how to tell which kind of page you are dealing with, compares the two tools on practical criteria and covers proxy setup for each.

Dedicated proxies for Scrapy and Playwright

See pricing Free Proxy Checker

Check the target page before choosing

A common mistake is defaulting to a browser for everything because it always "sees" the page, then paying for it in speed and server resources. Spend a few minutes on the target first.

Fetch the page with curl or view source (Ctrl+U) and search for the data you want, such as a product name or price. If it is in the HTML, the page is static or server-rendered and Scrapy will do. If the HTML is nearly empty and the data only shows in the browser, open DevTools, filter the Network tab to Fetch/XHR and see which API the page calls. Many modern sites load data from a JSON endpoint; if it is public and you are allowed to use it, calling it from Scrapy is faster than rendering the whole page.

Scrapy and Playwright compared

Criterion	Scrapy	Playwright
What it is	Asynchronous scraping framework for Python	Library that drives Chromium, Firefox and WebKit
Runs JavaScript	No	Yes, like a real browser
Speed and resources	Very fast and light, many concurrent requests per machine	Slower, each page costs the CPU and RAM of a browser tab
UI interaction	None; it only sends HTTP requests	Click, type, scroll, wait for elements
Built in	Queueing, retries, throttling, export pipelines	Screenshots, tracing, network waits, several languages
Best for	Product listings, news, HTML pages, JSON APIs	SPAs, infinite scroll, flows that need a UI login

When Scrapy is the right call

Scrapy's strength is volume. You write a spider and Scrapy handles the URL queue, concurrency, retries, duplicate filtering and export to JSON or CSV. Settings such as CONCURRENT_REQUESTS_PER_DOMAIN, DOWNLOAD_DELAY and AutoThrottle keep you polite to the target without custom code.

Choose Scrapy when the data is in HTML or a public JSON API, when you need thousands of pages or more, and when the machine running it has limited resources. For a few dozen pages, requests plus BeautifulSoup may be simpler still.

When you need Playwright

Playwright is necessary when content is built by JavaScript after load, when data appears only on scroll or click, or when you must act like a user, for example applying filters. It is also handy for checking how a page renders in a given country, such as taking a screenshot through that country's proxy.

The cost is resources. Each browser context uses significant memory, so do not open hundreds of tabs at once. Block images, fonts and video with page.route when you only need data, and close contexts as soon as you are done. Automated browsers also carry traits that sites can spot; why headless browsers get detected explains them.

Proxy setup in Scrapy

Per request: yield scrapy.Request(url, meta={'proxy': 'http://user:pass@ip:port'}). Scrapy's built-in proxy middleware extracts the credentials and sends the auth header.
URL-encode passwords that contain special characters before building the string, or you will get 407 errors.
To rotate, write a small downloader middleware that sets request.meta['proxy'] from your list, or pins a proxy per account when sessions matter.
Full steps are in the Scrapy proxy setup guide.

Proxy setup in Playwright

For the whole browser: browser = p.chromium.launch(proxy={'server': 'http://ip:port', 'username': 'user', 'password': 'pass'}).
Per context, one IP each: context = browser.new_context(proxy={'server': 'http://ip:port', 'username': 'user', 'password': 'pass'}). Useful for running several sessions in one process.
Match the context's locale and timezone_id to the proxy's country so pages render with the right language and local time.
Before the job, open an IP echo page inside that context to confirm which IP is actually in use.
See the Playwright proxy setup guide; for Node.js, the Puppeteer guide works the same way.

Combining both, and choosing proxies

The scrapy-playwright package lets a Scrapy spider send most requests over plain HTTP and hand only JavaScript-heavy pages to a browser. You keep Scrapy's queue, retries and pipelines and spend browser resources only where needed.

For either tool, dedicated datacenter IPv4 proxies are usually enough for public data: fast, unlimited bandwidth and used only by you. Pick the country that matches the content; IPv4 is available in 7 countries including Vietnam, the United States and Japan. Whatever the tool, respect robots.txt and site terms, and keep a sensible pace as described in web scraping rate limits.

Frequently asked questions

Can Scrapy scrape React or Vue sites?

Only if the page is server-rendered or the data comes from a JSON API you can call directly. If the data exists only after JavaScript runs in a browser, use Playwright or scrapy-playwright.

Is Playwright much slower than Scrapy?

Usually, yes, because each page loads and runs as in a real browser. On the same machine, Scrapy handles far more pages concurrently.

Should each Playwright context have its own proxy?

Yes, if each context represents a session or account. Cookies and IP then stay together and no session jumps between IPs.

Do the proxies support SOCKS5 in Playwright?

VinaProxy proxies use HTTP/HTTPS with username and password authentication, which Playwright supports fully through the server, username and password options.

What about Selenium?

Selenium also drives a real browser and supports proxies; Playwright tends to be tidier for waiting on elements and authenticated proxies. Use the Selenium guide if your project already relies on it.