Web Scraping with a Headless Browser

There are various aspects of algorithms from browser history, browser fingerprinting, and of course using web proxies to scrape the web undetected. One of the most reliable solutions is to use a headless browser — and Puppeteer is one of the best tools for the job.

When we want to scrape websites at scale we have to think about how we can avoid getting detected and blocked. Simply fetching HTML with a standard HTTP client is fast and cheap, but it will get you blocked the moment a site checks your headers or missing JavaScript execution.

What is Puppeteer?

Puppeteer is a Node.js library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer runs headless by default, but can be configured to run full (non-headless) Chrome or Chromium. It gives you a real browser environment — cookies, JavaScript, local storage — everything a normal user session has.

The web ecosystem is incredibly powerful for web scraping. By appearing as a real browser process we can circumvent the most common fingerprinting methods used to block bots. Unlike simple HTTP requests, Puppeteer renders the full page including dynamically loaded JavaScript content.

• Renders JavaScript-heavy single-page applications
• Handles cookies, sessions and authentication flows
• Supports screenshots and PDF generation
• Can intercept and modify network requests
• Full DevTools Protocol access

The Basics

Getting started with Puppeteer is straightforward. Install it via npm and you have everything you need — Puppeteer bundles a compatible version of Chromium by default so there is no manual browser management required.

npm install puppeteer

Here is a minimal working example that navigates to a page, waits for content to load, and extracts data:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({
    headless: true,
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

  const page = await browser.newPage();

  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' +
    'AppleWebKit/537.36 (KHTML, like Gecko) ' +
    'Chrome/124.0.0.0 Safari/537.36'
  );

  await page.goto('https://example.com', {
    waitUntil: 'networkidle2',
  });

  const title = await page.title();
  console.log('Page title:', title);

  await browser.close();
})();

Always respect a website’s robots.txt and terms of service before scraping. Use this knowledge responsibly.

Scraping Optimization / Scaling — Take it up a Notch

Running a single browser instance works for small jobs, but once you need to process hundreds or thousands of pages you need to think about concurrency, resource usage, and how to avoid bans at scale.

Server-Side Optimizations

The biggest performance gains come from blocking resources you do not need — images, fonts, and stylesheets add latency without contributing to the data you are extracting.

await page.setRequestInterception(true);

page.on('request', (request) => {
  const blockedTypes = ['image', 'stylesheet', 'font', 'media'];
  if (blockedTypes.includes(request.resourceType())) {
    request.abort();
  } else {
    request.continue();
  }
});

Running Multiple Pages in Parallel

Instead of opening a new browser per URL, reuse a single browser and open multiple pages concurrently. Limit the concurrency to avoid memory exhaustion — a good starting point is one page per CPU core.

const pLimit = require('p-limit');
const limit  = pLimit(4);

const urls = ['https://example.com/page-1', 'https://example.com/page-2'];

const browser = await puppeteer.launch({ headless: true });

const results = await Promise.all(
  urls.map((url) =>
    limit(async () => {
      const page = await browser.newPage();
      await page.goto(url, { waitUntil: 'networkidle2' });
      const data = await page.evaluate(() => document.title);
      await page.close();
      return data;
    })
  )
);

await browser.close();

Rotating Proxies and User Agents

For large-scale scraping, rotate both your IP (via proxy) and your user-agent string on each request. This distributes your traffic across many identities and dramatically reduces the chance of a ban.

const proxies = [
  'http://proxy1.example.com:8080',
  'http://proxy2.example.com:8080',
];

const userAgents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15...',
];

const proxy     = proxies[Math.floor(Math.random() * proxies.length)];
const userAgent = userAgents[Math.floor(Math.random() * userAgents.length)];

const browser = await puppeteer.launch({
  headless: true,
  args: [`--proxy-server=${proxy}`],
});

const page = await browser.newPage();
await page.setUserAgent(userAgent);

Summary

Start with the basics, measure where your bottleneck is, then layer in optimisations one at a time. Happy scraping!

Typography Scale

All heading levels, inline styles, and text-level elements shown below for styling reference.

H1 — The Definitive Guide to Headless Browsing

H2 — Puppeteer, Playwright & Selenium Compared

H3 — Installing and Configuring Your Scraper

H4 — Handling Dynamic Content with waitForSelector

H5 — Network Idle vs DOM Content Loaded

H6 — A note on headless detection flags

Paragraph with bold text, italic text, strikethrough, inline code, a hyperlink, coloured text, and ^superscript and _subscript all in one run of prose to test inline typographic rhythm.

Ordered List

A step-by-step installation walkthrough demonstrating the ordered list block:

1. Install Node.js 18 LTS or later from nodejs.org
2. Create a new project directory and initialise it with npm init -y
3. Run npm install puppeteer — this downloads Chromium automatically
4. Create index.js and paste the minimal example from The Basics section above
5. Run node index.js and verify the page title is printed to the console
6. Add request interception, proxy rotation, and concurrency as your needs grow

Three-Column Layout

Comparing the three most popular headless browser libraries side by side:

Group Block (Callout / Highlight Box)

Buttons

All default Gutenberg button styles for reference — primary, outline, and link variants:

Details / Accordion Block

Frequently asked questions about web scraping, using the native HTML <details> element:

Preformatted Text

Raw terminal output from a scraping run — using wp:preformatted which preserves whitespace without syntax highlighting:

$ node scraper.js

[10:42:01] Starting browser...
[10:42:02] Navigating to https://example.com/products
[10:42:04] Page loaded — 143 product cards found
[10:42:04] Extracting: Product 1 of 143 — "Wireless Headphones Pro" £89.99
[10:42:04] Extracting: Product 2 of 143 — "USB-C Hub 7-Port"         £34.99
[10:42:04] Extracting: Product 3 of 143 — "Mechanical Keyboard TKL"  £129.99
...
[10:42:09] Extraction complete. 143 records written to products.json
[10:42:09] Browser closed. Total time: 7.8s

Verse Block

The wp:verse block preserves line breaks and whitespace — useful for poetry, ASCII diagrams, or structured plain text:

The spider crawls the web at night,
  no JavaScript left unrendered,
    no lazy-loaded image undetected.

It does not knock. It does not ask.
  It simply reads — politely, quietly —
    and moves on to the next page.

Image Block

Gallery Block

Three-image gallery demonstrating the grid layout block:

Cover Block

Ready to Build Your Own Scraper?

Start with a single Puppeteer script and scale to a distributed cluster — the architecture is the same, only the infrastructure grows.

Start Building

Media & Text Block

How Puppeteer Communicates with Chrome

Puppeteer talks to a running Chrome (or Chromium) process over a WebSocket connection using the Chrome DevTools Protocol (CDP). Every call you make — page.goto(), page.click(), page.evaluate() — translates into a CDP command sent over that socket.

This is fundamentally different from older tools like Selenium WebDriver, which added multiple abstraction layers between your test code and the actual browser process. The direct CDP connection is what makes Puppeteer fast, precise, and capable of things WebDriver simply cannot do — like intercepting and modifying network requests in flight.