Crawlee

Crawlee is an open-source web crawling and browser automation library for Node.js.

It provides a high-level framework for building scalable crawlers that can work with both static and dynamic websites. Crawlee supports HTTP-based scraping as well as browser automation through tools such as Puppeteer and Playwright.

Also known as: Crawling framework, Crawlee.js.

How Does Crawlee Work?

Crawlee manages many of the tasks required in a web crawling workflow, including:

• URL queues
• Concurrent requests
• Session management
• Request retries
• Proxy rotation
• Data storage
• Browser automation

Developers can choose between lightweight HTTP-based crawlers and browser-based crawlers depending on how the target website loads its content.

Crawlee vs. Puppeteer

Crawlee and Puppeteer can both be used in web automation, but they operate at different levels.

Puppeteer primarily controls Chromium-based browsers and provides direct access to browser actions such as navigation, clicking, and JavaScript execution.

Crawlee can use Puppeteer while also providing additional crawling features such as URL queues, proxy handling, session management, retries, and concurrency control.

In simple terms:

• Puppeteer: Focuses on browser control.
• Crawlee: Manages complete crawling workflows.

Crawlee vs. Cheerio

Cheerio is a lightweight library for parsing and querying static HTML.

Crawlee provides a broader crawling framework and can use Cheerio for static pages while also supporting browser-based scraping for dynamic content.

Cheerio is useful when only HTML parsing is required, while Crawlee is better suited to workflows that involve multiple URLs, queues, sessions, and automated request management.

Advantages of Crawlee

• Scalable crawling: Supports request queues, concurrency, and large crawling workloads.
• Multiple scraping strategies: Can work with HTTP requests or headless browsers.
• Built-in session management: Helps maintain and reuse browsing sessions.
• Automatic retries: Failed requests can be retried according to configured rules.
• Proxy support: Proxy rotation can be integrated into crawling workflows.
• Developer-friendly APIs: Provides abstractions that reduce the amount of low-level crawling code developers need to write.

Limitations of Crawlee

• Higher resource usage: Browser-based crawlers can consume significant memory and CPU.
• Learning curve: Developers need to understand concepts such as request queues, sessions, and datasets.
• More complex than lightweight parsers: Crawlee may be unnecessary for simple one-page scraping tasks.
• Node.js focused: It is primarily designed for JavaScript and TypeScript environments.

Crawlee and Proxies

Crawlee includes support for proxy configuration and rotation.

A crawler can route requests through different proxy servers while Crawlee manages sessions, retries, and request queues.

This can be useful for workflows such as:

• Web scraping
• Market research
• Geo-specific testing
• Price monitoring
• Public web data collection

Proxy management is part of the network layer, while Crawlee controls the crawling logic and extraction workflow.

Example

The following example uses Crawlee's CheerioCrawler to retrieve page titles from multiple URLs:

import { CheerioCrawler } from 'crawlee';

const crawler = new CheerioCrawler({
  async requestHandler({ request, $ }) {
    const title = $('title').text();
    console.log(`Title of ${request.url}: ${title}`);
  },
});

await crawler.run([
  { url: 'https://example.com' },
  { url: 'https://another-site.com' },
]);

The crawler visits each URL, parses the returned HTML, extracts the <title> element, and prints the page title.