crawlee

Expert guide for building web scrapers and crawlers using Crawlee (JavaScript/TypeScript and Python). Use this skill whenever the user wants to: scrape a website, build a web crawler, extract data from web pages, automate browser navigation, handle anti-bot blocking, manage proxies or sessions for scraping, use Playwright/Puppeteer/Cheerio/BeautifulSoup for web data extraction, crawl sitemaps, download files from URLs, or deploy a scraper to the cloud. Trigger even for loosely related phrases like "get data from a website", "automate browser", "scrape prices", "extract links", "crawl URLs", or "bypass bot detection". Covers CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler, HttpCrawler, JSDOMCrawler (JS), and BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler (Python).

# Crawlee Skill

Crawlee is a production-grade web scraping and browser automation library for **JavaScript/TypeScript** (Node.js 16+)
and **Python** (3.10+). It handles anti-blocking, proxies, session management, storage, and concurrency out of the box.

> **Docs**: https://crawlee.dev/js/docs | https://crawlee.dev/python/docs  
> **GitHub**: https://github.com/apify/crawlee

---

## 1. Choose Your Crawler

### JavaScript / TypeScript

| Crawler | When to Use | JS Required |
|---|---|---|
| `CheerioCrawler` | Fast HTML parsing, no JS rendering needed | ❌ |
| `HttpCrawler` | Raw HTTP responses, custom parsing | ❌ |
| `JSDOMCrawler` | DOM manipulation without full browser | ❌ |
| `PlaywrightCrawler` | Modern headless browser (Chromium/Firefox/WebKit) | ✅ |
| `PuppeteerCrawler` | Chromium/Chrome headless automation | ✅ |
| `AdaptivePlaywrightCrawler` | Auto-detects if JS rendering is needed | Auto |
| `BasicCrawler` | Custom HTTP logic from scratch | ❌ |

**Rule of thumb**: Start with `CheerioCrawler`. Upgrade to `PlaywrightCrawler` only when JS rendering is required.

### Python

| Crawler | When to Use |
|---|---|
| `BeautifulSoupCrawler` | HTML parsing with BeautifulSoup (fast, no JS) |
| `ParselCrawler` | CSS/XPath selectors, Scrapy-style (fast, no JS) |
| `PlaywrightCrawler` | Full browser automation (Chromium/Firefox/WebKit) |
| `AdaptivePlaywrightCrawler` | Auto HTTP vs browser decision |

---

## 2. Installation

### JavaScript
```bash
# Recommended: use the CLI
npx crawlee create my-crawler
cd my-crawler && npm install

# Or manually:
npm install crawlee

# For Playwright:
npm install crawlee playwright
npx playwright install

# For Puppeteer:
npm install crawlee puppeteer
```

Add to `package.json`:
```json
{ "type": "module" }
```

### Python
```bash
pip install crawlee

# With BeautifulSoup:
pip install 'crawlee[beautifulsoup]'

# With Playwright:
pip install 'crawlee[playwright]'
playwright install
```

---

## 3. Core Concepts

### The Two Questions Every Crawler Answers
1. **Where to go?** → `Request` objects in a `RequestQueue`
2. **What to do there?** → `requestHandler` function (JS) / decorated handler (Python)

### Key Classes (JS)
- `Request` — A single URL + metadata to crawl
- `RequestQueue` — Dynamic, deduplicated queue of URLs
- `Dataset` — Append-only structured result storage (like a table)
- `KeyValueStore` — Blob storage for screenshots, PDFs, state
- `ProxyConfiguration` — Manages proxy rotation
- `SessionPool` — Manages browser sessions + cookies

---

## 4. Quick Start Examples

### JavaScript — CheerioCrawler (Recommended Start)
```javascript
import { CheerioCrawler, Dataset } from 'crawlee';

const crawler = new CheerioCrawler({
  async requestHandler({ $, request, enqueueLinks, log }) {
    const title = $('title').text();
    log.info(`Title of ${request.loadedUrl}: ${title}`);

    await Dataset.pushData({ url: request.loadedUrl, title });

    // Enqueue all links found on this page
    await enqueueLinks();
  },
  maxRequestsPerCrawl: 100, // Safety limit
});

await crawler.run(['https://example.com']);
```

### JavaScript — PlaywrightCrawler
```javascript
import { PlaywrightCrawler, Dataset } from 'crawlee';

const crawler = new PlaywrightCrawler({
  // headless: false, // Uncomment to see the browser
  async requestHandler({ page, request, enqueueLinks, log }) {
    const title = await page.title();
    log.info(`${request.loadedUrl}: ${title}`);
    await Dataset.pushData({ url: request.loadedUrl, title });
    await enqueueLinks();
  },
});

await crawler.run(['https://example.com']);
```

### Python — BeautifulSoupCrawler
```python
import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main() -> None:
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def handler(context: BeautifulSoupCrawlingContext) -> None:
        title = context.soup.title.string if context.soup.title else None
        context.log.info(f'Processing {context.request.url}: {title}')
        await context.push_data({'url': context.request.url, 'title': title})
        await context.enqueue_links()

    await crawler.run(['https://example.com'])

if __name__ == '__main__':
    asyncio.run(main())
```

### Python — PlaywrightCrawler
```python
import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

async def main() -> None:
    crawler = PlaywrightCrawler(headless=True, browser_type='chromium')

    @crawler.router.default_handler
    async def handler(context: PlaywrightCrawlingContext) -> None:
        title = await context.page.title()
        await context.push_data({'url': context.request.url, 'title': title})
        await context.enqueue_links()

    await crawler.run(['https://example.com'])

if __name__ == '__main__':
    asyncio.run(main())
```

---

## 5. Routing — Handling Multiple Page Types

Use labels + router to handle different kinds of pages (list pages, detail pages, etc.).

### JavaScript
```javascript
import { PlaywrightCrawler, Dataset } from 'crawlee';
import { router } from './routes.js';

const crawler = new PlaywrightCrawler({ requestHandler: router });

await crawler.run([{ url: 'https://shop.example.com', label: 'START' }]);
```

```javascript
// routes.js
import { createPlaywrightRouter } from 'crawlee';

export const router = createPlaywrightRouter();

router.addHandler('START', async ({ page, enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.category', label: 'CATEGORY' });
});

router.addHandler('CATEGORY', async ({ page, enqueueLinks }) => {
  await enqueueLinks({ selector: 'a.product', label: 'DETAIL' });
  // Enqueue next page
  const next = await page.$('a.next-page');
  if (next) await enqueueLinks({ selector: 'a.next-page', label: 'CATEGORY' });
});

router.addDefaultHandler(async ({ page, request, pushData }) => {
  // DETAIL pages
  const title = await page.title();
  const price = await page.$eval('.price', el => el.textContent);
  await pushData({ url: request.url, title, price });
});
```

### Python
```python
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

crawler = BeautifulSoupCrawler()

@crawler.router.handler('CATEGORY')
async def category_handler(context: BeautifulSoupCrawlingContext) -> None:
    await context.enqueue_links(selector='a.product', label='DETAIL')

@crawler.router.default_handler
async def detail_handler(context: BeautifulSoupCrawlingContext) -> None:
    title = context.soup.title.string
    await context.push_data({'url': context.request.url, 'title': title})
```

---

## 6. Enqueuing Links

### JavaScript — `enqueueLinks()`
```javascript
// Enqueue all links on page
await enqueueLinks();

// Filter by glob pattern
await enqueueLinks({ globs: ['https://example.com/products/**'] });

// Filter by regex
await enqueueLinks({ regexps: [/\/product\/\d+/] });

// Enqueue only specific selector
await enqueueLinks({ selector: 'a.pagination', label: 'LIST' });

// Enqueue with custom label and transformations
await enqueueLinks({
  selector: 'a.item',
  label: 'DETAIL',
  transformRequestFunction: (req) => {
    req.userData.scrapedAt = new Date().toISOString();
    return req;
  },
});
```

### Python
```python
await context.enqueue_links()
await context.enqueue_links(selector='a.product', label='DETAIL')
await context.enqueue_links(include=[re.compile(r'/products/\d+')])
```

---

## 7. Storage

### Dataset (structured results)
```javascript
// JS — Write
await Dataset.pushData({ url, title, price });
await Dataset.pushData([item1, item2, item3]); // batch write

// JS — Read / Export
const dataset = await Dataset.open();
await dataset.exportToCSV('results'); // saves to KV store
await dataset.exportToJSON('results');

for await (const item of dataset) { console.log(item); }
```

```python
# Python — Write
await context.push_data({'url': url, 'title': title})

# Python — Read / Export
from crawlee.storage