scrapling

使用 Scrapling 进行网页抓取 — 一个具有反机器人绕过功能（Cloudflare Turnstile、指纹欺骗）、自适应元素跟踪、隐形无头浏览器和完整 CSS/XPath 提取的 Python 框架。当 web_fetch 失败时（Cloudflare、JS 渲染的页面）或从网站（价格、文章、列表）提取结构化数据时使用。支持 HTTP、隐形和完整浏览器模式。来源：github.com/D4Vinci/Scrapling（PyPI：scrapling）。仅在您有权抓取的网站上使用。

## 概述（中文）

使用 Scrapling 进行网页抓取 — 一个具有反机器人绕过功能（Cloudflare Turnstile、指纹欺骗）、自适应元素跟踪、隐形无头浏览器和完整 CSS/XPath 提取的 Python 框架。当 web_fetch 失败时（Cloudflare、JS 渲染的页面）或从网站（价格、文章、列表）提取结构化数据时使用。支持 HTTP、隐形和完整浏览器模式。来源：github.com/D4Vinci/Scrapling（PyPI：scrapling）。仅在您有权抓取的网站上使用。

## 原文

# Scrapling Skill

**Source:** https://github.com/D4Vinci/Scrapling (open source, MIT-like license)
**PyPI:** `scrapling` — install before first use (see below)

> ⚠️ Only scrape sites you have permission to access. Respect `robots.txt` and Terms of Service. Do not use stealth modes to bypass paywalls or access restricted content without authorization.

## Installation (one-time, confirm with user before running)

```bash
pip install scrapling[all]
patchright install chromium  # required for stealth/dynamic modes
```

- `scrapling[all]` installs `patchright` (a stealth fork of Playwright, bundled as a PyPI package — not a typo), `curl_cffi`, MCP server deps, and IPython shell.
- `patchright install chromium` downloads Chromium (~100 MB) via patchright's own installer (same mechanism as `playwright install chromium`).
- Confirm with user before running — installs ~200 MB of dependencies and browser binaries.

## Script

`scripts/scrape.py` — CLI wrapper for all three fetcher modes.

```bash
# Basic fetch (text output)
python3 ~/skills/scrapling/scripts/scrape.py <url> -q

# CSS selector extraction
python3 ~/skills/scrapling/scripts/scrape.py <url> --selector ".class" -q

# Stealth mode (Cloudflare bypass) — only on sites you're authorized to access
python3 ~/skills/scrapling/scripts/scrape.py <url> --mode stealth -q

# JSON output
python3 ~/skills/scrapling/scripts/scrape.py <url> --selector "h2" --json -q
```

## Fetcher Modes

- **http** (default) — Fast HTTP with browser TLS fingerprint spoofing. Most sites.
- **stealth** — Headless Chrome with anti-detect. For Cloudflare/anti-bot.
- **dynamic** — Full Playwright browser. For heavy JS SPAs.

## When to Use Each Mode

- `web_fetch` returns 403/429/Cloudflare challenge → use `--mode stealth`
- Page content requires JS execution → use `--mode dynamic`
- Regular site, just need text/data → use `--mode http` (default)

## Python Inline Usage

For custom logic beyond the CLI, write inline Python. See `references/patterns.md` for:
- Adaptive scraping (`auto_save` / `adaptive` — saves element fingerprints locally)
- Session/cookie handling
- Async usage
- XPath, find_similar, attribute extraction

## Notes

- **MCP server** (`scrapling mcp`): starts a local network service for AI-native scraping. Only start if explicitly needed and trusted — it exposes a local HTTP server.
- **`auto_save=True`**: persists element fingerprints to disk for adaptive re-scraping. Creates local state in working directory.
- Stealth/dynamic modes use Chromium headless — no `xvfb-run` needed.
- For large-scale crawls, use the Spider API (see Scrapling docs).