url-to-markdown

TotalClaw 作者 totalclaw

获取 URL 并将网页转换为干净的 Markdown，用于 AI 处理或知识库。在以下情况下使用：(1) 研究网页内容，(2) 构建知识库，(3) 提取文章，(4) 将页面转换为文档。

安装 / 下载方式

TotalClaw CLI推荐

totalclaw skills install shop:totalclaw:totalclaw~mattvalenta-pls-url-to-markdown

cURL直接下载，无需登录

curl -fsSL https://skills.taituai.com/api/skills/totalclaw%3Atotalclaw~mattvalenta-pls-url-to-markdown/file -o mattvalenta-pls-url-to-markdown.md

## 概述（中文）

获取 URL 并将网页转换为干净的 Markdown，用于 AI 处理或知识库。在以下情况下使用：(1) 研究网页内容，(2) 构建知识库，(3) 提取文章，(4) 将页面转换为文档。

## 原文

# URL to Markdown Converter

Fetches URLs and converts web pages to clean Markdown.

## Quick Start

### Python Method (`markdownify`)
```bash
pip install requests beautifulsoup4 markdownify

python3 -c "... fetching and converting URL ..."
```

### CLI Tools (`html2text`, `pandoc`)
```bash
curl -s URL | html2text
wget -q -O - URL | pandoc -f html -t markdown
```

## Full Extraction Script
```python
import requests
from bs4 import BeautifulSoup
from markdownify import markdownify as md

def url_to_markdown(url, output_file=None):
    # ... fetch, parse, convert logic ...
    pass
```

## Content Extraction Patterns

### Extract Article Body
```python
def extract_article(html):
    soup = BeautifulSoup(html, 'html.parser')
    article = soup.find('article') or soup.find('main')
    return md(str(article)) if article else None
```

### Preserve Code Blocks
```python
def preserve_code(html):
    # ... logic to wrap code in ``` ...
    pass
```

## CLI Usage

```bash
python url_to_markdown.py URL -o output.md
```

## Error Handling

```python
def safe_fetch(url, retries=3):
    # ... retry logic ...
    pass
```