handsfree-windows-control

指导通过免提 Windows CLI 控制本机 Windows 应用程序 (UIA) 和 Web 浏览器 (Playwright) 的技能。当您需要在 Windows 计算机上自动化或测试桌面应用程序或网站时使用：从“开始”菜单启动应用程序、无需猜测即可发现 UI 控件、在本机应用程序中单击/键入、在浏览器中打开/快照/单击、记录和重播混合桌面和 Web 步骤的 YAML 宏。需要免提 Windows CLI（首次使用时通过 setup.py 自动安装）。

# Handsfree Windows Control

A guide skill for automating native Windows apps (UIA) and web browsers (Playwright)
via the `handsfree-windows` CLI.

## First use: auto-setup

On first use, run setup before issuing any other commands:

```powershell
python scripts/setup.py
```

This will:
1. Clone `handsfree-windows` from GitHub into `~/.handsfree-windows/cli/` (public repo, read-only)
2. Install it via `pip install -e` (standard pip editable install)
3. Install Playwright + Chromium browser binaries (~200 MB one-time download from cdn.playwright.dev)
4. Run `check_setup.py` to verify everything is working

To skip browser installation (browser-* commands will not work):
```powershell
python scripts/setup.py --no-browser
```

To install to a custom directory:
```powershell
python scripts/setup.py --install-dir "C:\your\preferred\path"
```

Already installed? Verify anytime:
```powershell
python scripts/check_setup.py
```

### What is written to disk (transparent)
- CLI source code: `~/.handsfree-windows/cli/` (or --install-dir)
- pip editable link: standard site-packages egg-link (pip managed)
- Browser persistent profiles: `~/.handsfree-windows/browser-profiles/<engine>/`
  Contains cookies and login sessions. Delete to reset browser auth.
- Browser session state: `~/.handsfree-windows/browser-state.json` (last visited URL)
- Playwright browser binaries: `~\AppData\Local\ms-playwright\` (~800 MB, Windows)

To fully remove everything:
```powershell
pip uninstall handsfree-windows -y
Remove-Item -Recurse -Force "$env:USERPROFILE\.handsfree-windows"
```

---

## Core rules
- Do not guess UI controls. Run `hf tree` or `hf inspect` first, then act on what is actually there.
- Do not type credentials. Navigate to login screens; let the human complete auth.
- Prefer UIA selectors (name + control_type) over raw coordinates.
- Use `drag-canvas` only for canvas/ink surfaces (Paint, drawing apps, etc.).
- For destructive actions (delete, submit, send), ask the human for confirmation first.

---

## Workflow: Desktop app (UIA)

```powershell
# Launch any installed app
hf start --app "Outlook"

# Find the window
hf list-windows --json

# Discover controls (no guessing)
hf tree --title-regex "Outlook" --depth 10 --max-nodes 30000

# Act on what was found
hf click --title "Outlook" --name "New mail" --control-type "Button"

# Inspect element under cursor
hf inspect --json
```

## Workflow: Browser (Playwright)

```powershell
# Open URL - login sessions saved in profile automatically
hf browser-open --url "https://example.com"

# Inspect page before acting
hf browser-snapshot --fmt text

# Act
hf browser-click --text "Sign in"
hf browser-type --selector "#email" --text "user@example.com"

# Verify
hf browser-screenshot --out result.png
```

## Mixed macro (desktop + web in one YAML)

```yaml
- action: start
  args:
    app: "My Desktop App"

- action: browser-open
  args:
    url: "https://app.example.com"
    headless: false

- action: browser-click
  args:
    text: "Get Started"

- action: sleep
  args:
    seconds: 1
```

Run with: `hf run macro.yaml`

---

## References
- Full command reference + selector schema: references/api_reference.md

---

## 中文说明

# Handsfree Windows Control

一个指导性技能，通过 `handsfree-windows` CLI 自动化本机 Windows 应用程序（UIA）和 Web 浏览器（Playwright）。

## 首次使用：自动设置

首次使用时，在发出任何其他命令前先运行设置：

```powershell
python scripts/setup.py
```

这将：
1. 从 GitHub 将 `handsfree-windows` 克隆到 `~/.handsfree-windows/cli/`（公开仓库，只读）
2. 通过 `pip install -e` 安装它（标准的 pip 可编辑安装）
3. 安装 Playwright + Chromium 浏览器二进制文件（约 200 MB，从 cdn.playwright.dev 一次性下载）
4. 运行 `check_setup.py` 验证一切正常

要跳过浏览器安装（browser-* 命令将无法工作）：
```powershell
python scripts/setup.py --no-browser
```

要安装到自定义目录：
```powershell
python scripts/setup.py --install-dir "C:\your\preferred\path"
```

已经安装了？随时验证：
```powershell
python scripts/check_setup.py
```

### 写入磁盘的内容（透明）
- CLI 源代码：`~/.handsfree-windows/cli/`（或 --install-dir）
- pip 可编辑链接：标准的 site-packages egg-link（由 pip 管理）
- 浏览器持久化配置文件：`~/.handsfree-windows/browser-profiles/<engine>/`
  包含 cookie 和登录会话。删除以重置浏览器认证。
- 浏览器会话状态：`~/.handsfree-windows/browser-state.json`（最后访问的 URL）
- Playwright 浏览器二进制文件：`~\AppData\Local\ms-playwright\`（约 800 MB，Windows）

要完全移除所有内容：
```powershell
pip uninstall handsfree-windows -y
Remove-Item -Recurse -Force "$env:USERPROFILE\.handsfree-windows"
```

---

## 核心规则
- 不要猜测 UI 控件。先运行 `hf tree` 或 `hf inspect`，然后根据实际存在的内容操作。
- 不要键入凭据。导航至登录界面；让人类完成认证。
- 优先使用 UIA 选择器（name + control_type）而非原始坐标。
- 仅对画布/墨迹表面（Paint、绘图应用等）使用 `drag-canvas`。
- 对于破坏性操作（删除、提交、发送），先向人类请求确认。

---

## 工作流：桌面应用（UIA）

```powershell
# Launch any installed app
hf start --app "Outlook"

# Find the window
hf list-windows --json

# Discover controls (no guessing)
hf tree --title-regex "Outlook" --depth 10 --max-nodes 30000

# Act on what was found
hf click --title "Outlook" --name "New mail" --control-type "Button"

# Inspect element under cursor
hf inspect --json
```

## 工作流：浏览器（Playwright）

```powershell
# Open URL - login sessions saved in profile automatically
hf browser-open --url "https://example.com"

# Inspect page before acting
hf browser-snapshot --fmt text

# Act
hf browser-click --text "Sign in"
hf browser-type --selector "#email" --text "user@example.com"

# Verify
hf browser-screenshot --out result.png
```

## 混合宏（桌面 + Web 于一个 YAML 中）

```yaml
- action: start
  args:
    app: "My Desktop App"

- action: browser-open
  args:
    url: "https://app.example.com"
    headless: false

- action: browser-click
  args:
    text: "Get Started"

- action: sleep
  args:
    seconds: 1
```

运行方式：`hf run macro.yaml`

---

## 参考
- 完整命令参考 + 选择器 schema：references/api_reference.md