openclaw-pyautogui

Cross-platform mouse/keyboard automation skill. Supports mouse control (move/click/drag/scroll), keyboard control (key press/hotkeys/type text), screen operations (screenshots/mouse position/screen size), image utilities (metadata/crop), screen overlay markers, drawing markers on images, image locating (template matching + OCR), and file cleanup to free disk space. Activate when the user needs UI automation, screenshots, coordinate verification, image analysis/annotation, on-screen element locating, or cleanup.

# PyAutoGUI Automation Skill

Cross-platform mouse/keyboard automation for Windows, Linux, and macOS.

## Features

- **Mouse control**: move, click, drag, scroll
- **Keyboard control**: key press, hotkeys, type text
- **Screen operations**: screenshot, mouse position, screen size
- **Image utilities**: image metadata (size/format/file size), crop images
- **Screen overlay**: draw temporary markers to validate coordinates
- **Draw on images**: draw persistent markers into an image and save
- **Image locating**: template matching and OCR-based text locating
- **Cleanup**: remove generated screenshots/marked files to free disk space

## Activation

Activate when the user asks to do things like:
- "Click a position on the screen"
- "Move the mouse to (x, y)"
- "Type text / press keys"
- "Take a screenshot"
- "Run repetitive UI automation"
- "Get the current mouse position"
- "Get image size / image info"
- "Crop an image"
- "Draw a marker on the screen"
- "Draw a marker on an image"
- "Locate an element by template"
- "Locate text on the screen (OCR)"
- "Clean up screenshots / temporary files"

## Usage

### Install dependencies

```bash
# Mouse/keyboard automation
pip3 install pyautogui

# Image utilities
pip3 install Pillow
```

### Screen info

```bash
# Screen size
python3 scripts/keyboard_mouse.py screen_size

# Mouse position
python3 scripts/keyboard_mouse.py mouse_position
```

### Mouse actions

```bash
# Move mouse to (x, y)
python3 scripts/keyboard_mouse.py mouse_move 500 300
python3 scripts/keyboard_mouse.py mouse_move 500 300 --duration 1.0

# Mouse click (left/right/middle)
python3 scripts/keyboard_mouse.py mouse_click left
python3 scripts/keyboard_mouse.py mouse_click right
python3 scripts/keyboard_mouse.py mouse_click middle --clicks 2

# Click at a specific location
python3 scripts/keyboard_mouse.py mouse_click_at 500 300 left
python3 scripts/keyboard_mouse.py mouse_click_at 500 300 right --clicks 2

# Double click
python3 scripts/keyboard_mouse.py mouse_double_click 500 300

# Drag
python3 scripts/keyboard_mouse.py mouse_drag 500 300 800 600
python3 scripts/keyboard_mouse.py mouse_drag 500 300 800 600 --duration 2.0

# Scroll (positive = up, negative = down)
python3 scripts/keyboard_mouse.py mouse_scroll 5
python3 scripts/keyboard_mouse.py mouse_scroll -3
```

### Keyboard actions

```bash
# Single key
python3 scripts/keyboard_mouse.py key_press enter
python3 scripts/keyboard_mouse.py key_press escape
python3 scripts/keyboard_mouse.py key_press tab
python3 scripts/keyboard_mouse.py key_press space

# Hotkeys
python3 scripts/keyboard_mouse.py key_hotkey ctrl c
python3 scripts/keyboard_mouse.py key_hotkey ctrl v
python3 scripts/keyboard_mouse.py key_hotkey win r
python3 scripts/keyboard_mouse.py key_hotkey alt tab
python3 scripts/keyboard_mouse.py key_hotkey ctrl alt t

# Type text
python3 scripts/keyboard_mouse.py type_text "Hello World"
python3 scripts/keyboard_mouse.py type_text "你好世界" --interval 0.05
```

### Screenshot

```bash
# Save a screenshot (primary screen)
python3 scripts/keyboard_mouse.py screenshot /tmp/screenshot.png

# Windows example
python scripts/keyboard_mouse.py screenshot "E:\\temp\\screenshot.png"
```

**Screenshot notes:**
- Supported formats: PNG (recommended), JPG, BMP, etc.
- Scope: primary monitor (in multi-monitor setups)

### Region Screenshot

```bash
# Screenshot specific region (x1, y1, x2, y2)
python3 scripts/keyboard_mouse.py screenshot_region region.png 100 100 500 500

# Windows example - capture QQ chat window area
python scripts/keyboard_mouse.py screenshot_region qq_window.png 2800 300 3800 1200
```

**Parameters:**
- `x1, y1`: Top-left corner coordinates
- `x2, y2`: Bottom-right corner coordinates
- Order doesn't matter (automatically calculated)

### Copy & Paste

```bash
# Copy text to clipboard
python3 scripts/keyboard_mouse.py copy "Text to copy"

# Paste from clipboard (Ctrl+V)
python3 scripts/keyboard_mouse.py paste

# Copy and paste in one command (fastest way to input text)
python3 scripts/keyboard_mouse.py copy_paste "Text to input directly"
```

**Use cases:**
- `copy_paste` is faster than `type_text` for long text
- Use `copy_paste` when you want to skip typing animation
- Use `type_text` when you need to simulate realistic typing

## Common key names

- Letters: `a` `b` `c` ...
- Numbers: `0` `1` `2` ...
- Function keys: `f1` `f2` ... `f12`
- Modifiers: `ctrl` `alt` `shift` `win`
- Others: `enter` `esc` `tab` `space` `backspace` `delete` `up` `down` `left` `right`

## Safety

⚠️ **Important:**
1. Make sure the target window is focused before executing actions
2. Be careful with system hotkeys to avoid unintended actions
3. Add delays when needed to give yourself time to interrupt
4. Moving the mouse to the top-left corner (0, 0) triggers PyAutoGUI failsafe

## Cross-platform notes

- **Windows**: Full support; admin permission may be needed in some environments
- **Linux**: Requires X11; Wayland may not work
- **macOS**: Grant Accessibility permission to Terminal/Python in System Settings

## Example scenarios

### Open Calculator (Windows)
```bash
python3 scripts/keyboard_mouse.py key_hotkey win r
python3 scripts/keyboard_mouse.py type_text "calc"
python3 scripts/keyboard_mouse.py key_press enter
```

### Auto-fill a form
```bash
python3 scripts/keyboard_mouse.py mouse_click_at 500 300 left
python3 scripts/keyboard_mouse.py type_text "example@email.com"
python3 scripts/keyboard_mouse.py key_press tab
python3 scripts/keyboard_mouse.py type_text "password123"
```

### Batch clicking
```bash
python3 scripts/keyboard_mouse.py mouse_click_at 100 100 left
python3 scripts/keyboard_mouse.py mouse_click_at 200 200 left
python3 scripts/keyboard_mouse.py mouse_click_at 300 300 left
```

## Included scripts

- `scripts/keyboard_mouse.py` - Mouse/keyboard control
- `scripts/image_utils.py` - Image utilities
- `scripts/draw_overlay.py` - Screen overlay markers
- `scripts/draw_on_image.py` - Draw markers on images
- `scripts/image_finder.py` - Image locating (template + OCR)
- `scripts/cleanup.py` - Cleanup tool

## Image utilities

### Image info

```bash
python3 scripts/image_utils.py info screenshot.png
python3 scripts/image_utils.py size photo.jpg
```

### Crop image

```bash
python3 scripts/image_utils.py crop screenshot.png 100 100 500 500
python3 scripts/image_utils.py crop screenshot.png 100 100 500 500 -o output.png
```

### Output example

```bash
$ python3 scripts/image_utils.py info screenshot.png
{
  "path": "screenshot.png",
  "filename": "screenshot.png",
  "size": {
    "width": 3840,
    "height": 2160
  },
  "format": "PNG",
  "mode": "RGB",
  "file_size_bytes": 2097152,
  "file_size_kb": 2048.0
}
```

### Image fields

| Field | Meaning | Example |
|------|------|--------|
| `width` | Image width (px) | 1920, 3840 |
| `height` | Image height (px) | 1080, 2160 |
| `format` | Image format | PNG, JPEG, GIF, BMP, WEBP |
| `mode` | Color mode | RGB, RGBA, L |
| `file_size_bytes` | File size (bytes) | 2097152 |
| `file_size_kb` | File size (KB) | 2048.0 |

### Coordinate system

**Screen coordinates:**
- Origin (0, 0) is the top-left corner
- X increases to the right
- Y increases downward

**Crop coordinates:**
- `x1, y1`: top-left corner of crop
- `x2, y2`: bottom-right corner of crop
- Cropped size = (x2 - x1) × (y2 - y1)

**Example:**
```bash
python3 scripts/image_utils.py crop screenshot.png 1520 880 1920 1080
```

### Typical workflows

#### Analyze positions in a screenshot
```bash
python3 scripts/image_utils.py size screenshot.png
python3 scripts/image_utils.py crop screenshot.png 3440 1960 3840 2160 -o bottom_right.png
```

#### Batch image sizing
```bash
for img in *.png; do
    echo -n "$img: "
    python3 scripts/image_utils.py size "$img"
done
```

#### Capture a region of the screen
```bash
python3 scripts/keyboard_mouse.py screenshot full.png
python3 scripts/image_utils.py crop full.png 500 300 1000 800 -o region.png
```

---

## Screen overlay markers

Draw temporary m