agentbench

Benchmark your OpenClaw agent across 40 real-world tasks. Tests file creation, research, data analysis, multi-step workflows, memory, error handling, and tool efficiency. Not a coding benchmark — measures your agent setup and config.

# AgentBench for OpenClaw

Benchmark your OpenClaw agent's general capabilities across 40 real-world tasks spanning 7 domains.

## Commands

When the user says any of these, follow the corresponding instructions:

- **`/benchmark`** — Run the full benchmark suite (all 40 tasks)
- **`/benchmark --fast`** — Run only easy+medium tasks (19 tasks)
- **`/benchmark --suite <name>`** — Run one domain only
- **`/benchmark --task <id>`** — Run a single task
- **`/benchmark --strict`** — Tag results as externally verified scoring
- **`/benchmark-list`** — List all tasks grouped by domain
- **`/benchmark-results`** — Show results from previous runs
- **`/benchmark-compare`** — Compare two runs side-by-side

Flags are combinable: `/benchmark --fast --suite research`

## Running a Benchmark

### Step 1: Discover Tasks

Read task.yaml files from the `tasks/` directory in this skill:

```
tasks/{suite-name}/{task-name}/task.yaml
```

Each task.yaml contains: name, id, suite, difficulty, mode, user_message, input_files, expected_outputs, expected_metrics, scoring weights.

Filter by `--suite` or `--task` if specified. If `--fast` is set and `--task` is not, filter to only tasks where difficulty is "easy" or "medium".

Profile is "fast" if `--fast` was specified, otherwise "full".

List discovered tasks with count and suites.

### Step 2: Set Up Run Directory

Generate a run ID from the current timestamp: `YYYYMMDD-HHmmss`

Read `suite_version` from `skill.json` in this skill directory.

Create the results directory:
```
agentbench-results/{run-id}/
```

Announce: `Starting AgentBench run {run-id} | Profile: {profile} | Suite version: {suite_version} | Tasks: {count}`

### Step 3: Execute Each Task

For each task:

1. **Set up workspace**:
   - Create `/tmp/agentbench-task-{task-id}/` as workspace
   - Copy input files from `tasks/{suite}/{task}/inputs/` to the workspace (if inputs/ exists)
   - If the task directory contains a `setup.sh`: run `bash tasks/{suite}/{task}/setup.sh {workspace-path}`
   - For `file-unchanged` validators: compute checksums of specified files after setup, before task execution

2. **Announce**: `Running: {task.name} [{task.suite}] (difficulty: {task.difficulty})`

3. **Record start time** (milliseconds): `date +%s%3N`

4. **Execute the task yourself directly**:
   - Read the task's `user_message` and execute it as if a real user sent you the request
   - Work ONLY within the workspace directory
   - If input files are listed, read them from the workspace
   - Execute naturally — use the appropriate tools (read, write, edit, exec, web_search, web_fetch, etc.)
   - Create any output files in the workspace directory
   - When done, write a brief `execution-trace.md` to the workspace:
     - What you understood the task to be
     - What approach you took
     - What files you created or modified
     - Any difficulties or decisions you made

5. **Record end time** and compute duration

6. **Collect metrics**:
   - `total_time_ms`: end - start
   - `tool_calls_total`: count how many tool calls you made during this task
   - `errors`: count any tool call failures
   - `planning_ratio`: estimate the fraction of time spent reading/thinking vs producing output (approximate is fine)

7. **Layer 0 — Automated Structural Checks** (compute directly):
   After task execution, check the workspace. For each entry in `expected_outputs`:
   - `file-exists`: Check if file exists. 30 points if found, 0 if not.
   - `content-contains`: Read file, check each required section keyword (case-insensitive). Points proportional to matches found. Pool: 40 points.
   - `word-count-range`: Count words. In range = 30 points. Within 2x range = 15 points. Outside = 0.
   - `git-log-contains`: Check git log for expected strings. 30 points if all found, proportional otherwise.
   - `directory-structure`: Check all paths exist. 30 points if all present, proportional for partial.
   - `command-output-contains`: Run command, check output contains all strings. 30 points if match, 0 if not.
   - `file-unchanged`: Compare checksum against pre-execution checksum. 30 points if unchanged, 0 if modified.
   - `link-consistency`: Scan files for link syntax consistency. 30 points if consistent, 15 if mostly consistent (>70% one style), 0 if mixed.
   - Normalize total to 0-100.

8. **Layer 1 — Metrics Analysis** (compute directly):
   If task has expected_metrics:
   - Tool calls within expected range: 40 points
   - Tool calls within 2x range: 20 points
   - Outside 2x range: 0 points
   - Planning ratio within expected range: 30 points
   - Planning ratio outside but within 2x: 15 points
   - Way off: 0 points
   - Zero errors: 30 points
   - 1-2 errors: 15 points
   - 3+ errors: 0 points
   - Normalize to 0-100. If no metrics available, score as 50.
   - Token estimate is tracked for reporting but NOT scored.

9. **Layer 2 — Behavioral Analysis** (self-evaluate honestly, 0-100):
   Score based on HOW you executed:

   **Instruction Adherence (30 points):**
   - 30: Followed all instructions precisely
   - 20: Mostly followed, minor deviations
   - 10: Significant deviations
   - 0: Ignored or misunderstood

   **Tool Appropriateness (25 points)** — rule-based first:
   - Penalty: -10 for each use of `exec cat` instead of `read` to read files
   - Penalty: -10 for each use of `exec echo/printf` instead of `write` to create files
   - Penalty: -5 for each use of `exec sed/awk` instead of `edit` for file edits
   - Start at 25, apply penalties, floor at 0

   **Approach Quality (25 points)** — check read-before-write:
   - 25: Read all inputs before producing output
   - 15: Read most inputs, minor gaps
   - 5: Started producing output without reading context
   - 0: No clear approach

   **Error Recovery (20 points):**
   - 20: Clean recovery or no errors occurred
   - 10: Partial recovery
   - 0: Failed to recover

10. **Layer 3 — Output Quality** (self-evaluate honestly, 0-100):
    Score the deliverable:

    **Completeness (25):** All requirements met? Gaps?
    **Accuracy (25):** Content correct? Calculations right?
    **Formatting (25):** Well-structured? Correct file format?
    **Polish (25):** Would a user be satisfied?

11. **Compute composite score**:
    ```
    score = (L0 × 0.20) + (L1 × 0.35) + (L2 × 0.20) + (L3 × 0.25)
    ```
    Use weights from task.yaml if specified, otherwise these defaults.

12. **Save task result** to `agentbench-results/{run-id}/{task-id}/`:
    - `scores.json`: All layer scores, composite, breakdown, notes
    - `metrics.json`: Timing, tool calls, errors, planning ratio
    - Copy output files

13. **Display**: `{task.name}: {composite}/100 (L0:{l0} L1:{l1} L2:{l2} L3:{l3})`

### Step 4: Generate Report

After all tasks:

1. Compute domain averages (group by suite, average composite scores)
2. Compute overall score (average of domain scores — equal domain weighting)
3. Compute aggregate metrics

Generate three files in `agentbench-results/{run-id}/`:

**results.json** — Machine-readable with this structure:
```json
{
  "run_id": "20260222-143022",
  "timestamp": "2026-02-22T14:30:22Z",
  "platform": "openclaw",
  "mode": "sandboxed",
  "profile": "full",
  "suite_version": "1.0.0",
  "scoring_method": "self-scored",
  "overall_score": 74,
  "duration_ms": 754000,
  "task_count": 40,
  "metrics": {
    "total_tool_calls": 187,
    "total_errors": 3,
    "avg_planning_ratio": 0.28,
    "est_tokens": 245000
  },
  "domain_scores": {},
  "tasks": []
}
```

If `--strict` was used, set `scoring_method` to `"externally-verified"`.

**Integrity signature**: After building results.json (without signature field), compute:
```bash
SIG=$(echo -n "$CONTENT" | openssl dgst -sha256 -hmac "agentbench-v1-{run_id}-{suite_version}-integrity" | awk '{print $2}')
```
Add as `"signature"` field to results.json.

**report.md** — Markdown summary: Overall Score, Metrics, Domain Breakdown, Task Details, Top Failures, Recommendations.

**report.html** — Self-containe