eisonme-node-transfer

# node-transfer

High-speed, memory-efficient file transfer between OpenClaw nodes using native Node.js streams.

## 📋 Table of Contents

- [Problem Solved](#problem-solved)
- [Architecture](#architecture)
- [Requirements](#requirements)
- [Installation](#installation)
- [Usage](#usage)
- [API Reference](#api-reference)
- [Troubleshooting](#troubleshooting)

---

## 🎯 Problem Solved

### The Original Problem

When transferring large files between OpenClaw nodes using the standard `nodes.invoke` mechanism, we encountered several critical issues:

| Issue | Impact |
|-------|--------|
| **Base64 Encoding Overhead** | 33% larger payload, slower transfers |
| **Memory Exhaustion (OOM)** | Loading multi-GB files into memory crashes the process |
| **Transfer Latency** | JSON serialization/deserialization adds significant delay |
| **9-Minute Deployments** | Re-deploying scripts on every transfer |

### The Solution

`node-transfer` uses **native HTTP streaming** with Node.js streams, providing:

- ✅ **Zero memory overhead** - Files stream directly from disk to network
- ✅ **No Base64 encoding** - Raw binary transfer
- ✅ **Speed** - Line-speed limited only by network bandwidth
- ✅ **Install Once, Run Many** - Scripts persist on nodes after first deployment

### Performance Comparison

| Metric | Base64 Transfer | node-transfer | Improvement |
|--------|----------------|---------------|-------------|
| 1GB file transfer time | ~15-30 min | ~8 sec | **~150x faster** |
| Memory usage | 1GB+ | <10MB | **99% reduction** |
| First transfer overhead | N/A | ~30 sec (one-time install) | - |
| Subsequent transfers | ~15-30 min | **<1 sec** check + ~8 sec transfer | **~200x faster** |

---

## 🏗️ Architecture

### How It Works

```
┌──────────────┐     HTTP Stream      ┌──────────────┐
│  send.js     │ ◄──────────────────► │ receive.js   │
│  (Source)    │   (Token-protected)  │ (Destination)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  Read Stream │                      │ Write Stream │
│  (fs.create  │                      │ (fs.create   │
│   ReadStream)│                      │  WriteStream)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  File on     │                      │  File on     │
│  Disk        │                      │  Disk        │
└──────────────┘                      └──────────────┘
```

### Security Model

1. **One-time Token**: 256-bit cryptographically random token (64 hex chars)
2. **Single Connection**: Only one download allowed per token
3. **Auto-shutdown**: Server closes after transfer completes or disconnects
4. **Token Validation**: Every request must include the correct token

### Data Flow

1. **Sender** (`send.js`):
   - Generates random port and security token
   - Starts HTTP server on ephemeral port
   - Streams file directly from disk to HTTP response
   - Auto-shutdown after transfer or timeout (5 min default)

2. **Receiver** (`receive.js`):
   - Connects to sender URL with token
   - Streams HTTP response directly to disk
   - Reports progress, speed, and completion status
   - Validates received bytes match expected size

---

## 📦 Requirements

- **Node.js**: 14.0.0 or higher
- **Network**: TCP connectivity between nodes (any port 1024-65535)
- **Firewall**: Must allow outbound connections and inbound on ephemeral ports
- **Disk Space**: Sufficient space on destination for received files

---

## 🚀 Installation

### The "Install Once" Pattern

Instead of deploying scripts on every transfer, we deploy them **once per node** and use a fast version check for subsequent transfers.

### Method 1: Using deploy.js (Recommended)

```bash
# Generate deployment script for a target node
node deploy.js E3V3

# This outputs a PowerShell script that you can execute via nodes.invoke()
```

### Method 2: Manual Deployment

On each target node, create the directory and copy files:

```powershell
# Create directory
mkdir C:/openclaw/skills/node-transfer/scripts -Force

# Copy these files (ensure UTF-8 without BOM encoding):
# - send.js
# - receive.js
# - ensure-installed.js
# - version.js
```

### Method 3: Via OpenClaw Agent

```javascript
// 1. Check if already installed (< 100ms)
const check = await nodes.invoke({
    node: 'E3V3',
    command: ['node', 'C:/openclaw/skills/node-transfer/scripts/ensure-installed.js', 
              'C:/openclaw/skills/node-transfer/scripts']
});

const checkResult = JSON.parse(check.output);

if (!checkResult.installed) {
    // 2. Deploy if needed (one-time, ~30 seconds)
    // Use the deploy.js output or manually copy files
    console.log('Deploying node-transfer to E3V3...');
    // ... deployment code ...
}
```

---

## 💡 Usage

### Basic Transfer Workflow

```javascript
const INSTALL_DIR = 'C:/openclaw/skills/node-transfer/scripts';
const SOURCE_NODE = 'E3V3';
const DEST_NODE = 'E3V3-Docker';

// Step 1: Check installation on both nodes (fast!)
const [sourceCheck, destCheck] = await Promise.all([
    nodes.invoke({
        node: SOURCE_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    }),
    nodes.invoke({
        node: DEST_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    })
]);

// Deploy if needed (usually only once per node ever)
// ... deployment code if not installed ...

// Step 2: Start sender on source node
const sendResult = await nodes.invoke({
    node: SOURCE_NODE,
    command: ['node', `${INSTALL_DIR}/send.js`, 'C:/data/large-file.zip']
});

const { url, token, fileSize, fileName } = JSON.parse(sendResult.output);

// Step 3: Start receiver on destination node
const receiveResult = await nodes.invoke({
    node: DEST_NODE,
    command: ['node', `${INSTALL_DIR}/receive.js`, url, token, '/incoming/file.zip']
});

const result = JSON.parse(receiveResult.output);
console.log(`Transferred ${result.bytesReceived} bytes in ${result.duration}s at ${result.speedMBps} MB/s`);
```

### Using the Command Line

#### Sender

```bash
node send.js /path/to/file.zip
```

Output:
```json
{
  "url": "http://192.168.1.10:54321/transfer",
  "token": "a1b2c3d4e5f6789...",
  "fileSize": 1073741824,
  "fileName": "file.zip",
  "sourceIp": "192.168.1.10",
  "port": 54321,
  "version": "1.0.0"
}
```

Options:
```bash
node send.js /path/to/file.zip --port 8080 --timeout 10
node send.js --help
node send.js --version
```

#### Receiver

```bash
node receive.js "http://192.168.1.10:54321/transfer" "token-here..." /path/to/save.zip
```

Output:
```json
{
  "success": true,
  "bytesReceived": 1073741824,
  "totalBytes": 1073741824,
  "duration": 8.42,
  "speedMBps": 121.5,
  "outputPath": "/path/to/save.zip"
}
```

Options:
```bash
node receive.js <url> <token> <output> --timeout 60 --no-progress
node receive.js --help
node receive.js --version
```

---

## 📚 API Reference

### send.js

Starts an HTTP server to stream a file.

**Usage:** `node send.js <filePath> [options]`

**Arguments:**
- `filePath` (required): Path to the file to send

**Options:**
- `--port <n>`: Use specific port (default: random ephemeral)
- `--timeout <n>`: Timeout in minutes (default: 5)

**Output (JSON):**
| Field | Type | Description |
|-------|------|-------------|
| `url` | string | HTTP URL for receiver to connect to |
| `token` | string | Security token (64 hex chars) |
| `fileSize` | number | File size in bytes |
| `fileName` | string | Original filename |
| `sourceIp` | string | IP address of sender |
| `port` | number | TCP port used |
| `version` | string | Version of send.js |

**Exit Codes:**
- `0`: Success (transfer completed or info displayed)
- `1`: Error (check stderr for JSON error details)

**Error Output (JSON):**
```json
{
  "