encoding-formats

编码、解码以及数据格式之间的转换。在使用 Base64、URL 编码、十六进制、Unicode、JWT 令牌、哈希、校验和或在序列化格式（如 JSON、MessagePack 和 protobuf 有线格式）之间进行转换时使用。

# Encoding & Formats

Encode, decode, and inspect data in common formats. Covers Base64, URL encoding, hex, Unicode, JWTs, hashing, checksums, and serialization formats.

## When to Use

- Decoding a Base64 string from an API response or config
- URL-encoding parameters for HTTP requests
- Inspecting hex dumps of binary data
- Decoding JWT tokens to see claims
- Computing or verifying file checksums
- Converting between character encodings (UTF-8, Latin-1, etc.)
- Understanding wire formats (protobuf, MessagePack)

## Base64

### Encode and decode

```bash
# Encode string
echo -n "Hello, World!" | base64
# SGVsbG8sIFdvcmxkIQ==

# Decode string
echo "SGVsbG8sIFdvcmxkIQ==" | base64 -d
# Hello, World!

# Encode a file
base64 image.png > image.b64
cat file.bin | base64

# Decode a file
base64 -d image.b64 > image.png

# Base64url (URL-safe variant: + → -, / → _, no padding)
echo -n "Hello" | base64 | tr '+/' '-_' | tr -d '='
# Base64url decode
echo "SGVsbG8" | tr '-_' '+/' | base64 -d
```

### In code

```javascript
// JavaScript (browser + Node.js 16+)
btoa('Hello');                    // "SGVsbG8="
atob('SGVsbG8=');                 // "Hello"

// Node.js Buffer
Buffer.from('Hello').toString('base64');           // "SGVsbG8="
Buffer.from('SGVsbG8=', 'base64').toString();      // "Hello"

// Binary data
Buffer.from(binaryData).toString('base64');
Buffer.from(b64String, 'base64');
```

```python
# Python
import base64

base64.b64encode(b"Hello").decode()     # "SGVsbG8="
base64.b64decode("SGVsbG8=")            # b"Hello"

# URL-safe Base64
base64.urlsafe_b64encode(b"Hello").decode()
base64.urlsafe_b64decode("SGVsbG8=")
```

## URL Encoding

### Encode and decode

```bash
# Python one-liner
python3 -c "from urllib.parse import quote; print(quote('hello world & foo=bar'))"
# hello%20world%20%26%20foo%3Dbar

# Decode
python3 -c "from urllib.parse import unquote; print(unquote('hello%20world%20%26%20foo%3Dbar'))"
# hello world & foo=bar

# curl does it automatically for --data-urlencode
curl -G --data-urlencode "q=hello world & more" https://api.example.com/search
```

### In code

```javascript
// JavaScript
encodeURIComponent('hello world & foo=bar');
// "hello%20world%20%26%20foo%3Dbar"

decodeURIComponent('hello%20world%20%26%20foo%3Dbar');
// "hello world & foo=bar"

// encodeURI vs encodeURIComponent:
encodeURI('https://example.com/path?q=hello world');
// "https://example.com/path?q=hello%20world" (preserves URL structure)
encodeURIComponent('https://example.com/path?q=hello world');
// "https%3A%2F%2Fexample.com%2Fpath%3Fq%3Dhello%20world" (encodes everything)
```

```python
from urllib.parse import quote, unquote, urlencode

quote('hello world')            # 'hello%20world'
unquote('hello%20world')        # 'hello world'
urlencode({'q': 'hello world', 'page': 1})  # 'q=hello+world&page=1'
```

## Hex

### View and convert

```bash
# File hex dump
xxd file.bin | head -20
xxd -l 64 file.bin          # First 64 bytes only

# Hex dump (compact, no ASCII)
xxd -p file.bin

# Convert hex to binary
echo "48656c6c6f" | xxd -r -p
# Hello

# od (alternative)
od -A x -t x1z file.bin | head -20

# hexdump
hexdump -C file.bin | head -20

# Python
python3 -c "print(bytes.fromhex('48656c6c6f').decode())"  # Hello
python3 -c "print('Hello'.encode().hex())"                 # 48656c6c6f
```

### In code

```javascript
// JavaScript
Buffer.from('Hello').toString('hex');           // "48656c6c6f"
Buffer.from('48656c6c6f', 'hex').toString();    // "Hello"

// Number to hex
(255).toString(16);     // "ff"
parseInt('ff', 16);     // 255
```

```python
# Python
"Hello".encode().hex()                  # '48656c6c6f'
bytes.fromhex('48656c6c6f').decode()    # 'Hello'
hex(255)                                # '0xff'
int('ff', 16)                           # 255
```

## Unicode

### Inspect characters

```bash
# Show Unicode code points
echo -n "Hello 世界" | python3 -c "
import sys
for char in sys.stdin.read():
    print(f'U+{ord(char):04X}  {char}  {char.encode(\"utf-8\").hex()}')"
# U+0048  H  48
# U+0065  e  65
# ...
# U+4E16  世  e4b896
# U+754C  界  e7958c

# Convert Unicode escape to character
printf '\u0048\u0065\u006c\u006c\u006f'   # Hello
echo -e '\xE4\xB8\x96\xE7\x95\x8C'       # 世界

# File encoding detection
file -bi document.txt
# text/plain; charset=utf-8
```

### Encoding conversion

```bash
# Convert between encodings
iconv -f ISO-8859-1 -t UTF-8 input.txt > output.txt
iconv -f UTF-16 -t UTF-8 input.txt > output.txt

# List available encodings
iconv -l

# Python
python3 -c "
with open('latin1.txt', 'r', encoding='iso-8859-1') as f:
    content = f.read()
with open('utf8.txt', 'w', encoding='utf-8') as f:
    f.write(content)
"
```

### Common Unicode issues

```
BOM (Byte Order Mark):
  UTF-8 BOM: EF BB BF at start of file
  Remove: sed -i '1s/^\xEF\xBB\xBF//' file.txt

Normalization (NFC vs NFD):
  "é" can be U+00E9 (one char) or U+0065 U+0301 (e + combining accent)
  Python: import unicodedata; unicodedata.normalize('NFC', text)

Mojibake (wrong encoding):
  "café" appears as "café" → file is UTF-8 but read as Latin-1
  Fix: re-read with correct encoding
```

## JWT (JSON Web Tokens)

### Decode a JWT

```bash
# JWT has 3 parts separated by dots: header.payload.signature
# Each part is Base64url-encoded

# Decode header and payload
TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.SflKxwRJSMeKKF2QT4fwpMeJf36POk6yJV_adQssw5c"

# Decode header
echo "$TOKEN" | cut -d. -f1 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"alg":"HS256","typ":"JWT"}

# Decode payload
echo "$TOKEN" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
# {"sub":"1234567890","name":"John Doe","iat":1516239022}

# One-liner function
jwt_decode() {
    echo "$1" | cut -d. -f2 | tr '-_' '+/' | base64 -d 2>/dev/null | jq
}
jwt_decode "$TOKEN"
```

### In code

```javascript
// JavaScript (no library needed for decoding)
function decodeJWT(token) {
    const [header, payload] = token.split('.').slice(0, 2)
        .map(part => JSON.parse(atob(part.replace(/-/g, '+').replace(/_/g, '/'))));
    return { header, payload };
}

// Check expiry
function isJWTExpired(token) {
    const { payload } = decodeJWT(token);
    return payload.exp && payload.exp < Math.floor(Date.now() / 1000);
}
```

```python
# Python
import json, base64

def decode_jwt(token):
    parts = token.split('.')
    # Add padding
    def pad(s): return s + '=' * (4 - len(s) % 4)
    header = json.loads(base64.urlsafe_b64decode(pad(parts[0])))
    payload = json.loads(base64.urlsafe_b64decode(pad(parts[1])))
    return header, payload

header, payload = decode_jwt(token)
```

## Hashing

### Common hash functions

```bash
# MD5 (not for security — only for checksums/dedup)
echo -n "Hello" | md5sum        # Linux
echo -n "Hello" | md5           # macOS

# SHA-256 (standard for integrity)
echo -n "Hello" | sha256sum
echo -n "Hello" | shasum -a 256

# SHA-1 (deprecated for security, still used in git)
echo -n "Hello" | sha1sum

# SHA-512
echo -n "Hello" | sha512sum

# Hash a file
sha256sum file.bin
md5sum file.bin

# openssl (works everywhere)
echo -n "Hello" | openssl dgst -sha256
openssl dgst -sha256 file.bin
```

### In code

```javascript
// Node.js
const crypto = require('crypto');
crypto.createHash('sha256').update('Hello').digest('hex');
// "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"

// File hash
const fs = require('fs');
const hash = crypto.createHash('sha256');
hash.update(fs.readFileSync('file.bin'));
console.log(hash.digest('hex'));
```

```python
import hashlib
hashlib.sha256(b"Hello").hexdigest()
# "185f8db32271fe25f561a6fc938b2e264306ec304eda518007d1764826381969"

# File hash
with open("file.bin", "rb") as f:
    print(hashlib.sha256(f.read()).hexdigest())
```

### Checksums for file integrity

```bash
# Generate checksum file
sha256sum *.tar.gz > checksums.sha256

# Verify checksums
sha256sum -c checksums.sha256

# Compare