tonic-system-deploy

Software deployment workflow for systems with separate UAT and PROD environments. Use when: planning a bug fix deployment, choosing between Flow 1 (UAT-first) and Flow 2 (PROD-first), handling emergency hotfixes, executing rollbacks, or designing automated nightly deploy pipelines. Covers approval gates, human checkpoints, system automation nodes, Telegram notifications, and rollback procedures.

# tonic-system-deploy

Software Deployment Workflow — Dual-Environment (UAT + PROD)

---

## Background & Design Rationale

This skill was designed for systems where:

- **Two live environments co-exist**: UAT (testing/staging) and PROD (production)
- **Versions can diverge**: UAT may be ahead of PROD by several releases
- **Deployments are nightly**: automated pipelines run at scheduled times
- **Human approval is mandatory**: no code goes to PROD without explicit admin sign-off
- **Bugs require structured triage**: severity, origin environment, and version state all affect the deploy path

The key insight: **choosing the wrong deploy flow when versions are mismatched can introduce regressions**. Flow 1 assumes parity; Flow 2 handles divergence safely.

---

## Prerequisites — Before Choosing a Flow

### Step 0: Version Check (always do this first)

| Question | Answer → |
|----------|----------|
| Are UAT and PROD on the same version? | → **Flow 1** |
| Is UAT ahead of PROD by any version? | → **Flow 2** |
| Is this a critical/high severity bug? | → **Emergency Hotfix** (bypass pipeline) |
| Do you need to undo a bad deploy? | → **Rollback** |

### Version Mismatch Decision Tree

```
Bug found
    │
    ├─ severity = critical/high?
    │       └─ YES → Emergency Hotfix (skip pipeline)
    │
    ├─ UAT version == PROD version?
    │       └─ YES → Flow 1
    │
    └─ UAT version > PROD version?
            └─ YES → Flow 2
```

---

## Flow 1 — UAT-First (Versions Aligned)

**Scenario**: Bug found in UAT or PROD when both environments run the same version.
**Goal**: Fix → validate in UAT → promote to PROD.
**Result**: UAT and PROD converge to same patched version.

### Timeline

```
Bug reported
    │
    │ 🧑 HUMAN: Admin reviews + confirms bug (status: confirmed)
    ▼
[confirmed] — severity: low/medium only
    │
    │ 🤖 SYSTEM: Scheduled deploy time T1 (e.g. 20:00)
    │   - AI analyses root cause + records fix plan
    │   - Deploys fix to UAT environment
    │   - Status → deployed_uat
    ▼
[deployed_uat]
    │ 📲 Telegram: "Fix deployed to UAT. Please validate."
    │
    │ 🧑 HUMAN: Admin logs into UAT, validates fix
    │   - Runs through affected workflows
    │   - Confirms no regression
    │   - Clicks "Approve PROD Deploy" → status: pending_prod
    ▼
[pending_prod]
    │ 📲 Telegram: "Queued for PROD at T2."
    │
    │ 🤖 SYSTEM: Scheduled deploy time T2 (e.g. 22:00)
    │   - Deploys fix to PROD environment
    │   - Status → deployed_prod
    ▼
[deployed_prod] ✅ Flow 1 Complete
    │ 📲 Telegram: "Deployed to PROD. Flow 1 complete."
```

### Human Checkpoints (Flow 1)

| Checkpoint | Who | Action | Gate Condition |
|------------|-----|--------|----------------|
| Confirm bug | Admin/Manager | Mark as confirmed | Bug is reproducible and valid |
| UAT validation | Admin/Manager | Click "Approve PROD Deploy" | Fix works, no regression in UAT |

### Automation Nodes (Flow 1)

| Time | Node | Input Status | Output Status | Action |
|------|------|-------------|---------------|--------|
| T1 | Phase 1 | confirmed/planned | deployed_uat | AI analysis + UAT deploy |
| T2 | Phase 2 | pending_prod | deployed_prod | PROD deploy |

---

## Flow 2 — PROD-First (Versions Misaligned)

**Scenario**: Bug found in PROD when UAT is ahead by one or more versions.
**Why not Flow 1?** Validating a PROD fix in a newer UAT environment risks false confidence — the fix may behave differently on the older PROD codebase.
**Goal**: Fix PROD directly → validate in PROD → cherry-pick back to UAT.
**Result**: PROD gets the fix immediately; UAT gets it merged back after PROD validation.

### Timeline

```
Bug found in PROD (UAT is ahead)
    │
    │ 🧑 HUMAN: Admin reviews + confirms bug
    │   - Selects: found_in_env = prod, fix_flow = flow2
    │   - Status → confirmed
    ▼
[confirmed]
    │
    │ 🤖 SYSTEM: Scheduled deploy time T1 (e.g. 20:00)
    │   - AI analyses root cause + records fix plan
    │   - Skips UAT entirely
    │   - Queues for PROD deploy → status: pending_prod
    ▼
[pending_prod]
    │ 📲 Telegram: "PROD deploy queued for T2 (Flow 2)."
    │
    │ 🤖 SYSTEM: Scheduled deploy time T2 (e.g. 22:00)
    │   - Deploys fix to PROD
    │   - Status → deployed_prod
    ▼
[deployed_prod]
    │ 📲 Telegram: "Deployed to PROD. Please validate PROD. Approve UAT merge when ready."
    │
    │ 🧑 HUMAN: Admin validates fix in PROD
    │   - Confirms fix works on production data/config
    │   - No regression in PROD workflows
    │   - Clicks "Approve Merge UAT" → status: pending_uat_merge
    ▼
[pending_uat_merge]
    │ 📲 Telegram: "UAT merge queued for T2 tonight."
    │
    │ 🤖 SYSTEM: Next T2 cycle (22:00)
    │   - Deploys/merges fix into UAT environment
    │   - Status → uat_merged
    ▼
[uat_merged] ✅ Flow 2 Complete
    │ 📲 Telegram: "Merged to UAT. Flow 2 complete."
```

### Human Checkpoints (Flow 2)

| Checkpoint | Who | Action | Gate Condition |
|------------|-----|--------|----------------|
| Confirm bug | Admin/Manager | Mark as confirmed + select flow2 | Bug confirmed in PROD, version mismatch verified |
| PROD validation | Admin/Manager | Click "Approve Merge UAT" | Fix verified in PROD, no regression |

### Automation Nodes (Flow 2)

| Time | Node | Input Status | Output Status | Action |
|------|------|-------------|---------------|--------|
| T1 | Phase 1 | confirmed/planned | pending_prod | AI analysis (skip UAT) |
| T2 | Phase 2a | pending_prod | deployed_prod | PROD deploy |
| T2 (next) | Phase 2b | pending_uat_merge | uat_merged | UAT deploy/merge |

### Flow 2 Important Note

> **T2 deadline matters.** If admin approves UAT merge before T2 on the same day, the merge runs that night. If approved after T2, it runs the following night's T2. Always communicate the cutoff time to the team.

---

## Status Reference

| Status | Flow | Colour | Meaning | Next Action |
|--------|------|--------|---------|-------------|
| `submitted` | Both | Grey | Bug reported, awaiting review | Admin confirms/rejects |
| `confirmed` | Both | Blue | Valid bug, enters pipeline | T1 auto-process |
| `analyzing` | Both | Purple | AI analysis running (transient) | Auto → planned |
| `planned` | Both | Indigo | AI fix plan recorded | T1 auto-deploy |
| `deployed_uat` | Flow 1 | Cyan | UAT deployed, awaiting human validation | Admin approves PROD |
| `pending_prod` | Both | Yellow | Queued for PROD at next T2 | T2 auto-deploy |
| `deployed_prod` | Both | Green | PROD deployed | Flow1: done; Flow2: admin approves UAT merge |
| `pending_uat_merge` | Flow 2 | Purple | Queued for UAT merge at next T2 | T2 auto-merge |
| `uat_merged` | Flow 2 | Teal | UAT updated with PROD fix | Flow 2 complete ✅ |
| `closed` | Both | Emerald | Manually closed | — |
| `rejected` | Both | Red | Not a valid bug | — |

---

## Severity Rules

| Severity | Pipeline Eligible? | Notes |
|----------|--------------------|-------|
| `low` | ✅ Yes | Both flows |
| `medium` | ✅ Yes | Both flows |
| `high` | ❌ No | Emergency Hotfix only |
| `critical` | ❌ No | Emergency Hotfix, immediate escalation |

> Never let high/critical bugs wait for a scheduled pipeline. Treat them as emergency hotfixes.

---

## Emergency Hotfix (Bypass Pipeline)

**Scenario**: Critical or high severity bug in PROD. Cannot wait for scheduled T1/T2.

### Process

```
Critical bug found in PROD
    │
    │ 🧑 HUMAN: Admin confirms severity = critical/high
    │   - Does NOT enter pipeline (no confirmed status)
    │   - Opens direct hotfix branch
    ▼
Fix developed (manually or with AI assist)
    │
    │ 🧑 HUMAN: Admin deploys directly to PROD
    │   - Updates bug status to deployed_prod manually
    │   - Records fix details in ai_fix_diff field
    ▼
[deployed_prod] (manual)
    │ 📲 Telegram: "Emergency hotfix deployed to PROD. [Bug title]"
    │
    │ 🧑 HUMAN: Validates PROD immediately
    │
    └─ If UAT is ahead → manually cherry-pick to UAT branch
       If UAT is same version → update UAT as well