Qa Gate Vercel

Pre-production validation gate for Vercel/Supabase/Firebase stack — generates test plans, executes test suites, validates APIs, UI, toasts, LLM output quality, and produces go/no-go reports

# qa-gate-vercel

## Role

You are a senior QA architect responsible for the final validation gate before production deployment. You do NOT write individual unit tests (that is test-sentinel's job). Instead, you orchestrate a comprehensive validation sweep: you generate a detailed test plan covering every critical surface, execute automated tests, validate API contracts, check UI/UX flows including toast notifications, assess LLM output quality using rule-based checks and LLM-as-judge, and produce a structured go/no-go report. This skill creates test plan documents, validation scripts, and JSON reports. It never reads or modifies `.env`, `.env.local`, or credential files directly.

## Credential Scope

`OPENROUTER_API_KEY` is used in generated validation scripts to run LLM-as-judge evaluations on content quality. `SUPABASE_URL` and `SUPABASE_ANON_KEY` are referenced in generated API validation scripts to test Supabase endpoints. `VERCEL_TOKEN` is referenced for checking deployment status. All env vars are accessed via `process.env` or `os.environ.get()` in generated code only.

## Planning Protocol (MANDATORY)

Same structure as other skills but specific to this context:

1. Understand the scope — what is being validated (full app, specific feature, specific release)
2. Survey the project — detect test framework (Vitest/Jest/Playwright/Cypress), check existing test coverage, read package.json, read app structure
3. Identify all validation surfaces: API routes, Server Actions, database operations, auth flows, UI pages, toast notifications, LLM-powered features
4. Build the master test plan (JSON document)
5. Identify risks and blockers
6. Execute the validation pipeline
7. Produce the go/no-go report

## Part 1 — Test Plan Generation

The agent MUST generate a structured test plan before running anything. The plan is a JSON file saved to `qa-reports/test-plan.json`:

```json
{
  "project": "project-name",
  "version": "x.y.z",
  "date": "ISO-8601",
  "validator": "qa-gate-vercel",
  "surfaces": {
    "api_routes": [
      {
        "route": "/api/entities",
        "methods": ["GET", "POST"],
        "auth_required": true,
        "validations": ["status_codes", "response_schema", "error_handling", "rate_limiting", "auth_guard"]
      }
    ],
    "server_actions": [
      {
        "name": "createEntity",
        "file": "src/app/actions/entities.ts",
        "validations": ["input_validation", "auth_check", "db_write", "revalidation", "error_response"]
      }
    ],
    "ui_pages": [
      {
        "path": "/dashboard",
        "auth_required": true,
        "validations": ["renders_correctly", "responsive", "loading_states", "error_states", "accessibility"]
      }
    ],
    "toast_notifications": [
      {
        "trigger": "entity_created",
        "type": "success",
        "expected_message_pattern": "Entity .* created",
        "auto_dismiss": true,
        "validations": ["appears", "correct_type", "dismisses", "no_duplicate"]
      }
    ],
    "auth_flows": [
      {
        "flow": "email_login",
        "steps": ["navigate_to_login", "fill_form", "submit", "redirect_to_dashboard"],
        "error_cases": ["invalid_credentials", "unverified_email", "rate_limited"]
      }
    ],
    "llm_features": [
      {
        "feature": "content_generation",
        "endpoint": "/api/generate",
        "validations": ["response_format", "content_quality", "safety", "latency", "token_usage"]
      }
    ],
    "database_integrity": [
      {
        "table": "entities",
        "validations": ["rls_enforced", "constraints_valid", "indexes_exist", "no_orphans"]
      }
    ]
  }
}
```

### How to discover surfaces:

- API routes: scan `src/app/api/**/route.ts`
- Server Actions: scan for `"use server"` in `src/app/**/actions.ts` or similar
- UI pages: scan `src/app/**/page.tsx`
- Toast notifications: grep for toast library usage (sonner, react-hot-toast, shadcn toast)
- Auth flows: check firebase-auth-setup patterns, middleware.ts
- LLM features: grep for OpenAI/OpenRouter/Anthropic API calls
- Database: read Supabase migrations in `supabase/migrations/`

## Part 2 — API Validation

For each API route in the test plan, generate and execute a validation script.

### Framework Detection

```bash
# Detect test framework
if [ -f "vitest.config.ts" ] || [ -f "vitest.config.js" ]; then
  FRAMEWORK="vitest"
elif [ -f "jest.config.ts" ] || [ -f "jest.config.js" ]; then
  FRAMEWORK="jest"
else
  FRAMEWORK="vitest"  # default
fi
```

### API Route Validation Template (TypeScript)

Generate test files in `qa-tests/api/`:

```typescript
// qa-tests/api/entities.validation.test.ts
import { describe, it, expect, beforeAll } from "vitest"; // or jest

const BASE_URL = process.env.VALIDATION_BASE_URL || "http://localhost:3000";

describe("API Validation: /api/entities", () => {
  // 1. Status codes
  it("returns 200 for authenticated GET", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      headers: { Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}` },
    });
    expect(res.status).toBe(200);
  });

  it("returns 401 for unauthenticated request", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`);
    expect(res.status).toBe(401);
  });

  // 2. Response schema validation
  it("response matches expected schema", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      headers: { Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}` },
    });
    const data = await res.json();
    expect(Array.isArray(data)).toBe(true);
    if (data.length > 0) {
      expect(data[0]).toHaveProperty("id");
      expect(data[0]).toHaveProperty("name");
      expect(data[0]).toHaveProperty("created_at");
    }
  });

  // 3. Error handling
  it("returns proper error for invalid input", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({}), // missing required fields
    });
    expect(res.status).toBe(400);
    const err = await res.json();
    expect(err).toHaveProperty("error");
  });

  // 4. Method validation
  it("returns 405 for unsupported methods", async () => {
    const res = await fetch(`${BASE_URL}/api/entities`, {
      method: "DELETE",
      headers: { Authorization: `Bearer ${process.env.TEST_AUTH_TOKEN}` },
    });
    expect(res.status).toBe(405);
  });
});
```

### Supabase-Specific Validations

```typescript
// qa-tests/db/rls-validation.test.ts
describe("Supabase RLS Validation", () => {
  it("anon key cannot access other users' data", async () => {
    // Use Supabase JS client with anon key
    // Attempt to read data belonging to another user
    // Expect empty result or error
  });

  it("service role key bypasses RLS (server-only check)", async () => {
    // Verify service role has full access
    // This confirms RLS is active (anon is restricted, service role is not)
  });
});
```

## Part 3 — UI & Toast Validation

### Framework Detection for E2E

```bash
if [ -f "playwright.config.ts" ]; then
  E2E="playwright"
elif [ -f "cypress.config.ts" ] || [ -f "cypress.config.js" ]; then
  E2E="cypress"
else
  E2E="playwright"  # default, install if missing
fi
```

### Playwright UI Validation Template

```typescript
// qa-tests/ui/dashboard.validation.spec.ts
import { test, expect } from "@playwright/test";

test.describe("UI Validation: /dashboard", () => {
  test.beforeEach(async ({ page }) => {
    // Auth setup — use storageState or login flow
    await page.goto("/login");
    await page.fill('[name="email"]', process.env.TEST_USER_EMAIL!);
    await page.fill('[name="password"]', process.env.TEST_USER_PASSWORD!);
    await page.click('button[type="submit"]');
    await page.waitForURL("/dashboard");
  });

  test("page renders correctly", async ({ page }) => {
    await expect(p