grafana-lens

Grafana tools for data visualization, monitoring, alerting, security, and SRE investigation. Use grafana_query, grafana_query_logs, grafana_query_traces, grafana_create_dashboard, grafana_update_dashboard, grafana_create_alert, grafana_share_dashboard, grafana_annotate, grafana_explore_datasources, grafana_list_metrics, grafana_search, grafana_get_dashboard, grafana_check_alerts, grafana_push_metrics, grafana_explain_metric, grafana_security_check, and grafana_investigate. Trigger when asked about metrics, dashboards, monitoring, alerts, costs, token usage, data visualization, PromQL, Prometheus, LogQL, Loki, log queries, error logs, log search, TraceQL, Tempo, traces, distributed tracing, span search, find slow traces, debug session traces, annotations, deployments, sharing charts, investigating alert notifications, pushing custom data (calendar, git, fitness, finance) to Grafana for visualization, pushing historical data, backfilling metrics, recording past data with timestamps, modifying dashboards, adding panels, removing panels, changing dashboard settings, updating dashboard time range, explain metric, metric trend, what is this metric, how has this changed, is this metric normal, why did my bill spike, cost visibility, security monitoring, security check, security audit, am I being attacked, is my agent compromised, suspicious activity, threat detection, prompt injection detection, set up security alerts, investigate, debug, triage, root cause, what's wrong, why is X broken, anomaly detection, RED method, USE method, alert fatigue, postmortem, incident summary.

# Grafana Lens

You have full native Grafana access — query data, create dashboards, set alerts, receive alert notifications, annotate events, explore datasources, push custom data, and deliver visualizations inline. Works with ANY data in Grafana, not just agent metrics.

## Musts

- **Always call `grafana_explore_datasources` first** when you need a datasource UID — never guess UIDs
- **Always call `grafana_search` before creating a dashboard** — avoid duplicates
- **Always call `grafana_get_dashboard` before `grafana_share_dashboard`** — you need exact panel IDs
- **Always call `grafana_get_dashboard` before `grafana_update_dashboard`** — you need panel IDs and current structure
- **Prefer `grafana_query` for direct answers** over creating dashboards — "what's my cost?" needs a number, not a URL
- **Prefer `grafana_query` over `grafana_create_dashboard` + `grafana_share_dashboard`** for simple data questions — a number is faster than a chart
- **Use `grafana_query_logs` for log searches** — LogQL for logs, PromQL for metrics, TraceQL for traces. Never use `grafana_query` for Loki datasources
- **Use `grafana_query_traces` for trace searches** — TraceQL for traces, PromQL for metrics, LogQL for logs. Never use `grafana_query` or `grafana_query_logs` for Tempo datasources
- **All tools work with ANY Prometheus datasource** — not just `openclaw_lens_*` metrics
- **When you see "GRAFANA ALERTS" in prompt context**, investigate immediately with `grafana_check_alerts` — use the `suggestedInvestigation` field to go directly to querying (it provides the tool, query, and datasource)
- **Run `grafana_check_alerts` with action `setup` once** before alert notifications can reach the agent — this creates the webhook contact point
- **Push data before querying or dashboarding it** — data is pushed via OTLP and available immediately
- **Prefer `grafana_explain_metric` for "what is this metric?" questions** over manual `grafana_query` — it returns current value, trend, stats, and metadata in one call
- **Use `queryNames` from push response for PromQL queries** — don't guess metric names (counters get `_total` suffix)
- **Use `openclaw_ext_` prefix for custom metrics** — `grafana_push_metrics` auto-prepends it if missing
- **Follow statistics-first discipline for log investigation** — always run count/rate LogQL before reading individual entries. Use `grafana_query_logs` with metric-over-logs queries (`count_over_time`, `rate`, `topk`) before switching to raw log entries
- **Silence alerts during investigation** — use `grafana_check_alerts` with action `silence` to prevent repeat notifications while investigating
- **Use `list_rules` for complete alert health** — `grafana_check_alerts` with action `list_rules` returns all rules with live eval state (normal/firing/pending/nodata/error), health, and lastEvaluation — no need to cross-reference with `list` action
- **Use `dashboardUid` + `panelId` to re-run panel queries** — don't manually extract PromQL/LogQL from `get_dashboard` output. Both `grafana_query` and `grafana_query_logs` accept these params to auto-resolve the panel's query expression and datasource. The tool handles template variable replacement and datasource routing automatically
- **Confirm with user before deleting dashboards or alert rules** — `grafana_update_dashboard` with operation `delete` and `grafana_check_alerts` with action `delete_rule` are permanent and cannot be undone

## Quick Decision Tree

- "What is [metric]?" / "Why did it spike?" → `grafana_explain_metric`
- "What's the current value of X?" / complex PromQL → `grafana_query`
- "Find error logs" / "Search logs for..." → `grafana_query_logs`
- "Find slow traces" / "Show trace for session X" / "Debug distributed spans" → `grafana_query_traces`
- "Debug this session" / "Why did it fail?" / "What went wrong?" → `grafana_query_traces` (search error/slow) → `grafana_query_traces` (get → follow `correlationHint`) → `grafana_query_logs` → `grafana_query` → `grafana_annotate`
- "Show me a chart" / "Visualize..." → `grafana_search` → `grafana_get_dashboard` → `grafana_share_dashboard`
- "Create a dashboard for..." → `grafana_search` (check duplicates) → `grafana_create_dashboard`
- "Add a panel to my dashboard" → `grafana_get_dashboard` → `grafana_update_dashboard`
- "Delete this dashboard" → `grafana_update_dashboard` with operation `delete` (confirm with user first)
- "Alert me when..." → `grafana_check_alerts` (setup) → `grafana_create_alert`
- "List my alert rules" / "What alerts do I have?" → `grafana_check_alerts` with action `list_rules`
- "Delete alert rule X" → `grafana_check_alerts` with action `list_rules` → `delete_rule` with `ruleUid`
- "Track my [custom data]" / "Record my [past data]" → `grafana_push_metrics` (with optional `timestamp` for historical data, auto-registers, returns `queryNames`) → `grafana_query` with `queryNames`
- "What data sources do I have?" → `grafana_explore_datasources`
- "What metrics are available?" → `grafana_list_metrics`
- "Set up monitoring" / "Monitor my agent" / "What dashboards should I have?" → `grafana_search` (check existing) → `grafana_create_dashboard` with `llm-command-center` → follow `suggestedNext` chain through remaining templates
- "GenAI observability" / "OTel gen_ai metrics" / "Standard AI monitoring" → `grafana_create_dashboard` with `genai-observability` template
- "What happened in session X?" / "Debug this session" → `grafana_create_dashboard` with `session-explorer` template → paste session ID
- "Show me LLM traces" / "Show agent logs" → `grafana_create_dashboard` with `llm-command-center` template (Loki + Tempo)
- "How much am I spending?" / "Cost analysis" → `grafana_create_dashboard` with `cost-intelligence` template
- "Which tools are slow?" / "Tool errors" → `grafana_create_dashboard` with `tool-performance` template
- "Queue health" / "Webhook issues" / "Stuck sessions" → `grafana_create_dashboard` with `sre-operations` template
- "System health check" / "Status report" / "Review all dashboards" → `grafana_explore_datasources` → `grafana_check_alerts` (list + list_rules) → `grafana_search` → `grafana_get_dashboard` (audit=true for each) → summarize
- "Audit my dashboard" / "Which panels are broken?" → `grafana_get_dashboard` (audit=true) → review `auditSummary` + per-panel `health`
- "Am I being attacked?" / "Security check" / "Security status" → `grafana_security_check`
- "Set up security monitoring" → `grafana_check_alerts` (setup) → `grafana_create_dashboard` (`security-overview`) → `grafana_create_alert` (webhook error burst, cost spike, tool loops, injection signals)
- "Investigate security alert" → `grafana_security_check` → `grafana_query_logs` (correlate) → `grafana_annotate` (mark investigation) → `grafana_check_alerts` (silence)
- "Investigate this alert" / "Why is X broken?" / "Debug this issue" / "Triage" / "Root cause" → `grafana_investigate` (multi-signal triage) → follow `suggestedHypotheses.testWith` for deep-dives
- "Is this metric normal?" / "Is there an anomaly?" → `grafana_explain_metric` (returns `anomaly` z-score + `seasonality` vs 1d/7d ago for 24h period)
- "RED analysis" / "What's the error rate?" / "Service health" → RED Method queries (see sre-investigation.md §2)
- "Alert fatigue" / "Which alerts are noisy?" / "Alert health" → `grafana_check_alerts` with action `analyze` — fatigue report
- "Postmortem" / "Incident summary" / "What happened?" → `grafana_investigate` → 5-Phase methodology → postmortem template (see sre-investigation.md §9)
- "Compare before/after deployment" → `grafana_annotate` (list, tags: ["deploy"]) → `grafana_explain_metric` (compareWith: "previous")

## Tool Inventory

| Tool | What It Does |
|------|-------------|
| `grafana_explore_datasources` | Discover configured datasources (UIDs, types, query routing) — tells you which tool + query language to use for each datasource |
| `grafana_list_metrics` | Discover available metrics or label values from a datasource. Use `comp