added alerting rules for grafana
This commit is contained in:
rezayaghobi-dev
150
docs/Alerting.md
Normal file
150
docs/Alerting.md
Normal file
@@ -0,0 +1,150 @@
|
||||
# Alerting
|
||||
|
||||
Grafana's built-in unified alerting ties directly into the Prometheus and Loki datasources already running in the [Monitoring Stack](Monitoring-Stack.md) — no separate Alertmanager container needed. All rules, contact points, and notification policies are provisioned as code in `grafana/provisioning/alerting/`, so they survive container rebuilds and are version-controlled alongside everything else in this repo.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Prometheus ──┐
|
||||
├──► Grafana Unified Alerting ──► Gmail (SMTP)
|
||||
Loki ────────┘
|
||||
```
|
||||
|
||||
Grafana evaluates alert rules against its configured datasources (Prometheus for metrics, Loki for logs). When a rule fires, the notification policy routes it to a contact point, which delivers the notification via the configured channel (email over SMTP in this setup).
|
||||
|
||||
No separate Alertmanager instance is required — Grafana handles the full pipeline internally.
|
||||
|
||||
## Alert rules
|
||||
|
||||
All rules live in `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml` under the **Infra** folder:
|
||||
|
||||
| Rule | UID | Source | Condition | Severity | Duration |
|
||||
|---|---|---|---|---|---|
|
||||
| **Node exporter target down** | `node-down` | Prometheus | `up{job="node_exporter"} < 1` | `critical` | 2 min |
|
||||
| **High CPU usage** | `high-cpu` | Prometheus | `100 - (avg idle rate) > 85%` | `warning` | 5 min |
|
||||
| **Error log rate spike** | `loki-error-spike` | Loki | `count_over_time(… \|= "error" [5m]) > 20` | `warning` | 5 min |
|
||||
|
||||
### Node exporter target down (`node-down`)
|
||||
|
||||
The most critical alert — if the node exporter scrapes as down for 2 minutes, the host itself may be unreachable or the exporter process crashed. Fires at `critical` severity because it means all host-level visibility is lost.
|
||||
|
||||
**PromQL:**
|
||||
```promql
|
||||
up{job="node_exporter"}
|
||||
```
|
||||
|
||||
**Threshold:** `< 1` (target is down)
|
||||
|
||||
### High CPU usage (`high-cpu`)
|
||||
|
||||
Watches for sustained CPU usage above 85% across the 5-minute window. The query uses `rate(node_cpu_seconds_total{mode="idle"}[5m])` to smooth out momentary spikes — a brief compile or Docker pull won't trigger it, but a runaway process or resource-hungry container will.
|
||||
|
||||
**PromQL:**
|
||||
```promql
|
||||
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
|
||||
```
|
||||
|
||||
**Threshold:** `> 85%`
|
||||
|
||||
### Error log rate spike (`loki-error-spike`)
|
||||
|
||||
Queries Loki for log lines containing "error" across all jobs. If the count exceeds 20 within a 5-minute window and stays elevated for 5 minutes, it fires. This catches application-level issues that metrics alone won't surface — a service that's technically "up" but flooding its logs with errors.
|
||||
|
||||
**LogQL:**
|
||||
```logql
|
||||
sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)
|
||||
```
|
||||
|
||||
**Threshold:** `> 20` error lines in 5 minutes
|
||||
|
||||
## Contact points
|
||||
|
||||
Defined in `grafana/provisioning/alerting/contactpoints.yaml`:
|
||||
|
||||
| Name | Type | Channel |
|
||||
|---|---|---|
|
||||
| `gmail-alerts` | Email | SMTP via Gmail App Password |
|
||||
|
||||
The email address is set to a placeholder (`YOUR_EMAIL@example.com`) in the provisioned file — update it to your actual address after the first Grafana container starts, or edit it directly in the file before starting.
|
||||
|
||||
> **Note:** Grafana provisioning files do not expand Docker environment variables (`${...}` syntax). The email address must be set as a literal string in the YAML.
|
||||
|
||||
## Notification policy
|
||||
|
||||
Defined in `grafana/provisioning/alerting/policies.yaml`:
|
||||
|
||||
| Setting | Value | Meaning |
|
||||
|---|---|---|
|
||||
| **Receiver** | `gmail-alerts` | All alerts route to the email contact point |
|
||||
| **Group by** | `alertname`, `instance` | Alerts with the same name and instance are grouped into a single email |
|
||||
| **Group wait** | 30s | How long to wait before sending the first notification for a new group |
|
||||
| **Group interval** | 5 min | Minimum time between notifications for the same group |
|
||||
| **Repeat interval** | 4 hours | How often to re-notify about an ongoing (unresolved) alert |
|
||||
|
||||
This means a firing alert sends its first email within ~30 seconds, groups follow-ups every 5 minutes, and you get a reminder every 4 hours until the alert resolves.
|
||||
|
||||
## SMTP configuration
|
||||
|
||||
Grafana sends alert emails through Gmail's SMTP relay. The configuration lives in the `grafana` service's environment block in `docker-compose.yml` and pulls credentials from the `.env` file:
|
||||
|
||||
```yaml
|
||||
GF_SMTP_ENABLED=true
|
||||
GF_SMTP_HOST=smtp.gmail.com:587
|
||||
GF_SMTP_USER=${GMAIL_SMTP_USER}
|
||||
GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD}
|
||||
GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER}
|
||||
GF_SMTP_FROM_NAME=Grafana Alerts
|
||||
```
|
||||
|
||||
### Setting up Gmail SMTP
|
||||
|
||||
1. Enable 2-Step Verification on your Google account
|
||||
2. Generate an [App Password](https://myaccount.google.com/apppasswords) (requires 2FA)
|
||||
3. Add both values to `.env`:
|
||||
```
|
||||
GMAIL_SMTP_USER=your-email@gmail.com
|
||||
ALERT_EMAIL=your-email@gmail.com
|
||||
```
|
||||
4. Set the same email as your Grafana login (`GF_SECURITY_ADMIN_USER` in `.env`, or through the Grafana UI)
|
||||
|
||||
## How to add new alert rules
|
||||
|
||||
Rules are provisioned from YAML — no UI clicks needed:
|
||||
|
||||
1. Open `grafana/provisioning/alerting/rules.yaml`
|
||||
2. Add a new entry under the `rules` list in the `infra-alerts` group
|
||||
3. Give it a unique `uid`, a descriptive `title`, and set the `datasourceUid` to match your Prometheus (`efck3im1w4pvkf`) or Loki (`P8E80F9AEF21F6940`) datasource
|
||||
4. Set the `condition` refId to point to the threshold expression
|
||||
5. Set `for` to how long the condition must be true before firing
|
||||
6. Add `labels.severity` (`critical`, `warning`, `info`) and an `annotations.summary` message
|
||||
7. Restart the Grafana container: `docker compose restart grafana`
|
||||
|
||||
### Common PromQL examples for new rules
|
||||
|
||||
```promql
|
||||
# Memory usage above 90%
|
||||
100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100)
|
||||
|
||||
# Disk usage above 85%
|
||||
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)
|
||||
|
||||
# Container restart count (from cAdvisor)
|
||||
increase(container_restart_count[5m])
|
||||
|
||||
# Container memory above limit
|
||||
container_memory_working_set_bytes / container_spec_memory_limit_bytes
|
||||
```
|
||||
|
||||
## How to add new notification channels
|
||||
|
||||
Grafana supports Slack, Discord, PagerDuty, webhooks, and more — all configured through provisioning files:
|
||||
|
||||
1. Add a new receiver in `grafana/provisioning/alerting/contactpoints.yaml`
|
||||
2. Add the channel to the notification policy in `policies.yaml`, or create a new policy that matches specific alert labels
|
||||
3. Restart Grafana
|
||||
|
||||
## Lessons learned
|
||||
|
||||
- **Provisioning files don't expand env vars.** Unlike Docker Compose, Grafana's provisioning YAML is read as-is — `${VARIABLE}` syntax is treated as literal text. Email addresses and other non-secret values must be hardcoded in the provisioning files or managed through the Grafana UI after first boot.
|
||||
- **Promtail is dead.** The `for` duration and `group_interval` in alerting policies interact with log pipeline latency — if using an older Promtail-based setup, log alerts might fire with stale data. Alloy ships logs faster and more reliably, which makes log-based alerts more responsive. See [Monitoring Stack](Monitoring-Stack.md) for the migration details.
|
||||
- **Alert fatigue is real on small setups.** Three well-tuned rules cover the important failure modes without spamming. Adding too many rules to a single-node homelab generates noise faster than signal — each new rule should answer the question "would I actually get up at 3 AM for this?"
|
||||
@@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
|
||||
|
||||
- [Architecture and Hardware](Architecture-and-Hardware.md)
|
||||
- [Monitoring Stack](Monitoring-Stack.md)
|
||||
- [Alerting](Alerting.md)
|
||||
- [K3s](K3s.md)
|
||||
- [Network Services](Network-Services.md)
|
||||
- [Media Automation](Media-Automation.md)
|
||||
@@ -32,7 +33,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
|
||||
| **Host** | Ubuntu Server |
|
||||
| **Orchestration** | Docker Compose, one stack per service |
|
||||
| **Management** | Portainer, **DockScope** (3D visual dashboard) |
|
||||
| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) |
|
||||
| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) + **Grafana Alerting** (provisioned rules + email notifications) |
|
||||
| **Network** | Pi-hole (DNS filtering), Samba (file sharing), **Traefik** (reverse proxy, config-as-code) |
|
||||
| **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent |
|
||||
| **Disk health** | Hard Disk Sentinel |
|
||||
|
||||
@@ -11,6 +11,19 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi
|
||||
- **Loki** — stores and indexes logs by label, not full text, keeping it lightweight
|
||||
- **Grafana Alloy** — discovers every running container and ships its logs to Loki
|
||||
- **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics
|
||||
- **Grafana Alerting** — unified alerting rules evaluated against Prometheus and Loki, with email notifications via Gmail SMTP
|
||||
|
||||
## Alerting
|
||||
|
||||
Grafana's unified alerting is fully provisioned as code — rules, contact points, and notification policies live in `grafana/provisioning/alerting/` and survive container rebuilds. Three rules cover the critical failure modes:
|
||||
|
||||
| Rule | Source | What it catches | Severity |
|
||||
|---|---|---|---|
|
||||
| Node exporter target down | Prometheus | Host unreachable or exporter crashed | `critical` |
|
||||
| High CPU usage | Prometheus | Sustained CPU above 85% for 5 min | `warning` |
|
||||
| Error log rate spike | Loki | More than 20 error log lines in 5 min | `warning` |
|
||||
|
||||
Alerts route to Gmail over SMTP, grouped by alert name and instance, with a 4-hour repeat interval for ongoing issues. Full details — how to add new rules, common PromQL examples, adding Slack or other channels — in [Alerting](Alerting.md).
|
||||
|
||||
## Prometheus targets
|
||||
|
||||
@@ -65,4 +78,4 @@ Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.sou
|
||||
|
||||
## Why this matters
|
||||
|
||||
On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.
|
||||
On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. Alerting completes the loop: instead of actively watching dashboards, you get an email when something actually needs attention — a host going down, CPU spiking, or error logs flooding.
|
||||
|
||||
Reference in New Issue
Block a user