added alerting rules for grafana

2026-09-01 14:01:26 +03:30
parent e02f7b4bce
commit 45c7d53b6a
13 changed files with 1090 additions and 6 deletions
--- a/README.md
+++ b/README.md
@@ -24,7 +24,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s

 | Service | Folder | Purpose |
 |---|---|---|
-| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, plus Loki + Grafana Alloy for centralized container log aggregation — see [Monitoring Stack](docs/Monitoring-Stack.md) |
+| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, Loki + Grafana Alloy for centralized logs, and unified alerting with email notifications — see [Monitoring Stack](docs/Monitoring-Stack.md) and [Alerting](docs/Alerting.md) |
 | **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking |
 | **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host |
 | **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) |
@@ -65,12 +65,20 @@ homelab/
 ├── docker-compose/
 │   ├── monitoring-grafana-promethues-cadvisor-node-exporter/
 │   │   ├── docker-compose.yml
+│   │   ├── .env.example
 │   │   ├── alloy/
 │   │   │   └── config.alloy
 │   │   ├── grafana/
 │   │   │   └── provisioning/
+│   │   │       ├── alerting/
+│   │   │       │   ├── contactpoints.yaml
+│   │   │       │   ├── policies.yaml
+│   │   │       │   └── rules.yaml
 │   │   │       └── datasources/
-│   │   │           └── loki.yml
+│   │   │           ├── loki.yml
+│   │   │           └── prometheus.yml
+│   │   ├── loki/
+│   │   │   └── local-config.yaml
 │   │   └── prometheus/
 │   │       └── prometheus.yml
 │   ├── pihole/
@@ -269,12 +277,14 @@ Rather than pulling and recreating manually, every always-on stack has a matchin
 - A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it.
 - Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it.
 - Migrating from a UI-based reverse proxy (Nginx Proxy Manager) to a config-as-code one (Traefik) forced a cleanup of every hardcoded password in every compose file — the migration was the catalyst for adopting Vaultwarden and proper secret management across the stack. See [Traefik](docs/Traefik.md) and [Vaultwarden](docs/Vaultwarden.md).
+- Grafana provisioning files don't expand Docker Compose `${VARIABLE}` syntax — email addresses and other non-secret values must be literal strings in the provisioning YAML. Only secrets like API keys and passwords should go through `.env` files and `${...}` references in `docker-compose.yml`. See [Alerting](docs/Alerting.md).
+- Three well-tuned alert rules (target-down, high-CPU, error-log-spike) cover the important failure modes on a single-node homelab without generating alert fatigue. Adding more rules than you'd actually act on at 3 AM just creates noise — each rule should answer "would I get up for this?"

 ## Roadmap

 - [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md)
 - [ ] Auto-deploy via Watchtower on image updates
- [ ] Expand monitoring with alerting rules
+- [x] ~~Expand monitoring with alerting rules~~ — done via Grafana unified alerting, provisioned as code — see [Alerting](docs/Alerting.md)

 ## Author

--- a/dashboards/traefik-dashboards/traefik-access-logs-dashboard.json
+++ b/dashboards/traefik-dashboards/traefik-access-logs-dashboard.json
@@ -0,0 +1,262 @@
+{
+  "annotations": { "list": [] },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": true,
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "id": 100,
+      "panels": [],
+      "title": "Overview",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }, "unit": "reqps" },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 },
+      "id": 1,
+      "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
+          "legendFormat": "requests/s",
+          "refId": "A"
+        }
+      ],
+      "title": "Request Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 0.01 }] }, "unit": "percentunit" },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 },
+      "id": 2,
+      "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
+          "legendFormat": "5xx rate",
+          "refId": "A"
+        }
+      ],
+      "title": "5xx Error Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "orange", "value": 0.02 }] }, "unit": "percentunit" },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 },
+      "id": 3,
+      "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
+          "legendFormat": "4xx rate",
+          "refId": "A"
+        }
+      ],
+      "title": "4xx Error Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 300 }, { "color": "red", "value": 1000 }] }, "unit": "ms" },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 },
+      "id": 4,
+      "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "avg(avg_over_time({container_name=\"traefik\"} | json | __error__=\"\" | unwrap Duration [5m])) / 1e6",
+          "legendFormat": "avg duration",
+          "refId": "A"
+        }
+      ],
+      "title": "Avg Response Time",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
+      "id": 101,
+      "panels": [],
+      "title": "Traffic Breakdown",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "description": "Request volume by status code family over time.",
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 25, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": true, "stacking": { "group": "A", "mode": "normal" } }, "mappings": [], "unit": "reqps" },
+        "overrides": [
+          { "matcher": { "id": "byName", "options": "2xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] },
+          { "matcher": { "id": "byName", "options": "3xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }] },
+          { "matcher": { "id": "byName", "options": "4xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }] },
+          { "matcher": { "id": "byName", "options": "5xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] }
+        ]
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 6 },
+      "id": 5,
+      "options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } },
+      "targets": [
+        { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 200 | DownstreamStatus < 300 [5m]))", "legendFormat": "2xx", "refId": "A" },
+        { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 300 | DownstreamStatus < 400 [5m]))", "legendFormat": "3xx", "refId": "B" },
+        { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m]))", "legendFormat": "4xx", "refId": "C" },
+        { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m]))", "legendFormat": "5xx", "refId": "D" }
+      ],
+      "title": "Response Codes Over Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "description": "Which site (router) is generating the most log volume right now.",
+      "fieldConfig": {
+        "defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2, "pointSize": 5, "showPoints": "never", "spanNulls": true }, "mappings": [], "unit": "reqps" },
+        "overrides": []
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 6 },
+      "id": 6,
+      "options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "sum by (RouterName) (rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
+          "legendFormat": "{{RouterName}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Requests by Site (Router)",
+      "transformations": [{ "id": "renameByRegex", "options": { "regex": "(.*)@docker", "renamePattern": "$1" } }],
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 15 },
+      "id": 102,
+      "panels": [],
+      "title": "Live Logs",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "description": "Raw Traefik access log stream, most recent first.",
+      "fieldConfig": { "defaults": {}, "overrides": [] },
+      "gridPos": { "h": 10, "w": 24, "x": 0, "y": 16 },
+      "id": 7,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": true,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "{container_name=\"traefik\"}",
+          "refId": "A"
+        }
+      ],
+      "title": "All Access Logs",
+      "type": "logs"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 },
+      "id": 103,
+      "panels": [],
+      "title": "Problems",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "description": "Only requests that returned 4xx or 5xx.",
+      "fieldConfig": { "defaults": {}, "overrides": [] },
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 },
+      "id": 8,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": true,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "{container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400",
+          "refId": "A"
+        }
+      ],
+      "title": "Error Requests (4xx/5xx)",
+      "type": "logs"
+    },
+    {
+      "datasource": { "type": "loki", "uid": "${loki}" },
+      "description": "Requests that took longer than 500ms to respond.",
+      "fieldConfig": { "defaults": {}, "overrides": [] },
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 },
+      "id": 9,
+      "options": {
+        "dedupStrategy": "none",
+        "enableLogDetails": true,
+        "prettifyLogMessage": true,
+        "showCommonLabels": false,
+        "showLabels": false,
+        "showTime": true,
+        "sortOrder": "Descending",
+        "wrapLogMessage": true
+      },
+      "targets": [
+        {
+          "datasource": { "type": "loki", "uid": "${loki}" },
+          "expr": "{container_name=\"traefik\"} | json | __error__=\"\" | Duration > 500000000",
+          "refId": "A"
+        }
+      ],
+      "title": "Slow Requests (>500ms)",
+      "type": "logs"
+    }
+  ],
+  "refresh": "10s",
+  "schemaVersion": 38,
+  "tags": ["traefik", "logs", "loki", "homelab"],
+  "templating": {
+    "list": [
+      { "current": {}, "hide": 0, "label": "Loki", "name": "loki", "query": "loki", "refresh": 1, "regex": "", "type": "datasource" }
+    ]
+  },
+  "time": { "from": "now-1h", "to": "now" },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Traefik - Access Logs",
+  "uid": "traefik-access-logs",
+  "version": 1,
+  "weekStart": ""
+}
--- a/dashboards/traefik-dashboards/traefik-homelab-dashboard.json
+++ b/dashboards/traefik-dashboards/traefik-homelab-dashboard.json
@@ -0,0 +1,502 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+      "id": 100,
+      "panels": [],
+      "title": "Overview",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [{ "color": "green", "value": null }]
+          },
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 },
+      "id": 1,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+        "textMode": "auto"
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum(rate(traefik_router_requests_total[5m]))",
+          "legendFormat": "Total request rate",
+          "refId": "A"
+        }
+      ],
+      "title": "Total Request Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 0.01 },
+              { "color": "red", "value": 0.05 }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 },
+      "id": 2,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+        "textMode": "auto"
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum(rate(traefik_router_requests_total{code=~\"5..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))",
+          "legendFormat": "5xx error rate",
+          "refId": "A"
+        }
+      ],
+      "title": "5xx Error Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              { "color": "green", "value": null },
+              { "color": "yellow", "value": 0.02 },
+              { "color": "red", "value": 0.1 }
+            ]
+          },
+          "unit": "percentunit"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 },
+      "id": 3,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+        "textMode": "auto"
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum(rate(traefik_router_requests_total{code=~\"4..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))",
+          "legendFormat": "4xx error rate",
+          "refId": "A"
+        }
+      ],
+      "title": "4xx Error Rate",
+      "type": "stat"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "thresholds" },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [{ "color": "green", "value": null }]
+          },
+          "unit": "none"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 },
+      "id": 4,
+      "options": {
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "auto",
+        "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
+        "textMode": "auto"
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum(traefik_open_connections)",
+          "legendFormat": "Open connections",
+          "refId": "A"
+        }
+      ],
+      "title": "Open Connections",
+      "type": "stat"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
+      "id": 101,
+      "panels": [],
+      "title": "Which Sites Get Hit The Most",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "Total requests per site (router) over the selected time range, highest first.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "fillOpacity": 80,
+            "gradientMode": "none",
+            "lineWidth": 1
+          },
+          "mappings": [],
+          "unit": "short"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 10, "w": 12, "x": 0, "y": 6 },
+      "id": 5,
+      "options": {
+        "barRadius": 0,
+        "barWidth": 0.7,
+        "fullHighlight": false,
+        "groupWidth": 0.7,
+        "legend": { "displayMode": "list", "placement": "bottom", "showLegend": false },
+        "orientation": "horizontal",
+        "showValue": "auto",
+        "stacking": "none",
+        "tooltip": { "mode": "single", "sort": "none" },
+        "xTickLabelRotation": 0,
+        "xTickLabelSpacing": 0
+      },
+      "pluginVersion": "10.0.0",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "topk(10, sum by (router) (increase(traefik_router_requests_total[$__range])))",
+          "format": "table",
+          "instant": true,
+          "legendFormat": "{{router}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Top 10 Sites by Total Requests",
+      "transformations": [
+        {
+          "id": "renameByRegex",
+          "options": {
+            "regex": "(.*)@docker",
+            "renamePattern": "$1"
+          }
+        }
+      ],
+      "type": "barchart"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "Live request rate per site — which sites are being hit right now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 2,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 10, "w": 12, "x": 12, "y": 6 },
+      "id": 6,
+      "options": {
+        "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum by (router) (rate(traefik_router_requests_total[5m]))",
+          "legendFormat": "{{router}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Request Rate by Site (live)",
+      "transformations": [
+        {
+          "id": "renameByRegex",
+          "options": {
+            "regex": "(.*)@docker",
+            "renamePattern": "$1"
+          }
+        }
+      ],
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 16 },
+      "id": 102,
+      "panels": [],
+      "title": "Health & Errors",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "Requests per site broken down by status code family — spot which site is throwing errors.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "drawStyle": "line",
+            "fillOpacity": 25,
+            "lineWidth": 1,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": true,
+            "stacking": { "group": "A", "mode": "normal" }
+          },
+          "mappings": [],
+          "unit": "reqps"
+        },
+        "overrides": [
+          {
+            "matcher": { "id": "byRegexp", "options": "/5\\d\\d/" },
+            "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }]
+          },
+          {
+            "matcher": { "id": "byRegexp", "options": "/4\\d\\d/" },
+            "properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }]
+          },
+          {
+            "matcher": { "id": "byRegexp", "options": "/2\\d\\d/" },
+            "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }]
+          }
+        ]
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 17 },
+      "id": 7,
+      "options": {
+        "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum by (code) (rate(traefik_router_requests_total[5m]))",
+          "legendFormat": "{{code}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Response Codes Over Time",
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "Which specific site/router is generating 4xx or 5xx errors right now.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 2,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 17 },
+      "id": 8,
+      "options": {
+        "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum by (router) (rate(traefik_router_requests_total{code=~\"4..|5..\"}[5m]))",
+          "legendFormat": "{{router}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Errors by Site",
+      "transformations": [
+        {
+          "id": "renameByRegex",
+          "options": {
+            "regex": "(.*)@docker",
+            "renamePattern": "$1"
+          }
+        }
+      ],
+      "type": "timeseries"
+    },
+    {
+      "collapsed": false,
+      "gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 },
+      "id": 103,
+      "panels": [],
+      "title": "Performance",
+      "type": "row"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "How long each site takes to respond (average), per router.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "drawStyle": "line",
+            "fillOpacity": 10,
+            "lineWidth": 2,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 },
+      "id": 9,
+      "options": {
+        "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum by (router) (rate(traefik_router_request_duration_seconds_sum[5m])) / sum by (router) (rate(traefik_router_request_duration_seconds_count[5m]))",
+          "legendFormat": "{{router}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Average Response Time by Site",
+      "transformations": [
+        {
+          "id": "renameByRegex",
+          "options": {
+            "regex": "(.*)@docker",
+            "renamePattern": "$1"
+          }
+        }
+      ],
+      "type": "timeseries"
+    },
+    {
+      "datasource": { "type": "prometheus", "uid": "${datasource}" },
+      "description": "Traffic split between HTTP (web) and HTTPS (websecure) entrypoints — confirms redirects are working.",
+      "fieldConfig": {
+        "defaults": {
+          "color": { "mode": "palette-classic" },
+          "custom": {
+            "axisPlacement": "auto",
+            "drawStyle": "line",
+            "fillOpacity": 15,
+            "lineWidth": 2,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": true
+          },
+          "mappings": [],
+          "unit": "reqps"
+        },
+        "overrides": []
+      },
+      "gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 },
+      "id": 10,
+      "options": {
+        "legend": { "calcs": ["mean"], "displayMode": "table", "placement": "bottom", "showLegend": true },
+        "tooltip": { "mode": "multi", "sort": "desc" }
+      },
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "${datasource}" },
+          "expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total[5m]))",
+          "legendFormat": "{{entrypoint}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Traffic by Entrypoint (web vs websecure)",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "tags": ["traefik", "homelab"],
+  "templating": {
+    "list": [
+      {
+        "current": {},
+        "hide": 0,
+        "label": "Data source",
+        "name": "datasource",
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": { "from": "now-6h", "to": "now" },
+  "timepicker": {},
+  "timezone": "",
+  "title": "Traefik - Homelab Overview",
+  "uid": "traefik-homelab",
+  "version": 1,
+  "weekStart": ""
+}
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example
@@ -0,0 +1,7 @@
+# Grafana SMTP configuration for alert emails
+# Generate a Gmail App Password at: https://myaccount.google.com/apppasswords
+GMAIL_SMTP_USER=your-email@gmail.com
+GMAIL_APP_PASSWORD=CHANGE_ME
+
+# Alerting contact point email (must match your Grafana login)
+ALERT_EMAIL=your-email@gmail.com
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml
@@ -33,10 +33,20 @@ services:
    container_name: grafana
    restart: unless-stopped
    ports:
-      - "3000:3000"                    # keep for now
+      - "3000:3000"
+    env_file:
+      - .env
+    environment:
+      - GF_SMTP_ENABLED=true
+      - GF_SMTP_HOST=smtp.gmail.com:587
+      - GF_SMTP_USER=${GMAIL_SMTP_USER}
+      - GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD}
+      - GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER}
+      - GF_SMTP_FROM_NAME=Grafana Alerts
    volumes:
      - ./grafana/data:/var/lib/grafana
      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
+      - ./grafana/provisioning/alerting:/etc/grafana/provisioning/alerting:ro
    networks:
      - monitoring
      - web_net
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml
@@ -0,0 +1,10 @@
+apiVersion: 1
+contactPoints:
+  - orgId: 1
+    name: gmail-alerts
+    receivers:
+      - uid: gmail-email-receiver
+        type: email
+        settings:
+          addresses: YOUR_EMAIL@example.com
+          singleEmail: true
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml
@@ -0,0 +1,8 @@
+apiVersion: 1
+policies:
+  - orgId: 1
+    receiver: gmail-alerts
+    group_by: ['alertname', 'instance']
+    group_wait: 30s
+    group_interval: 5m
+    repeat_interval: 4h
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml
@@ -0,0 +1,93 @@
+apiVersion: 1
+groups:
+  - orgId: 1
+    name: infra-alerts
+    folder: Infra
+    interval: 1m
+    rules:
+      - uid: node-down
+        title: Node exporter target down
+        condition: C
+        data:
+          - refId: A
+            datasourceUid: efck3im1w4pvkf
+            relativeTimeRange:
+              from: 300
+              to: 0
+            model:
+              expr: up{job="node_exporter"}
+              instant: true
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              type: threshold
+              expression: A
+              conditions:
+                - evaluator:
+                    type: lt
+                    params: [1]
+        for: 2m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Node exporter target is down"
+
+      - uid: high-cpu
+        title: High CPU usage
+        condition: C
+        data:
+          - refId: A
+            datasourceUid: efck3im1w4pvkf
+            relativeTimeRange:
+              from: 300
+              to: 0
+            model:
+              expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+              instant: true
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              type: threshold
+              expression: A
+              conditions:
+                - evaluator:
+                    type: gt
+                    params: [85]
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "CPU usage above 85% on {{ $labels.instance }}"
+
+      - uid: loki-error-spike
+        title: Error log rate spike
+        condition: C
+        data:
+          - refId: A
+            datasourceUid: P8E80F9AEF21F6940
+            relativeTimeRange:
+              from: 300
+              to: 0
+            model:
+              expr: 'sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)'
+              instant: true
+          - refId: B
+            datasourceUid: __expr__
+            model:
+              type: reduce
+              expression: A
+              reducer: last
+          - refId: C
+            datasourceUid: __expr__
+            model:
+              type: threshold
+              expression: B
+              conditions:
+                - evaluator:
+                    type: gt
+                    params: [20]
+        for: 5m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Error log rate is elevated"
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml
@@ -0,0 +1,9 @@
+apiVersion: 1
+datasources:
+  - name: Loki
+    type: loki
+    uid: P8E80F9AEF21F6940
+    access: proxy
+    url: http://loki:3100
+    isDefault: false
+    editable: false
--- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml
+++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml
@@ -0,0 +1,9 @@
+apiVersion: 1
+datasources:
+  - name: prometheus
+    type: prometheus
+    uid: efck3im1w4pvkf
+    access: proxy
+    url: http://prometheus:9090
+    isDefault: true
+    editable: false
--- a/docs/Alerting.md
+++ b/docs/Alerting.md
@@ -0,0 +1,150 @@
+# Alerting
+
+Grafana's built-in unified alerting ties directly into the Prometheus and Loki datasources already running in the [Monitoring Stack](Monitoring-Stack.md) — no separate Alertmanager container needed. All rules, contact points, and notification policies are provisioned as code in `grafana/provisioning/alerting/`, so they survive container rebuilds and are version-controlled alongside everything else in this repo.
+
+## Architecture
+
+```
+Prometheus ──┐
+             ├──► Grafana Unified Alerting ──► Gmail (SMTP)
+Loki ────────┘
+```
+
+Grafana evaluates alert rules against its configured datasources (Prometheus for metrics, Loki for logs). When a rule fires, the notification policy routes it to a contact point, which delivers the notification via the configured channel (email over SMTP in this setup).
+
+No separate Alertmanager instance is required — Grafana handles the full pipeline internally.
+
+## Alert rules
+
+All rules live in `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml` under the **Infra** folder:
+
+| Rule | UID | Source | Condition | Severity | Duration |
+|---|---|---|---|---|---|
+| **Node exporter target down** | `node-down` | Prometheus | `up{job="node_exporter"} < 1` | `critical` | 2 min |
+| **High CPU usage** | `high-cpu` | Prometheus | `100 - (avg idle rate) > 85%` | `warning` | 5 min |
+| **Error log rate spike** | `loki-error-spike` | Loki | `count_over_time(… \|= "error" [5m]) > 20` | `warning` | 5 min |
+
+### Node exporter target down (`node-down`)
+
+The most critical alert — if the node exporter scrapes as down for 2 minutes, the host itself may be unreachable or the exporter process crashed. Fires at `critical` severity because it means all host-level visibility is lost.
+
+**PromQL:**
+```promql
+up{job="node_exporter"}
+```
+
+**Threshold:** `< 1` (target is down)
+
+### High CPU usage (`high-cpu`)
+
+Watches for sustained CPU usage above 85% across the 5-minute window. The query uses `rate(node_cpu_seconds_total{mode="idle"}[5m])` to smooth out momentary spikes — a brief compile or Docker pull won't trigger it, but a runaway process or resource-hungry container will.
+
+**PromQL:**
+```promql
+100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+**Threshold:** `> 85%`
+
+### Error log rate spike (`loki-error-spike`)
+
+Queries Loki for log lines containing "error" across all jobs. If the count exceeds 20 within a 5-minute window and stays elevated for 5 minutes, it fires. This catches application-level issues that metrics alone won't surface — a service that's technically "up" but flooding its logs with errors.
+
+**LogQL:**
+```logql
+sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)
+```
+
+**Threshold:** `> 20` error lines in 5 minutes
+
+## Contact points
+
+Defined in `grafana/provisioning/alerting/contactpoints.yaml`:
+
+| Name | Type | Channel |
+|---|---|---|
+| `gmail-alerts` | Email | SMTP via Gmail App Password |
+
+The email address is set to a placeholder (`YOUR_EMAIL@example.com`) in the provisioned file — update it to your actual address after the first Grafana container starts, or edit it directly in the file before starting.
+
+> **Note:** Grafana provisioning files do not expand Docker environment variables (`${...}` syntax). The email address must be set as a literal string in the YAML.
+
+## Notification policy
+
+Defined in `grafana/provisioning/alerting/policies.yaml`:
+
+| Setting | Value | Meaning |
+|---|---|---|
+| **Receiver** | `gmail-alerts` | All alerts route to the email contact point |
+| **Group by** | `alertname`, `instance` | Alerts with the same name and instance are grouped into a single email |
+| **Group wait** | 30s | How long to wait before sending the first notification for a new group |
+| **Group interval** | 5 min | Minimum time between notifications for the same group |
+| **Repeat interval** | 4 hours | How often to re-notify about an ongoing (unresolved) alert |
+
+This means a firing alert sends its first email within ~30 seconds, groups follow-ups every 5 minutes, and you get a reminder every 4 hours until the alert resolves.
+
+## SMTP configuration
+
+Grafana sends alert emails through Gmail's SMTP relay. The configuration lives in the `grafana` service's environment block in `docker-compose.yml` and pulls credentials from the `.env` file:
+
+```yaml
+GF_SMTP_ENABLED=true
+GF_SMTP_HOST=smtp.gmail.com:587
+GF_SMTP_USER=${GMAIL_SMTP_USER}
+GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD}
+GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER}
+GF_SMTP_FROM_NAME=Grafana Alerts
+```
+
+### Setting up Gmail SMTP
+
+1. Enable 2-Step Verification on your Google account
+2. Generate an [App Password](https://myaccount.google.com/apppasswords) (requires 2FA)
+3. Add both values to `.env`:
+   ```
+   GMAIL_SMTP_USER=your-email@gmail.com
+   ALERT_EMAIL=your-email@gmail.com
+   ```
+4. Set the same email as your Grafana login (`GF_SECURITY_ADMIN_USER` in `.env`, or through the Grafana UI)
+
+## How to add new alert rules
+
+Rules are provisioned from YAML — no UI clicks needed:
+
+1. Open `grafana/provisioning/alerting/rules.yaml`
+2. Add a new entry under the `rules` list in the `infra-alerts` group
+3. Give it a unique `uid`, a descriptive `title`, and set the `datasourceUid` to match your Prometheus (`efck3im1w4pvkf`) or Loki (`P8E80F9AEF21F6940`) datasource
+4. Set the `condition` refId to point to the threshold expression
+5. Set `for` to how long the condition must be true before firing
+6. Add `labels.severity` (`critical`, `warning`, `info`) and an `annotations.summary` message
+7. Restart the Grafana container: `docker compose restart grafana`
+
+### Common PromQL examples for new rules
+
+```promql
+# Memory usage above 90%
+100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100)
+
+# Disk usage above 85%
+100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)
+
+# Container restart count (from cAdvisor)
+increase(container_restart_count[5m])
+
+# Container memory above limit
+container_memory_working_set_bytes / container_spec_memory_limit_bytes
+```
+
+## How to add new notification channels
+
+Grafana supports Slack, Discord, PagerDuty, webhooks, and more — all configured through provisioning files:
+
+1. Add a new receiver in `grafana/provisioning/alerting/contactpoints.yaml`
+2. Add the channel to the notification policy in `policies.yaml`, or create a new policy that matches specific alert labels
+3. Restart Grafana
+
+## Lessons learned
+
+- **Provisioning files don't expand env vars.** Unlike Docker Compose, Grafana's provisioning YAML is read as-is — `${VARIABLE}` syntax is treated as literal text. Email addresses and other non-secret values must be hardcoded in the provisioning files or managed through the Grafana UI after first boot.
+- **Promtail is dead.** The `for` duration and `group_interval` in alerting policies interact with log pipeline latency — if using an older Promtail-based setup, log alerts might fire with stale data. Alloy ships logs faster and more reliably, which makes log-based alerts more responsive. See [Monitoring Stack](Monitoring-Stack.md) for the migration details.
+- **Alert fatigue is real on small setups.** Three well-tuned rules cover the important failure modes without spamming. Adding too many rules to a single-node homelab generates noise faster than signal — each new rule should answer the question "would I actually get up at 3 AM for this?"
--- a/docs/Home.md
+++ b/docs/Home.md
@@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This

 - [Architecture and Hardware](Architecture-and-Hardware.md)
 - [Monitoring Stack](Monitoring-Stack.md)
+- [Alerting](Alerting.md)
 - [K3s](K3s.md)
 - [Network Services](Network-Services.md)
 - [Media Automation](Media-Automation.md)
@@ -32,7 +33,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 | **Host** | Ubuntu Server |
 | **Orchestration** | Docker Compose, one stack per service |
 | **Management** | Portainer, **DockScope** (3D visual dashboard) |
-| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) |
+| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) + **Grafana Alerting** (provisioned rules + email notifications) |
 | **Network** | Pi-hole (DNS filtering), Samba (file sharing), **Traefik** (reverse proxy, config-as-code) |
 | **Automation**     | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent |
 | **Disk health** | Hard Disk Sentinel |
--- a/docs/Monitoring-Stack.md
+++ b/docs/Monitoring-Stack.md
@@ -11,6 +11,19 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi
 - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight
 - **Grafana Alloy** — discovers every running container and ships its logs to Loki
 - **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics
+- **Grafana Alerting** — unified alerting rules evaluated against Prometheus and Loki, with email notifications via Gmail SMTP
+
+## Alerting
+
+Grafana's unified alerting is fully provisioned as code — rules, contact points, and notification policies live in `grafana/provisioning/alerting/` and survive container rebuilds. Three rules cover the critical failure modes:
+
+| Rule | Source | What it catches | Severity |
+|---|---|---|---|
+| Node exporter target down | Prometheus | Host unreachable or exporter crashed | `critical` |
+| High CPU usage | Prometheus | Sustained CPU above 85% for 5 min | `warning` |
+| Error log rate spike | Loki | More than 20 error log lines in 5 min | `warning` |
+
+Alerts route to Gmail over SMTP, grouped by alert name and instance, with a 4-hour repeat interval for ongoing issues. Full details — how to add new rules, common PromQL examples, adding Slack or other channels — in [Alerting](Alerting.md).

 ## Prometheus targets

@@ -65,4 +78,4 @@ Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.sou

 ## Why this matters

-On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.
+On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. Alerting completes the loop: instead of actively watching dashboards, you get an email when something actually needs attention — a host going down, CPU spiking, or error logs flooding.