diff --git a/README.md b/README.md index fcee0c9..46ab78e 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s | Service | Folder | Purpose | |---|---|---| -| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, plus Loki + Grafana Alloy for centralized container log aggregation — see [Monitoring Stack](docs/Monitoring-Stack.md) | +| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, Loki + Grafana Alloy for centralized logs, and unified alerting with email notifications — see [Monitoring Stack](docs/Monitoring-Stack.md) and [Alerting](docs/Alerting.md) | | **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking | | **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host | | **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) | @@ -65,12 +65,20 @@ homelab/ ├── docker-compose/ │ ├── monitoring-grafana-promethues-cadvisor-node-exporter/ │ │ ├── docker-compose.yml +│ │ ├── .env.example │ │ ├── alloy/ │ │ │ └── config.alloy │ │ ├── grafana/ │ │ │ └── provisioning/ +│ │ │ ├── alerting/ +│ │ │ │ ├── contactpoints.yaml +│ │ │ │ ├── policies.yaml +│ │ │ │ └── rules.yaml │ │ │ └── datasources/ -│ │ │ └── loki.yml +│ │ │ ├── loki.yml +│ │ │ └── prometheus.yml +│ │ ├── loki/ +│ │ │ └── local-config.yaml │ │ └── prometheus/ │ │ └── prometheus.yml │ ├── pihole/ @@ -269,12 +277,14 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it. - Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it. - Migrating from a UI-based reverse proxy (Nginx Proxy Manager) to a config-as-code one (Traefik) forced a cleanup of every hardcoded password in every compose file — the migration was the catalyst for adopting Vaultwarden and proper secret management across the stack. See [Traefik](docs/Traefik.md) and [Vaultwarden](docs/Vaultwarden.md). +- Grafana provisioning files don't expand Docker Compose `${VARIABLE}` syntax — email addresses and other non-secret values must be literal strings in the provisioning YAML. Only secrets like API keys and passwords should go through `.env` files and `${...}` references in `docker-compose.yml`. See [Alerting](docs/Alerting.md). +- Three well-tuned alert rules (target-down, high-CPU, error-log-spike) cover the important failure modes on a single-node homelab without generating alert fatigue. Adding more rules than you'd actually act on at 3 AM just creates noise — each rule should answer "would I get up for this?" ## Roadmap - [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md) - [ ] Auto-deploy via Watchtower on image updates -- [ ] Expand monitoring with alerting rules +- [x] ~~Expand monitoring with alerting rules~~ — done via Grafana unified alerting, provisioned as code — see [Alerting](docs/Alerting.md) ## Author diff --git a/dashboards/traefik-dashboards/traefik-access-logs-dashboard.json b/dashboards/traefik-dashboards/traefik-access-logs-dashboard.json new file mode 100644 index 0000000..dbef770 --- /dev/null +++ b/dashboards/traefik-dashboards/traefik-access-logs-dashboard.json @@ -0,0 +1,262 @@ +{ + "annotations": { "list": [] }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "id": null, + "links": [], + "liveNow": true, + "panels": [ + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "id": 100, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "fieldConfig": { + "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }, "unit": "reqps" }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 }, + "id": 1, + "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))", + "legendFormat": "requests/s", + "refId": "A" + } + ], + "title": "Request Rate", + "type": "stat" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "fieldConfig": { + "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 0.01 }] }, "unit": "percentunit" }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 }, + "id": 2, + "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))", + "legendFormat": "5xx rate", + "refId": "A" + } + ], + "title": "5xx Error Rate", + "type": "stat" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "fieldConfig": { + "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "orange", "value": 0.02 }] }, "unit": "percentunit" }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 }, + "id": 3, + "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))", + "legendFormat": "4xx rate", + "refId": "A" + } + ], + "title": "4xx Error Rate", + "type": "stat" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "fieldConfig": { + "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 300 }, { "color": "red", "value": 1000 }] }, "unit": "ms" }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 }, + "id": 4, + "options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "avg(avg_over_time({container_name=\"traefik\"} | json | __error__=\"\" | unwrap Duration [5m])) / 1e6", + "legendFormat": "avg duration", + "refId": "A" + } + ], + "title": "Avg Response Time", + "type": "stat" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, + "id": 101, + "panels": [], + "title": "Traffic Breakdown", + "type": "row" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "description": "Request volume by status code family over time.", + "fieldConfig": { + "defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 25, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": true, "stacking": { "group": "A", "mode": "normal" } }, "mappings": [], "unit": "reqps" }, + "overrides": [ + { "matcher": { "id": "byName", "options": "2xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] }, + { "matcher": { "id": "byName", "options": "3xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }] }, + { "matcher": { "id": "byName", "options": "4xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }] }, + { "matcher": { "id": "byName", "options": "5xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] } + ] + }, + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 6 }, + "id": 5, + "options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } }, + "targets": [ + { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 200 | DownstreamStatus < 300 [5m]))", "legendFormat": "2xx", "refId": "A" }, + { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 300 | DownstreamStatus < 400 [5m]))", "legendFormat": "3xx", "refId": "B" }, + { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m]))", "legendFormat": "4xx", "refId": "C" }, + { "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m]))", "legendFormat": "5xx", "refId": "D" } + ], + "title": "Response Codes Over Time", + "type": "timeseries" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "description": "Which site (router) is generating the most log volume right now.", + "fieldConfig": { + "defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2, "pointSize": 5, "showPoints": "never", "spanNulls": true }, "mappings": [], "unit": "reqps" }, + "overrides": [] + }, + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 6 }, + "id": 6, + "options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "sum by (RouterName) (rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))", + "legendFormat": "{{RouterName}}", + "refId": "A" + } + ], + "title": "Requests by Site (Router)", + "transformations": [{ "id": "renameByRegex", "options": { "regex": "(.*)@docker", "renamePattern": "$1" } }], + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 15 }, + "id": 102, + "panels": [], + "title": "Live Logs", + "type": "row" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "description": "Raw Traefik access log stream, most recent first.", + "fieldConfig": { "defaults": {}, "overrides": [] }, + "gridPos": { "h": 10, "w": 24, "x": 0, "y": 16 }, + "id": 7, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": true, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "{container_name=\"traefik\"}", + "refId": "A" + } + ], + "title": "All Access Logs", + "type": "logs" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 }, + "id": 103, + "panels": [], + "title": "Problems", + "type": "row" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "description": "Only requests that returned 4xx or 5xx.", + "fieldConfig": { "defaults": {}, "overrides": [] }, + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 }, + "id": 8, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": true, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "{container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400", + "refId": "A" + } + ], + "title": "Error Requests (4xx/5xx)", + "type": "logs" + }, + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "description": "Requests that took longer than 500ms to respond.", + "fieldConfig": { "defaults": {}, "overrides": [] }, + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 }, + "id": 9, + "options": { + "dedupStrategy": "none", + "enableLogDetails": true, + "prettifyLogMessage": true, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "targets": [ + { + "datasource": { "type": "loki", "uid": "${loki}" }, + "expr": "{container_name=\"traefik\"} | json | __error__=\"\" | Duration > 500000000", + "refId": "A" + } + ], + "title": "Slow Requests (>500ms)", + "type": "logs" + } + ], + "refresh": "10s", + "schemaVersion": 38, + "tags": ["traefik", "logs", "loki", "homelab"], + "templating": { + "list": [ + { "current": {}, "hide": 0, "label": "Loki", "name": "loki", "query": "loki", "refresh": 1, "regex": "", "type": "datasource" } + ] + }, + "time": { "from": "now-1h", "to": "now" }, + "timepicker": {}, + "timezone": "", + "title": "Traefik - Access Logs", + "uid": "traefik-access-logs", + "version": 1, + "weekStart": "" +} diff --git a/dashboards/traefik-dashboards/traefik-homelab-dashboard.json b/dashboards/traefik-dashboards/traefik-homelab-dashboard.json new file mode 100644 index 0000000..61539b0 --- /dev/null +++ b/dashboards/traefik-dashboards/traefik-homelab-dashboard.json @@ -0,0 +1,502 @@ +{ + "annotations": { + "list": [] + }, + "editable": true, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "id": null, + "links": [], + "liveNow": false, + "panels": [ + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, + "id": 100, + "panels": [], + "title": "Overview", + "type": "row" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "fieldConfig": { + "defaults": { + "color": { "mode": "thresholds" }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [{ "color": "green", "value": null }] + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 }, + "id": 1, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "textMode": "auto" + }, + "pluginVersion": "10.0.0", + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum(rate(traefik_router_requests_total[5m]))", + "legendFormat": "Total request rate", + "refId": "A" + } + ], + "title": "Total Request Rate", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "fieldConfig": { + "defaults": { + "color": { "mode": "thresholds" }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 0.01 }, + { "color": "red", "value": 0.05 } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 }, + "id": 2, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "textMode": "auto" + }, + "pluginVersion": "10.0.0", + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum(rate(traefik_router_requests_total{code=~\"5..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))", + "legendFormat": "5xx error rate", + "refId": "A" + } + ], + "title": "5xx Error Rate", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "fieldConfig": { + "defaults": { + "color": { "mode": "thresholds" }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [ + { "color": "green", "value": null }, + { "color": "yellow", "value": 0.02 }, + { "color": "red", "value": 0.1 } + ] + }, + "unit": "percentunit" + }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 }, + "id": 3, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "textMode": "auto" + }, + "pluginVersion": "10.0.0", + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum(rate(traefik_router_requests_total{code=~\"4..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))", + "legendFormat": "4xx error rate", + "refId": "A" + } + ], + "title": "4xx Error Rate", + "type": "stat" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "fieldConfig": { + "defaults": { + "color": { "mode": "thresholds" }, + "mappings": [], + "thresholds": { + "mode": "absolute", + "steps": [{ "color": "green", "value": null }] + }, + "unit": "none" + }, + "overrides": [] + }, + "gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 }, + "id": 4, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "auto", + "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, + "textMode": "auto" + }, + "pluginVersion": "10.0.0", + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum(traefik_open_connections)", + "legendFormat": "Open connections", + "refId": "A" + } + ], + "title": "Open Connections", + "type": "stat" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 }, + "id": 101, + "panels": [], + "title": "Which Sites Get Hit The Most", + "type": "row" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Total requests per site (router) over the selected time range, highest first.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "fillOpacity": 80, + "gradientMode": "none", + "lineWidth": 1 + }, + "mappings": [], + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { "h": 10, "w": 12, "x": 0, "y": 6 }, + "id": 5, + "options": { + "barRadius": 0, + "barWidth": 0.7, + "fullHighlight": false, + "groupWidth": 0.7, + "legend": { "displayMode": "list", "placement": "bottom", "showLegend": false }, + "orientation": "horizontal", + "showValue": "auto", + "stacking": "none", + "tooltip": { "mode": "single", "sort": "none" }, + "xTickLabelRotation": 0, + "xTickLabelSpacing": 0 + }, + "pluginVersion": "10.0.0", + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "topk(10, sum by (router) (increase(traefik_router_requests_total[$__range])))", + "format": "table", + "instant": true, + "legendFormat": "{{router}}", + "refId": "A" + } + ], + "title": "Top 10 Sites by Total Requests", + "transformations": [ + { + "id": "renameByRegex", + "options": { + "regex": "(.*)@docker", + "renamePattern": "$1" + } + } + ], + "type": "barchart" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Live request rate per site — which sites are being hit right now.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { "h": 10, "w": 12, "x": 12, "y": 6 }, + "id": 6, + "options": { + "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (router) (rate(traefik_router_requests_total[5m]))", + "legendFormat": "{{router}}", + "refId": "A" + } + ], + "title": "Request Rate by Site (live)", + "transformations": [ + { + "id": "renameByRegex", + "options": { + "regex": "(.*)@docker", + "renamePattern": "$1" + } + } + ], + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 16 }, + "id": 102, + "panels": [], + "title": "Health & Errors", + "type": "row" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Requests per site broken down by status code family — spot which site is throwing errors.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 25, + "lineWidth": 1, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true, + "stacking": { "group": "A", "mode": "normal" } + }, + "mappings": [], + "unit": "reqps" + }, + "overrides": [ + { + "matcher": { "id": "byRegexp", "options": "/5\\d\\d/" }, + "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] + }, + { + "matcher": { "id": "byRegexp", "options": "/4\\d\\d/" }, + "properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }] + }, + { + "matcher": { "id": "byRegexp", "options": "/2\\d\\d/" }, + "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] + } + ] + }, + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 17 }, + "id": 7, + "options": { + "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (code) (rate(traefik_router_requests_total[5m]))", + "legendFormat": "{{code}}", + "refId": "A" + } + ], + "title": "Response Codes Over Time", + "type": "timeseries" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Which specific site/router is generating 4xx or 5xx errors right now.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 17 }, + "id": 8, + "options": { + "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (router) (rate(traefik_router_requests_total{code=~\"4..|5..\"}[5m]))", + "legendFormat": "{{router}}", + "refId": "A" + } + ], + "title": "Errors by Site", + "transformations": [ + { + "id": "renameByRegex", + "options": { + "regex": "(.*)@docker", + "renamePattern": "$1" + } + } + ], + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 }, + "id": 103, + "panels": [], + "title": "Performance", + "type": "row" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "How long each site takes to respond (average), per router.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 }, + "id": 9, + "options": { + "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (router) (rate(traefik_router_request_duration_seconds_sum[5m])) / sum by (router) (rate(traefik_router_request_duration_seconds_count[5m]))", + "legendFormat": "{{router}}", + "refId": "A" + } + ], + "title": "Average Response Time by Site", + "transformations": [ + { + "id": "renameByRegex", + "options": { + "regex": "(.*)@docker", + "renamePattern": "$1" + } + } + ], + "type": "timeseries" + }, + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "description": "Traffic split between HTTP (web) and HTTPS (websecure) entrypoints — confirms redirects are working.", + "fieldConfig": { + "defaults": { + "color": { "mode": "palette-classic" }, + "custom": { + "axisPlacement": "auto", + "drawStyle": "line", + "fillOpacity": 15, + "lineWidth": 2, + "pointSize": 5, + "showPoints": "never", + "spanNulls": true + }, + "mappings": [], + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 }, + "id": 10, + "options": { + "legend": { "calcs": ["mean"], "displayMode": "table", "placement": "bottom", "showLegend": true }, + "tooltip": { "mode": "multi", "sort": "desc" } + }, + "targets": [ + { + "datasource": { "type": "prometheus", "uid": "${datasource}" }, + "expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total[5m]))", + "legendFormat": "{{entrypoint}}", + "refId": "A" + } + ], + "title": "Traffic by Entrypoint (web vs websecure)", + "type": "timeseries" + } + ], + "refresh": "30s", + "schemaVersion": 38, + "tags": ["traefik", "homelab"], + "templating": { + "list": [ + { + "current": {}, + "hide": 0, + "label": "Data source", + "name": "datasource", + "query": "prometheus", + "refresh": 1, + "regex": "", + "type": "datasource" + } + ] + }, + "time": { "from": "now-6h", "to": "now" }, + "timepicker": {}, + "timezone": "", + "title": "Traefik - Homelab Overview", + "uid": "traefik-homelab", + "version": 1, + "weekStart": "" +} diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example new file mode 100644 index 0000000..9740cf1 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example @@ -0,0 +1,7 @@ +# Grafana SMTP configuration for alert emails +# Generate a Gmail App Password at: https://myaccount.google.com/apppasswords +GMAIL_SMTP_USER=your-email@gmail.com +GMAIL_APP_PASSWORD=CHANGE_ME + +# Alerting contact point email (must match your Grafana login) +ALERT_EMAIL=your-email@gmail.com diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml index 0e3f014..fbdca0f 100644 --- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml @@ -33,10 +33,20 @@ services: container_name: grafana restart: unless-stopped ports: - - "3000:3000" # keep for now + - "3000:3000" + env_file: + - .env + environment: + - GF_SMTP_ENABLED=true + - GF_SMTP_HOST=smtp.gmail.com:587 + - GF_SMTP_USER=${GMAIL_SMTP_USER} + - GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD} + - GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER} + - GF_SMTP_FROM_NAME=Grafana Alerts volumes: - ./grafana/data:/var/lib/grafana - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro + - ./grafana/provisioning/alerting:/etc/grafana/provisioning/alerting:ro networks: - monitoring - web_net diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml new file mode 100644 index 0000000..18f5742 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml @@ -0,0 +1,10 @@ +apiVersion: 1 +contactPoints: + - orgId: 1 + name: gmail-alerts + receivers: + - uid: gmail-email-receiver + type: email + settings: + addresses: YOUR_EMAIL@example.com + singleEmail: true diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml new file mode 100644 index 0000000..ec58c65 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml @@ -0,0 +1,8 @@ +apiVersion: 1 +policies: + - orgId: 1 + receiver: gmail-alerts + group_by: ['alertname', 'instance'] + group_wait: 30s + group_interval: 5m + repeat_interval: 4h diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml new file mode 100644 index 0000000..3b55f74 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml @@ -0,0 +1,93 @@ +apiVersion: 1 +groups: + - orgId: 1 + name: infra-alerts + folder: Infra + interval: 1m + rules: + - uid: node-down + title: Node exporter target down + condition: C + data: + - refId: A + datasourceUid: efck3im1w4pvkf + relativeTimeRange: + from: 300 + to: 0 + model: + expr: up{job="node_exporter"} + instant: true + - refId: C + datasourceUid: __expr__ + model: + type: threshold + expression: A + conditions: + - evaluator: + type: lt + params: [1] + for: 2m + labels: + severity: critical + annotations: + summary: "Node exporter target is down" + + - uid: high-cpu + title: High CPU usage + condition: C + data: + - refId: A + datasourceUid: efck3im1w4pvkf + relativeTimeRange: + from: 300 + to: 0 + model: + expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) + instant: true + - refId: C + datasourceUid: __expr__ + model: + type: threshold + expression: A + conditions: + - evaluator: + type: gt + params: [85] + for: 5m + labels: + severity: warning + annotations: + summary: "CPU usage above 85% on {{ $labels.instance }}" + + - uid: loki-error-spike + title: Error log rate spike + condition: C + data: + - refId: A + datasourceUid: P8E80F9AEF21F6940 + relativeTimeRange: + from: 300 + to: 0 + model: + expr: 'sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)' + instant: true + - refId: B + datasourceUid: __expr__ + model: + type: reduce + expression: A + reducer: last + - refId: C + datasourceUid: __expr__ + model: + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [20] + for: 5m + labels: + severity: warning + annotations: + summary: "Error log rate is elevated" diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml new file mode 100644 index 0000000..3dbf0df --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml @@ -0,0 +1,9 @@ +apiVersion: 1 +datasources: + - name: Loki + type: loki + uid: P8E80F9AEF21F6940 + access: proxy + url: http://loki:3100 + isDefault: false + editable: false diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 0000000..dca10ab --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,9 @@ +apiVersion: 1 +datasources: + - name: prometheus + type: prometheus + uid: efck3im1w4pvkf + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: false diff --git a/docs/Alerting.md b/docs/Alerting.md new file mode 100644 index 0000000..e0796f0 --- /dev/null +++ b/docs/Alerting.md @@ -0,0 +1,150 @@ +# Alerting + +Grafana's built-in unified alerting ties directly into the Prometheus and Loki datasources already running in the [Monitoring Stack](Monitoring-Stack.md) — no separate Alertmanager container needed. All rules, contact points, and notification policies are provisioned as code in `grafana/provisioning/alerting/`, so they survive container rebuilds and are version-controlled alongside everything else in this repo. + +## Architecture + +``` +Prometheus ──┐ + ├──► Grafana Unified Alerting ──► Gmail (SMTP) +Loki ────────┘ +``` + +Grafana evaluates alert rules against its configured datasources (Prometheus for metrics, Loki for logs). When a rule fires, the notification policy routes it to a contact point, which delivers the notification via the configured channel (email over SMTP in this setup). + +No separate Alertmanager instance is required — Grafana handles the full pipeline internally. + +## Alert rules + +All rules live in `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml` under the **Infra** folder: + +| Rule | UID | Source | Condition | Severity | Duration | +|---|---|---|---|---|---| +| **Node exporter target down** | `node-down` | Prometheus | `up{job="node_exporter"} < 1` | `critical` | 2 min | +| **High CPU usage** | `high-cpu` | Prometheus | `100 - (avg idle rate) > 85%` | `warning` | 5 min | +| **Error log rate spike** | `loki-error-spike` | Loki | `count_over_time(… \|= "error" [5m]) > 20` | `warning` | 5 min | + +### Node exporter target down (`node-down`) + +The most critical alert — if the node exporter scrapes as down for 2 minutes, the host itself may be unreachable or the exporter process crashed. Fires at `critical` severity because it means all host-level visibility is lost. + +**PromQL:** +```promql +up{job="node_exporter"} +``` + +**Threshold:** `< 1` (target is down) + +### High CPU usage (`high-cpu`) + +Watches for sustained CPU usage above 85% across the 5-minute window. The query uses `rate(node_cpu_seconds_total{mode="idle"}[5m])` to smooth out momentary spikes — a brief compile or Docker pull won't trigger it, but a runaway process or resource-hungry container will. + +**PromQL:** +```promql +100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) +``` + +**Threshold:** `> 85%` + +### Error log rate spike (`loki-error-spike`) + +Queries Loki for log lines containing "error" across all jobs. If the count exceeds 20 within a 5-minute window and stays elevated for 5 minutes, it fires. This catches application-level issues that metrics alone won't surface — a service that's technically "up" but flooding its logs with errors. + +**LogQL:** +```logql +sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0) +``` + +**Threshold:** `> 20` error lines in 5 minutes + +## Contact points + +Defined in `grafana/provisioning/alerting/contactpoints.yaml`: + +| Name | Type | Channel | +|---|---|---| +| `gmail-alerts` | Email | SMTP via Gmail App Password | + +The email address is set to a placeholder (`YOUR_EMAIL@example.com`) in the provisioned file — update it to your actual address after the first Grafana container starts, or edit it directly in the file before starting. + +> **Note:** Grafana provisioning files do not expand Docker environment variables (`${...}` syntax). The email address must be set as a literal string in the YAML. + +## Notification policy + +Defined in `grafana/provisioning/alerting/policies.yaml`: + +| Setting | Value | Meaning | +|---|---|---| +| **Receiver** | `gmail-alerts` | All alerts route to the email contact point | +| **Group by** | `alertname`, `instance` | Alerts with the same name and instance are grouped into a single email | +| **Group wait** | 30s | How long to wait before sending the first notification for a new group | +| **Group interval** | 5 min | Minimum time between notifications for the same group | +| **Repeat interval** | 4 hours | How often to re-notify about an ongoing (unresolved) alert | + +This means a firing alert sends its first email within ~30 seconds, groups follow-ups every 5 minutes, and you get a reminder every 4 hours until the alert resolves. + +## SMTP configuration + +Grafana sends alert emails through Gmail's SMTP relay. The configuration lives in the `grafana` service's environment block in `docker-compose.yml` and pulls credentials from the `.env` file: + +```yaml +GF_SMTP_ENABLED=true +GF_SMTP_HOST=smtp.gmail.com:587 +GF_SMTP_USER=${GMAIL_SMTP_USER} +GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD} +GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER} +GF_SMTP_FROM_NAME=Grafana Alerts +``` + +### Setting up Gmail SMTP + +1. Enable 2-Step Verification on your Google account +2. Generate an [App Password](https://myaccount.google.com/apppasswords) (requires 2FA) +3. Add both values to `.env`: + ``` + GMAIL_SMTP_USER=your-email@gmail.com + ALERT_EMAIL=your-email@gmail.com + ``` +4. Set the same email as your Grafana login (`GF_SECURITY_ADMIN_USER` in `.env`, or through the Grafana UI) + +## How to add new alert rules + +Rules are provisioned from YAML — no UI clicks needed: + +1. Open `grafana/provisioning/alerting/rules.yaml` +2. Add a new entry under the `rules` list in the `infra-alerts` group +3. Give it a unique `uid`, a descriptive `title`, and set the `datasourceUid` to match your Prometheus (`efck3im1w4pvkf`) or Loki (`P8E80F9AEF21F6940`) datasource +4. Set the `condition` refId to point to the threshold expression +5. Set `for` to how long the condition must be true before firing +6. Add `labels.severity` (`critical`, `warning`, `info`) and an `annotations.summary` message +7. Restart the Grafana container: `docker compose restart grafana` + +### Common PromQL examples for new rules + +```promql +# Memory usage above 90% +100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) + +# Disk usage above 85% +100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) + +# Container restart count (from cAdvisor) +increase(container_restart_count[5m]) + +# Container memory above limit +container_memory_working_set_bytes / container_spec_memory_limit_bytes +``` + +## How to add new notification channels + +Grafana supports Slack, Discord, PagerDuty, webhooks, and more — all configured through provisioning files: + +1. Add a new receiver in `grafana/provisioning/alerting/contactpoints.yaml` +2. Add the channel to the notification policy in `policies.yaml`, or create a new policy that matches specific alert labels +3. Restart Grafana + +## Lessons learned + +- **Provisioning files don't expand env vars.** Unlike Docker Compose, Grafana's provisioning YAML is read as-is — `${VARIABLE}` syntax is treated as literal text. Email addresses and other non-secret values must be hardcoded in the provisioning files or managed through the Grafana UI after first boot. +- **Promtail is dead.** The `for` duration and `group_interval` in alerting policies interact with log pipeline latency — if using an older Promtail-based setup, log alerts might fire with stale data. Alloy ships logs faster and more reliably, which makes log-based alerts more responsive. See [Monitoring Stack](Monitoring-Stack.md) for the migration details. +- **Alert fatigue is real on small setups.** Three well-tuned rules cover the important failure modes without spamming. Adding too many rules to a single-node homelab generates noise faster than signal — each new rule should answer the question "would I actually get up at 3 AM for this?" diff --git a/docs/Home.md b/docs/Home.md index 3eb5285..61a7198 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This - [Architecture and Hardware](Architecture-and-Hardware.md) - [Monitoring Stack](Monitoring-Stack.md) +- [Alerting](Alerting.md) - [K3s](K3s.md) - [Network Services](Network-Services.md) - [Media Automation](Media-Automation.md) @@ -32,7 +33,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This | **Host** | Ubuntu Server | | **Orchestration** | Docker Compose, one stack per service | | **Management** | Portainer, **DockScope** (3D visual dashboard) | -| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) | +| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) + **Grafana Alerting** (provisioned rules + email notifications) | | **Network** | Pi-hole (DNS filtering), Samba (file sharing), **Traefik** (reverse proxy, config-as-code) | | **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent | | **Disk health** | Hard Disk Sentinel | diff --git a/docs/Monitoring-Stack.md b/docs/Monitoring-Stack.md index f612357..d4cbba0 100644 --- a/docs/Monitoring-Stack.md +++ b/docs/Monitoring-Stack.md @@ -11,6 +11,19 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight - **Grafana Alloy** — discovers every running container and ships its logs to Loki - **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics +- **Grafana Alerting** — unified alerting rules evaluated against Prometheus and Loki, with email notifications via Gmail SMTP + +## Alerting + +Grafana's unified alerting is fully provisioned as code — rules, contact points, and notification policies live in `grafana/provisioning/alerting/` and survive container rebuilds. Three rules cover the critical failure modes: + +| Rule | Source | What it catches | Severity | +|---|---|---|---| +| Node exporter target down | Prometheus | Host unreachable or exporter crashed | `critical` | +| High CPU usage | Prometheus | Sustained CPU above 85% for 5 min | `warning` | +| Error log rate spike | Loki | More than 20 error log lines in 5 min | `warning` | + +Alerts route to Gmail over SMTP, grouped by alert name and instance, with a 4-hour repeat interval for ongoing issues. Full details — how to add new rules, common PromQL examples, adding Slack or other channels — in [Alerting](Alerting.md). ## Prometheus targets @@ -65,4 +78,4 @@ Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.sou ## Why this matters -On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. +On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. Alerting completes the loop: instead of actively watching dashboards, you get an email when something actually needs attention — a host going down, CPU spiking, or error logs flooding.