rezayaghobi/homelab
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added alerting rules for grafana

Browse Source
This commit is contained in:
rezayaghobi-dev
2026-09-01 14:01:26 +03:30
parent e02f7b4bce
commit 45c7d53b6a
13 changed files with 1090 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
README.md
View File

@@ -24,7 +24,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s
| Service | Folder | Purpose | | Service | Folder | Purpose |
|---|---|---| |---|---|---|
| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, plus Loki + Grafana Alloy for centralized container log aggregation — see [Monitoring Stack](docs/Monitoring-Stack.md) | | **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, Loki + Grafana Alloy for centralized logs, and unified alerting with email notifications — see [Monitoring Stack](docs/Monitoring-Stack.md) and [Alerting](docs/Alerting.md) |
| **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking | | **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking |
| **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host | | **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host |
| **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) | | **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) |
@@ -65,12 +65,20 @@ homelab/
├── docker-compose/ ├── docker-compose/
│ ├── monitoring-grafana-promethues-cadvisor-node-exporter/ │ ├── monitoring-grafana-promethues-cadvisor-node-exporter/
│ │ ├── docker-compose.yml │ │ ├── docker-compose.yml
│ │ ├── .env.example
│ │ ├── alloy/ │ │ ├── alloy/
│ │ │ └── config.alloy │ │ │ └── config.alloy
│ │ ├── grafana/ │ │ ├── grafana/
│ │ │ └── provisioning/ │ │ │ └── provisioning/
│ │ │ ├── alerting/
│ │ │ │ ├── contactpoints.yaml
│ │ │ │ ├── policies.yaml
│ │ │ │ └── rules.yaml
│ │ │ └── datasources/ │ │ │ └── datasources/
│ │ │ └── loki.yml │ │ │ ├── loki.yml
│ │ │ └── prometheus.yml
│ │ ├── loki/
│ │ │ └── local-config.yaml
│ │ └── prometheus/ │ │ └── prometheus/
│ │ └── prometheus.yml │ │ └── prometheus.yml
│ ├── pihole/ │ ├── pihole/
@@ -269,12 +277,14 @@ Rather than pulling and recreating manually, every always-on stack has a matchin
- A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it. - A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it.
- Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it. - Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it.
- Migrating from a UI-based reverse proxy (Nginx Proxy Manager) to a config-as-code one (Traefik) forced a cleanup of every hardcoded password in every compose file — the migration was the catalyst for adopting Vaultwarden and proper secret management across the stack. See [Traefik](docs/Traefik.md) and [Vaultwarden](docs/Vaultwarden.md). - Migrating from a UI-based reverse proxy (Nginx Proxy Manager) to a config-as-code one (Traefik) forced a cleanup of every hardcoded password in every compose file — the migration was the catalyst for adopting Vaultwarden and proper secret management across the stack. See [Traefik](docs/Traefik.md) and [Vaultwarden](docs/Vaultwarden.md).
- Grafana provisioning files don't expand Docker Compose `${VARIABLE}` syntax — email addresses and other non-secret values must be literal strings in the provisioning YAML. Only secrets like API keys and passwords should go through `.env` files and `${...}` references in `docker-compose.yml`. See [Alerting](docs/Alerting.md).
- Three well-tuned alert rules (target-down, high-CPU, error-log-spike) cover the important failure modes on a single-node homelab without generating alert fatigue. Adding more rules than you'd actually act on at 3 AM just creates noise — each rule should answer "would I get up for this?"
## Roadmap ## Roadmap
- [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md) - [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md)
- [ ] Auto-deploy via Watchtower on image updates - [ ] Auto-deploy via Watchtower on image updates
- [ ] Expand monitoring with alerting rules - [x] ~~Expand monitoring with alerting rules~~ — done via Grafana unified alerting, provisioned as code — see [Alerting](docs/Alerting.md)
## Author ## Author

262
dashboards/traefik-dashboards/traefik-access-logs-dashboard.json Normal file
View File

@@ -0,0 +1,262 @@
{
"annotations": { "list": [] },
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": true,
"panels": [
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"id": 100,
"panels": [],
"title": "Overview",
"type": "row"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"fieldConfig": {
"defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }] }, "unit": "reqps" },
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 },
"id": 1,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
"legendFormat": "requests/s",
"refId": "A"
}
],
"title": "Request Rate",
"type": "stat"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"fieldConfig": {
"defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 0.01 }] }, "unit": "percentunit" },
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 },
"id": 2,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
"legendFormat": "5xx rate",
"refId": "A"
}
],
"title": "5xx Error Rate",
"type": "stat"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"fieldConfig": {
"defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "orange", "value": 0.02 }] }, "unit": "percentunit" },
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 },
"id": 3,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m])) / sum(rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
"legendFormat": "4xx rate",
"refId": "A"
}
],
"title": "4xx Error Rate",
"type": "stat"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"fieldConfig": {
"defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 300 }, { "color": "red", "value": 1000 }] }, "unit": "ms" },
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 },
"id": 4,
"options": { "colorMode": "value", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false }, "textMode": "auto" },
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "avg(avg_over_time({container_name=\"traefik\"} | json | __error__=\"\" | unwrap Duration [5m])) / 1e6",
"legendFormat": "avg duration",
"refId": "A"
}
],
"title": "Avg Response Time",
"type": "stat"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
"id": 101,
"panels": [],
"title": "Traffic Breakdown",
"type": "row"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"description": "Request volume by status code family over time.",
"fieldConfig": {
"defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 25, "lineWidth": 1, "pointSize": 5, "showPoints": "never", "spanNulls": true, "stacking": { "group": "A", "mode": "normal" } }, "mappings": [], "unit": "reqps" },
"overrides": [
{ "matcher": { "id": "byName", "options": "2xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "3xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "4xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }] },
{ "matcher": { "id": "byName", "options": "5xx" }, "properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }] }
]
},
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 6 },
"id": 5,
"options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } },
"targets": [
{ "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 200 | DownstreamStatus < 300 [5m]))", "legendFormat": "2xx", "refId": "A" },
{ "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 300 | DownstreamStatus < 400 [5m]))", "legendFormat": "3xx", "refId": "B" },
{ "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400 | DownstreamStatus < 500 [5m]))", "legendFormat": "4xx", "refId": "C" },
{ "datasource": { "type": "loki", "uid": "${loki}" }, "expr": "sum(rate({container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 500 [5m]))", "legendFormat": "5xx", "refId": "D" }
],
"title": "Response Codes Over Time",
"type": "timeseries"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"description": "Which site (router) is generating the most log volume right now.",
"fieldConfig": {
"defaults": { "color": { "mode": "palette-classic" }, "custom": { "drawStyle": "line", "fillOpacity": 10, "lineWidth": 2, "pointSize": 5, "showPoints": "never", "spanNulls": true }, "mappings": [], "unit": "reqps" },
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 6 },
"id": 6,
"options": { "legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "multi", "sort": "desc" } },
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "sum by (RouterName) (rate({container_name=\"traefik\"} | json | __error__=\"\" [5m]))",
"legendFormat": "{{RouterName}}",
"refId": "A"
}
],
"title": "Requests by Site (Router)",
"transformations": [{ "id": "renameByRegex", "options": { "regex": "(.*)@docker", "renamePattern": "$1" } }],
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 15 },
"id": 102,
"panels": [],
"title": "Live Logs",
"type": "row"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"description": "Raw Traefik access log stream, most recent first.",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 16 },
"id": 7,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": true,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "{container_name=\"traefik\"}",
"refId": "A"
}
],
"title": "All Access Logs",
"type": "logs"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 },
"id": 103,
"panels": [],
"title": "Problems",
"type": "row"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"description": "Only requests that returned 4xx or 5xx.",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 },
"id": 8,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": true,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "{container_name=\"traefik\"} | json | __error__=\"\" | DownstreamStatus >= 400",
"refId": "A"
}
],
"title": "Error Requests (4xx/5xx)",
"type": "logs"
},
{
"datasource": { "type": "loki", "uid": "${loki}" },
"description": "Requests that took longer than 500ms to respond.",
"fieldConfig": { "defaults": {}, "overrides": [] },
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 },
"id": 9,
"options": {
"dedupStrategy": "none",
"enableLogDetails": true,
"prettifyLogMessage": true,
"showCommonLabels": false,
"showLabels": false,
"showTime": true,
"sortOrder": "Descending",
"wrapLogMessage": true
},
"targets": [
{
"datasource": { "type": "loki", "uid": "${loki}" },
"expr": "{container_name=\"traefik\"} | json | __error__=\"\" | Duration > 500000000",
"refId": "A"
}
],
"title": "Slow Requests (>500ms)",
"type": "logs"
}
],
"refresh": "10s",
"schemaVersion": 38,
"tags": ["traefik", "logs", "loki", "homelab"],
"templating": {
"list": [
{ "current": {}, "hide": 0, "label": "Loki", "name": "loki", "query": "loki", "refresh": 1, "regex": "", "type": "datasource" }
]
},
"time": { "from": "now-1h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Traefik - Access Logs",
"uid": "traefik-access-logs",
"version": 1,
"weekStart": ""
}

502
dashboards/traefik-dashboards/traefik-homelab-dashboard.json Normal file
View File

@@ -0,0 +1,502 @@
{
"annotations": {
"list": []
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
"id": 100,
"panels": [],
"title": "Overview",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [{ "color": "green", "value": null }]
},
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 1 },
"id": 1,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(traefik_router_requests_total[5m]))",
"legendFormat": "Total request rate",
"refId": "A"
}
],
"title": "Total Request Rate",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.01 },
{ "color": "red", "value": 0.05 }
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 6, "y": 1 },
"id": 2,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(traefik_router_requests_total{code=~\"5..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))",
"legendFormat": "5xx error rate",
"refId": "A"
}
],
"title": "5xx Error Rate",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{ "color": "green", "value": null },
{ "color": "yellow", "value": 0.02 },
{ "color": "red", "value": 0.1 }
]
},
"unit": "percentunit"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 1 },
"id": 3,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(rate(traefik_router_requests_total{code=~\"4..\"}[5m])) / sum(rate(traefik_router_requests_total[5m]))",
"legendFormat": "4xx error rate",
"refId": "A"
}
],
"title": "4xx Error Rate",
"type": "stat"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"fieldConfig": {
"defaults": {
"color": { "mode": "thresholds" },
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [{ "color": "green", "value": null }]
},
"unit": "none"
},
"overrides": []
},
"gridPos": { "h": 4, "w": 6, "x": 18, "y": 1 },
"id": 4,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
"textMode": "auto"
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum(traefik_open_connections)",
"legendFormat": "Open connections",
"refId": "A"
}
],
"title": "Open Connections",
"type": "stat"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 5 },
"id": 101,
"panels": [],
"title": "Which Sites Get Hit The Most",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Total requests per site (router) over the selected time range, highest first.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"fillOpacity": 80,
"gradientMode": "none",
"lineWidth": 1
},
"mappings": [],
"unit": "short"
},
"overrides": []
},
"gridPos": { "h": 10, "w": 12, "x": 0, "y": 6 },
"id": 5,
"options": {
"barRadius": 0,
"barWidth": 0.7,
"fullHighlight": false,
"groupWidth": 0.7,
"legend": { "displayMode": "list", "placement": "bottom", "showLegend": false },
"orientation": "horizontal",
"showValue": "auto",
"stacking": "none",
"tooltip": { "mode": "single", "sort": "none" },
"xTickLabelRotation": 0,
"xTickLabelSpacing": 0
},
"pluginVersion": "10.0.0",
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "topk(10, sum by (router) (increase(traefik_router_requests_total[$__range])))",
"format": "table",
"instant": true,
"legendFormat": "{{router}}",
"refId": "A"
}
],
"title": "Top 10 Sites by Total Requests",
"transformations": [
{
"id": "renameByRegex",
"options": {
"regex": "(.*)@docker",
"renamePattern": "$1"
}
}
],
"type": "barchart"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Live request rate per site — which sites are being hit right now.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true
},
"mappings": [],
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 10, "w": 12, "x": 12, "y": 6 },
"id": 6,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum by (router) (rate(traefik_router_requests_total[5m]))",
"legendFormat": "{{router}}",
"refId": "A"
}
],
"title": "Request Rate by Site (live)",
"transformations": [
{
"id": "renameByRegex",
"options": {
"regex": "(.*)@docker",
"renamePattern": "$1"
}
}
],
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 16 },
"id": 102,
"panels": [],
"title": "Health & Errors",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Requests per site broken down by status code family — spot which site is throwing errors.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 25,
"lineWidth": 1,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true,
"stacking": { "group": "A", "mode": "normal" }
},
"mappings": [],
"unit": "reqps"
},
"overrides": [
{
"matcher": { "id": "byRegexp", "options": "/5\\d\\d/" },
"properties": [{ "id": "color", "value": { "fixedColor": "red", "mode": "fixed" } }]
},
{
"matcher": { "id": "byRegexp", "options": "/4\\d\\d/" },
"properties": [{ "id": "color", "value": { "fixedColor": "orange", "mode": "fixed" } }]
},
{
"matcher": { "id": "byRegexp", "options": "/2\\d\\d/" },
"properties": [{ "id": "color", "value": { "fixedColor": "green", "mode": "fixed" } }]
}
]
},
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 17 },
"id": 7,
"options": {
"legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum by (code) (rate(traefik_router_requests_total[5m]))",
"legendFormat": "{{code}}",
"refId": "A"
}
],
"title": "Response Codes Over Time",
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Which specific site/router is generating 4xx or 5xx errors right now.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true
},
"mappings": [],
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 17 },
"id": 8,
"options": {
"legend": { "calcs": ["sum"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum by (router) (rate(traefik_router_requests_total{code=~\"4..|5..\"}[5m]))",
"legendFormat": "{{router}}",
"refId": "A"
}
],
"title": "Errors by Site",
"transformations": [
{
"id": "renameByRegex",
"options": {
"regex": "(.*)@docker",
"renamePattern": "$1"
}
}
],
"type": "timeseries"
},
{
"collapsed": false,
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 26 },
"id": 103,
"panels": [],
"title": "Performance",
"type": "row"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "How long each site takes to respond (average), per router.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 10,
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true
},
"mappings": [],
"unit": "s"
},
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 0, "y": 27 },
"id": 9,
"options": {
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum by (router) (rate(traefik_router_request_duration_seconds_sum[5m])) / sum by (router) (rate(traefik_router_request_duration_seconds_count[5m]))",
"legendFormat": "{{router}}",
"refId": "A"
}
],
"title": "Average Response Time by Site",
"transformations": [
{
"id": "renameByRegex",
"options": {
"regex": "(.*)@docker",
"renamePattern": "$1"
}
}
],
"type": "timeseries"
},
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"description": "Traffic split between HTTP (web) and HTTPS (websecure) entrypoints — confirms redirects are working.",
"fieldConfig": {
"defaults": {
"color": { "mode": "palette-classic" },
"custom": {
"axisPlacement": "auto",
"drawStyle": "line",
"fillOpacity": 15,
"lineWidth": 2,
"pointSize": 5,
"showPoints": "never",
"spanNulls": true
},
"mappings": [],
"unit": "reqps"
},
"overrides": []
},
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 27 },
"id": 10,
"options": {
"legend": { "calcs": ["mean"], "displayMode": "table", "placement": "bottom", "showLegend": true },
"tooltip": { "mode": "multi", "sort": "desc" }
},
"targets": [
{
"datasource": { "type": "prometheus", "uid": "${datasource}" },
"expr": "sum by (entrypoint) (rate(traefik_entrypoint_requests_total[5m]))",
"legendFormat": "{{entrypoint}}",
"refId": "A"
}
],
"title": "Traffic by Entrypoint (web vs websecure)",
"type": "timeseries"
}
],
"refresh": "30s",
"schemaVersion": 38,
"tags": ["traefik", "homelab"],
"templating": {
"list": [
{
"current": {},
"hide": 0,
"label": "Data source",
"name": "datasource",
"query": "prometheus",
"refresh": 1,
"regex": "",
"type": "datasource"
}
]
},
"time": { "from": "now-6h", "to": "now" },
"timepicker": {},
"timezone": "",
"title": "Traefik - Homelab Overview",
"uid": "traefik-homelab",
"version": 1,
"weekStart": ""
}

7
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/.env.example Normal file
View File

@@ -0,0 +1,7 @@
# Grafana SMTP configuration for alert emails
# Generate a Gmail App Password at: https://myaccount.google.com/apppasswords
GMAIL_SMTP_USER=your-email@gmail.com
GMAIL_APP_PASSWORD=CHANGE_ME
# Alerting contact point email (must match your Grafana login)
ALERT_EMAIL=your-email@gmail.com

12
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml
View File

@@ -33,10 +33,20 @@ services:
container_name: grafana container_name: grafana
restart: unless-stopped restart: unless-stopped
ports: ports:
- "3000:3000" # keep for now - "3000:3000"
env_file:
- .env
environment:
- GF_SMTP_ENABLED=true
- GF_SMTP_HOST=smtp.gmail.com:587
- GF_SMTP_USER=${GMAIL_SMTP_USER}
- GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD}
- GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER}
- GF_SMTP_FROM_NAME=Grafana Alerts
volumes: volumes:
- ./grafana/data:/var/lib/grafana - ./grafana/data:/var/lib/grafana
- ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
- ./grafana/provisioning/alerting:/etc/grafana/provisioning/alerting:ro
networks: networks:
- monitoring - monitoring
- web_net - web_net

10
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/contactpoints.yaml Normal file
View File

@@ -0,0 +1,10 @@
apiVersion: 1
contactPoints:
- orgId: 1
name: gmail-alerts
receivers:
- uid: gmail-email-receiver
type: email
settings:
addresses: YOUR_EMAIL@example.com
singleEmail: true

8
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/policies.yaml Normal file
View File

@@ -0,0 +1,8 @@
apiVersion: 1
policies:
- orgId: 1
receiver: gmail-alerts
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h

93
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml Normal file
View File

@@ -0,0 +1,93 @@
apiVersion: 1
groups:
- orgId: 1
name: infra-alerts
folder: Infra
interval: 1m
rules:
- uid: node-down
title: Node exporter target down
condition: C
data:
- refId: A
datasourceUid: efck3im1w4pvkf
relativeTimeRange:
from: 300
to: 0
model:
expr: up{job="node_exporter"}
instant: true
- refId: C
datasourceUid: __expr__
model:
type: threshold
expression: A
conditions:
- evaluator:
type: lt
params: [1]
for: 2m
labels:
severity: critical
annotations:
summary: "Node exporter target is down"
- uid: high-cpu
title: High CPU usage
condition: C
data:
- refId: A
datasourceUid: efck3im1w4pvkf
relativeTimeRange:
from: 300
to: 0
model:
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
instant: true
- refId: C
datasourceUid: __expr__
model:
type: threshold
expression: A
conditions:
- evaluator:
type: gt
params: [85]
for: 5m
labels:
severity: warning
annotations:
summary: "CPU usage above 85% on {{ $labels.instance }}"
- uid: loki-error-spike
title: Error log rate spike
condition: C
data:
- refId: A
datasourceUid: P8E80F9AEF21F6940
relativeTimeRange:
from: 300
to: 0
model:
expr: 'sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)'
instant: true
- refId: B
datasourceUid: __expr__
model:
type: reduce
expression: A
reducer: last
- refId: C
datasourceUid: __expr__
model:
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [20]
for: 5m
labels:
severity: warning
annotations:
summary: "Error log rate is elevated"

9
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/loki.yml Normal file
View File

@@ -0,0 +1,9 @@
apiVersion: 1
datasources:
- name: Loki
type: loki
uid: P8E80F9AEF21F6940
access: proxy
url: http://loki:3100
isDefault: false
editable: false

9
docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/datasources/prometheus.yml Normal file
View File

@@ -0,0 +1,9 @@
apiVersion: 1
datasources:
- name: prometheus
type: prometheus
uid: efck3im1w4pvkf
access: proxy
url: http://prometheus:9090
isDefault: true
editable: false

150
docs/Alerting.md Normal file
View File

@@ -0,0 +1,150 @@
# Alerting
Grafana's built-in unified alerting ties directly into the Prometheus and Loki datasources already running in the [Monitoring Stack](Monitoring-Stack.md) — no separate Alertmanager container needed. All rules, contact points, and notification policies are provisioned as code in `grafana/provisioning/alerting/`, so they survive container rebuilds and are version-controlled alongside everything else in this repo.
## Architecture
```
Prometheus ──┐
├──► Grafana Unified Alerting ──► Gmail (SMTP)
Loki ────────┘
```
Grafana evaluates alert rules against its configured datasources (Prometheus for metrics, Loki for logs). When a rule fires, the notification policy routes it to a contact point, which delivers the notification via the configured channel (email over SMTP in this setup).
No separate Alertmanager instance is required — Grafana handles the full pipeline internally.
## Alert rules
All rules live in `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/grafana/provisioning/alerting/rules.yaml` under the **Infra** folder:
| Rule | UID | Source | Condition | Severity | Duration |
|---|---|---|---|---|---|
| **Node exporter target down** | `node-down` | Prometheus | `up{job="node_exporter"} < 1` | `critical` | 2 min |
| **High CPU usage** | `high-cpu` | Prometheus | `100 - (avg idle rate) > 85%` | `warning` | 5 min |
| **Error log rate spike** | `loki-error-spike` | Loki | `count_over_time(… \|= "error" [5m]) > 20` | `warning` | 5 min |
### Node exporter target down (`node-down`)
The most critical alert — if the node exporter scrapes as down for 2 minutes, the host itself may be unreachable or the exporter process crashed. Fires at `critical` severity because it means all host-level visibility is lost.
**PromQL:**
```promql
up{job="node_exporter"}
```
**Threshold:** `< 1` (target is down)
### High CPU usage (`high-cpu`)
Watches for sustained CPU usage above 85% across the 5-minute window. The query uses `rate(node_cpu_seconds_total{mode="idle"}[5m])` to smooth out momentary spikes — a brief compile or Docker pull won't trigger it, but a runaway process or resource-hungry container will.
**PromQL:**
```promql
100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
```
**Threshold:** `> 85%`
### Error log rate spike (`loki-error-spike`)
Queries Loki for log lines containing "error" across all jobs. If the count exceeds 20 within a 5-minute window and stays elevated for 5 minutes, it fires. This catches application-level issues that metrics alone won't surface — a service that's technically "up" but flooding its logs with errors.
**LogQL:**
```logql
sum(count_over_time({job=~".+"} |= "error" [5m])) or vector(0)
```
**Threshold:** `> 20` error lines in 5 minutes
## Contact points
Defined in `grafana/provisioning/alerting/contactpoints.yaml`:
| Name | Type | Channel |
|---|---|---|
| `gmail-alerts` | Email | SMTP via Gmail App Password |
The email address is set to a placeholder (`YOUR_EMAIL@example.com`) in the provisioned file — update it to your actual address after the first Grafana container starts, or edit it directly in the file before starting.
> **Note:** Grafana provisioning files do not expand Docker environment variables (`${...}` syntax). The email address must be set as a literal string in the YAML.
## Notification policy
Defined in `grafana/provisioning/alerting/policies.yaml`:
| Setting | Value | Meaning |
|---|---|---|
| **Receiver** | `gmail-alerts` | All alerts route to the email contact point |
| **Group by** | `alertname`, `instance` | Alerts with the same name and instance are grouped into a single email |
| **Group wait** | 30s | How long to wait before sending the first notification for a new group |
| **Group interval** | 5 min | Minimum time between notifications for the same group |
| **Repeat interval** | 4 hours | How often to re-notify about an ongoing (unresolved) alert |
This means a firing alert sends its first email within ~30 seconds, groups follow-ups every 5 minutes, and you get a reminder every 4 hours until the alert resolves.
## SMTP configuration
Grafana sends alert emails through Gmail's SMTP relay. The configuration lives in the `grafana` service's environment block in `docker-compose.yml` and pulls credentials from the `.env` file:
```yaml
GF_SMTP_ENABLED=true
GF_SMTP_HOST=smtp.gmail.com:587
GF_SMTP_USER=${GMAIL_SMTP_USER}
GF_SMTP_PASSWORD=${GMAIL_APP_PASSWORD}
GF_SMTP_FROM_ADDRESS=${GMAIL_SMTP_USER}
GF_SMTP_FROM_NAME=Grafana Alerts
```
### Setting up Gmail SMTP
1. Enable 2-Step Verification on your Google account
2. Generate an [App Password](https://myaccount.google.com/apppasswords) (requires 2FA)
3. Add both values to `.env`:
```
GMAIL_SMTP_USER=your-email@gmail.com
ALERT_EMAIL=your-email@gmail.com
```
4. Set the same email as your Grafana login (`GF_SECURITY_ADMIN_USER` in `.env`, or through the Grafana UI)
## How to add new alert rules
Rules are provisioned from YAML — no UI clicks needed:
1. Open `grafana/provisioning/alerting/rules.yaml`
2. Add a new entry under the `rules` list in the `infra-alerts` group
3. Give it a unique `uid`, a descriptive `title`, and set the `datasourceUid` to match your Prometheus (`efck3im1w4pvkf`) or Loki (`P8E80F9AEF21F6940`) datasource
4. Set the `condition` refId to point to the threshold expression
5. Set `for` to how long the condition must be true before firing
6. Add `labels.severity` (`critical`, `warning`, `info`) and an `annotations.summary` message
7. Restart the Grafana container: `docker compose restart grafana`
### Common PromQL examples for new rules
```promql
# Memory usage above 90%
100 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100)
# Disk usage above 85%
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)
# Container restart count (from cAdvisor)
increase(container_restart_count[5m])
# Container memory above limit
container_memory_working_set_bytes / container_spec_memory_limit_bytes
```
## How to add new notification channels
Grafana supports Slack, Discord, PagerDuty, webhooks, and more — all configured through provisioning files:
1. Add a new receiver in `grafana/provisioning/alerting/contactpoints.yaml`
2. Add the channel to the notification policy in `policies.yaml`, or create a new policy that matches specific alert labels
3. Restart Grafana
## Lessons learned
- **Provisioning files don't expand env vars.** Unlike Docker Compose, Grafana's provisioning YAML is read as-is — `${VARIABLE}` syntax is treated as literal text. Email addresses and other non-secret values must be hardcoded in the provisioning files or managed through the Grafana UI after first boot.
- **Promtail is dead.** The `for` duration and `group_interval` in alerting policies interact with log pipeline latency — if using an older Promtail-based setup, log alerts might fire with stale data. Alloy ships logs faster and more reliably, which makes log-based alerts more responsive. See [Monitoring Stack](Monitoring-Stack.md) for the migration details.
- **Alert fatigue is real on small setups.** Three well-tuned rules cover the important failure modes without spamming. Adding too many rules to a single-node homelab generates noise faster than signal — each new rule should answer the question "would I actually get up at 3 AM for this?"

3
docs/Home.md
View File

@@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
- [Architecture and Hardware](Architecture-and-Hardware.md) - [Architecture and Hardware](Architecture-and-Hardware.md)
- [Monitoring Stack](Monitoring-Stack.md) - [Monitoring Stack](Monitoring-Stack.md)
- [Alerting](Alerting.md)
- [K3s](K3s.md) - [K3s](K3s.md)
- [Network Services](Network-Services.md) - [Network Services](Network-Services.md)
- [Media Automation](Media-Automation.md) - [Media Automation](Media-Automation.md)
@@ -32,7 +33,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
| **Host** | Ubuntu Server | | **Host** | Ubuntu Server |
| **Orchestration** | Docker Compose, one stack per service | | **Orchestration** | Docker Compose, one stack per service |
| **Management** | Portainer, **DockScope** (3D visual dashboard) | | **Management** | Portainer, **DockScope** (3D visual dashboard) |
| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) | | **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) + **Grafana Alerting** (provisioned rules + email notifications) |
| **Network** | Pi-hole (DNS filtering), Samba (file sharing), **Traefik** (reverse proxy, config-as-code) | | **Network** | Pi-hole (DNS filtering), Samba (file sharing), **Traefik** (reverse proxy, config-as-code) |
| **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent | | **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent |
| **Disk health** | Hard Disk Sentinel | | **Disk health** | Hard Disk Sentinel |

15
docs/Monitoring-Stack.md
View File

@@ -11,6 +11,19 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi
- **Loki** — stores and indexes logs by label, not full text, keeping it lightweight - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight
- **Grafana Alloy** — discovers every running container and ships its logs to Loki - **Grafana Alloy** — discovers every running container and ships its logs to Loki
- **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics - **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics
- **Grafana Alerting** — unified alerting rules evaluated against Prometheus and Loki, with email notifications via Gmail SMTP
## Alerting
Grafana's unified alerting is fully provisioned as code — rules, contact points, and notification policies live in `grafana/provisioning/alerting/` and survive container rebuilds. Three rules cover the critical failure modes:
| Rule | Source | What it catches | Severity |
|---|---|---|---|
| Node exporter target down | Prometheus | Host unreachable or exporter crashed | `critical` |
| High CPU usage | Prometheus | Sustained CPU above 85% for 5 min | `warning` |
| Error log rate spike | Loki | More than 20 error log lines in 5 min | `warning` |
Alerts route to Gmail over SMTP, grouped by alert name and instance, with a 4-hour repeat interval for ongoing issues. Full details — how to add new rules, common PromQL examples, adding Slack or other channels — in [Alerting](Alerting.md).
## Prometheus targets ## Prometheus targets
@@ -65,4 +78,4 @@ Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.sou
## Why this matters ## Why this matters
On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. Alerting completes the loop: instead of actively watching dashboards, you get an email when something actually needs attention — a host going down, CPU spiking, or error logs flooding.