diff --git a/README.md b/README.md index e6cc6f7..624ae78 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ A self-hosted homelab running on Docker Compose, built and maintained on an Ubun ![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?style=flat&logo=prometheus&logoColor=white) ![Grafana](https://img.shields.io/badge/Grafana-F46800?style=flat&logo=grafana&logoColor=white) ![Loki](https://img.shields.io/badge/Loki-F5A623?style=flat&logo=grafana&logoColor=white) +![Kubernetes](https://img.shields.io/badge/K3s-326CE5?style=flat&logo=kubernetes&logoColor=white) ## Overview @@ -37,6 +38,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s | **Custom CI images** | `docker-compose/gitlab/ci-images-reference/` | Self-built, minimal Docker images for CI pipelines, stored in a self-hosted Container Registry, rebuilt biweekly — see [Custom CI Runner Images](docs/CI-Custom-Images.md) | | **Voting app CI/CD** | `voting-app-monorepo` (self-hosted GitLab, referenced here) | Full pipeline: build, test, vulnerability scan, 3-environment deploy with rollback, load test, DB backup to MinIO — see [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md) | | **Object storage** | `docker-compose/minio/` | Self-hosted MinIO (S3-compatible), `-cpuv1` release tag for CPU compatibility — backup target for the voting app's Postgres dumps | +| **K3s cluster** | `kubernetes/` | Single-node K3s cluster running alongside the Docker Compose stack, fronted by the same reverse proxy via NodePort, monitored through the same Prometheus/Grafana/Loki stack — see [K3s](docs/K3s.md) | ## Learning labs (not always-on) @@ -137,6 +139,7 @@ homelab/ │ ├── Home.md │ ├── Architecture-and-Hardware.md │ ├── Monitoring-Stack.md +│ ├── K3s.md │ ├── Network-Services.md │ ├── Media-Automation.md │ ├── Workflow-Automation.md @@ -152,8 +155,16 @@ homelab/ │ ├── Voting-App-Monorepo.md │ ├── ElasticStack.md │ └── images/ +├── kubernetes/ +│ ├── namespace-apps.yaml +│ ├── limitrange-resourcequota.yaml +│ ├── kube-state-metrics/ +│ │ └── rbac-patch.yaml +│ └── backups/ +│ └── k3s-state-backup.sh ├── dashboards/ -│ └── gitlab-application-runner-metrics.json +│ ├── gitlab-application-runner-metrics.json +│ └── k3s-professional-log-analyzer.json └── .gitignore ``` @@ -199,6 +210,9 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - Docker images with `ENTRYPOINT` hardcoded to the tool itself (Trivy, k6, MinIO's `mc`) break GitLab CI's default script invocation with a confusing "unknown command sh" error — fixed the same way every time with `entrypoint: [""]`. See [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md). - In Docker-outside-of-Docker setups, bind mounts (`-v host:container`) always resolve against the **host's** filesystem regardless of which container issued the `docker run` command — `docker cp` is the correct way to move a file into a container across that boundary, not a shared path assumption. - MinIO's current images require `x86-64-v2` CPU instructions and crash outright on older hardware (Phenom II here) with a `Fatal glibc error` — fixed by pinning to a `-cpuv1` release tag, a legacy compatibility branch MinIO still publishes for exactly this case. +- `kube-state-metrics`'s default ClusterRole doesn't cover every resource it tries to watch (`NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration`, and more) — surfaces as repeated `reflector.go` `Failed to list ... is forbidden` warnings rather than a single clear error. Fixed with a supplementary `ClusterRole`/`ClusterRoleBinding` granting just the missing `list`/`watch` verbs. See [K3s](docs/K3s.md). +- A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it. +- Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it. ## Roadmap diff --git a/dashboards/kubernetes-dashboards/README.md b/dashboards/kubernetes-dashboards/README.md new file mode 100644 index 0000000..4b78f57 --- /dev/null +++ b/dashboards/kubernetes-dashboards/README.md @@ -0,0 +1,27 @@ +# Kubernetes Dashboards + +Grafana dashboards for the K3s cluster (see [K3s](../../docs/K3s.md)). Two are imported from a community collection, two are hand-built for this homelab specifically. + +## Dashboards + +### `dashboard-1786902530431.json` — Kubernetes / Views / Pods + +Per-pod detail view: CPU/memory usage against requests and limits, network bandwidth/packets/errors, OOM events, container restarts, throttling, QOS and priority class, and unscheduled/problem pods. Imported from [dotdc/grafana-dashboards-kubernetes](https://github.com/dotdc/grafana-dashboards-kubernetes), built for the `kube-prometheus-stack` metric set — works as-is against this cluster's `kube-state-metrics` + cAdvisor scrape targets (see [Monitoring Stack](../../docs/Monitoring-Stack.md)). Data source: Prometheus. + +### `dashboard-1786902563692.json` — Kubernetes / Views / K3s Cluster + +Cluster-wide overview: total/used CPU and memory gauges, node health count, per-namespace and per-node CPU/memory/network breakdowns, and cluster filesystem usage (the storage panels — Ceph, Longhorn, ZFS ZVOLs — read `N/A` here since none of those are in use; this cluster runs local-path storage only). Also from [dotdc/grafana-dashboards-kubernetes](https://github.com/dotdc/grafana-dashboards-kubernetes). Data source: Prometheus. + +![K3s Cluster view — total and cluster usage panels](../../docs/images/k3s-cluster-view.png) + +### `dashboard-1786902586059.json` — K3s Unified Application Dashboard + +Hand-built, namespace/pod-filtered application view: running vs. problem pod counts (`kube_pod_status_phase`), per-pod CPU and memory usage (`container_cpu_usage_seconds_total`, `container_memory_working_set_bytes`), and a live Loki log stream scoped to `cluster="k3s-homeserver"`. A lighter-weight day-to-day view than the two imported dashboards above — built to answer "is *this* app healthy" rather than "how's the whole cluster." Data sources: Prometheus + Loki. + +### `dashboard-1786902623268.json` — K3s Professional Log Analyzer + +Hand-built, log-focused: errors-vs-warnings timeline, top error-producing pods, and a filtered live log stream with namespace/pod/container/severity/text filters. LogQL queries carry a `!~` exclusion for known-benign noise (CoreDNS's config-glob warning, `kube-state-metrics`'s `EndpointSlice` deprecation notice) so the error/warning counts reflect real issues — see [Lessons Learned](../../docs/Lessons-Learned.md). Data source: Loki. Already referenced in [Monitoring Stack](../../docs/Monitoring-Stack.md). + +## Why two sets + +The two imported dashboards (Pods, K3s Cluster) give broad, standard Kubernetes observability out of the box — useful for general cluster health without building anything. The two hand-built ones (Unified Application Dashboard, Professional Log Analyzer) are scoped to this specific setup: single `k3s-homeserver` cluster, Loki log correlation, and noise already filtered for the warnings this homelab actually produces. Kept side by side rather than merged, so the general-purpose community dashboards stay easy to update/re-import from upstream without losing the custom ones. diff --git a/docs/Home.md b/docs/Home.md index f80c909..edbd63a 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This - [Architecture and Hardware](Architecture-and-Hardware.md) - [Monitoring Stack](Monitoring-Stack.md) +- [K3s](K3s.md) - [Network Services](Network-Services.md) - [Media Automation](Media-Automation.md) - [Workflow Automation](Workflow-Automation.md) @@ -34,6 +35,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This | **Disk health** | Hard Disk Sentinel | | **AI Agent** | OpenHands + Ollama | | **CI/CD** | Self-hosted **GitLab** (Omnibus) + Docker-executor **Runner** | +| **Orchestration (K8s)** | Single-node **K3s** cluster, monitored via the same Prometheus/Grafana/Loki stack | This is a running system, not a demo — screenshots throughout this wiki are from the live dashboards. diff --git a/docs/Lessons-Learned.md b/docs/Lessons-Learned.md index 795c8b9..00c3367 100644 --- a/docs/Lessons-Learned.md +++ b/docs/Lessons-Learned.md @@ -18,6 +18,19 @@ Standing up Prometheus + Grafana + cAdvisor + node-exporter early, before anythi Hard Disk Sentinel running continuously catches degrading drives via S.M.A.R.T. data well before failure — reallocated sectors and rising temperatures are visible weeks or months ahead of an actual crash. +## K3s: RBAC gaps look like a wall of errors, not one clear message + +`kube-state-metrics`'s bundled ClusterRole doesn't cover every resource type it tries to watch — `NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration` were all missing here. The symptom wasn't one obvious permissions error; it was a continuous stream of `reflector.go` `Failed to list ... is forbidden` lines, one per resource type, repeating every reconcile loop. The fix was a supplementary `ClusterRole` + `ClusterRoleBinding` granting just the missing `list`/`watch` verbs, applied alongside the chart's own RBAC rather than replacing it. + +Two things weren't obvious the first time through: + +- **Widening the RBAC grant isn't enough on its own.** The running pod's informers had already failed and cached that failure state — they don't retry against newly-granted permissions automatically. `kubectl rollout restart deployment kube-state-metrics -n monitoring` was required before the fix actually took effect. +- **Old errors don't vanish from Grafana the moment a fix ships.** Loki keeps every line it already ingested; a dashboard's "Errors" panel only reflects what falls inside its time window (e.g. "Last 1 hour"). Right after the restart the panel still showed the old forbidden-access errors, which briefly looked like the fix hadn't worked — it had. Checking `kubectl logs --since=` directly, and watching the panel's error *total* trend downward over a few minutes, confirmed it rather than expecting an instant clean slate. + +## Not every WARN/ERROR line is worth alerting on + +Two log patterns showed up constantly on the K3s log dashboard without indicating an actual problem: CoreDNS logging a `WARN` every time it globs for an optional custom-config file that doesn't exist, and `kube-state-metrics` logging a `v1 Endpoints is deprecated` notice ahead of a future Kubernetes API removal. Neither needed a cluster-side fix — the dashboard's LogQL queries got a `!~` (negative-match) exclusion for both known-benign patterns instead, so the Errors/Warnings panels reflect things actually worth investigating. + ## What's next - Self-hosted Gitea for private repos and CI runners diff --git a/docs/Monitoring-Stack.md b/docs/Monitoring-Stack.md index a0f4312..f612357 100644 --- a/docs/Monitoring-Stack.md +++ b/docs/Monitoring-Stack.md @@ -10,6 +10,7 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi - **Grafana** — dashboards on top of Prometheus and Loki data - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight - **Grafana Alloy** — discovers every running container and ships its logs to Loki +- **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics ## Prometheus targets @@ -51,6 +52,17 @@ Self-hosted GitLab (see [GitLab CI/CD](GitLab-CICD.md)) exposes its own Promethe Visualized on a hand-built Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) rather than GitLab's official community dashboard — that one targets metric names from Omnibus's own bundled Prometheus, which isn't what's running here. +## K3s cluster monitoring + +The K3s cluster (see [K3s](K3s.md)) is folded into this same stack rather than running a second, cluster-local Prometheus/Grafana/Loki — cAdvisor here only watches the Docker socket, so it has zero visibility into K8s pods on its own. Two additions close that gap: + +- **[`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)** — deployed into the cluster, scraped as a standard Prometheus target, exposing object-level state (replica counts, pod restarts, resource requests vs. limits). +- **Kubelet's built-in cAdvisor** — already running as part of K3s, scraped at `https://:10250/metrics/cadvisor` for per-pod CPU/memory, authenticated via a scoped bearer-token `ServiceAccount` rather than the open endpoint the Docker-side cAdvisor uses. + +Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.source.kubernetes` components stream pod logs into the same Loki instance as the Docker container logs, tagged with `cluster="k3s-homeserver"` to distinguish K3s-origin entries from Docker-origin ones at query time. + +**K3s Professional Log Analyzer** (`dashboards/k3s-professional-log-analyzer.json`) is the hand-built Grafana dashboard for this — namespace/pod/container filters, an errors-vs-warnings timeline, top error-producing pods, and a live filtered log stream. Its LogQL queries carry a `!~` exclusion for known-benign noise (CoreDNS's optional-config-file glob warning, `kube-state-metrics`'s `EndpointSlice` deprecation notice) so the Errors/Warnings panels stay focused on things actually worth investigating rather than cosmetic log lines. See [Lessons Learned](Lessons-Learned.md) for how that gap was found and closed. + ## Why this matters On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.