updated the docs for k3s and new dashboards

README.md

@@ -9,6 +9,7 @@ A self-hosted homelab running on Docker Compose, built and maintained on an Ubun
 ![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?style=flat&logo=prometheus&logoColor=white)
 ![Grafana](https://img.shields.io/badge/Grafana-F46800?style=flat&logo=grafana&logoColor=white)
 ![Loki](https://img.shields.io/badge/Loki-F5A623?style=flat&logo=grafana&logoColor=white)
+![Kubernetes](https://img.shields.io/badge/K3s-326CE5?style=flat&logo=kubernetes&logoColor=white)
 
 ## Overview
 
@@ -37,6 +38,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s
 | **Custom CI images** | `docker-compose/gitlab/ci-images-reference/` | Self-built, minimal Docker images for CI pipelines, stored in a self-hosted Container Registry, rebuilt biweekly — see [Custom CI Runner Images](docs/CI-Custom-Images.md) |
 | **Voting app CI/CD** | `voting-app-monorepo` (self-hosted GitLab, referenced here) | Full pipeline: build, test, vulnerability scan, 3-environment deploy with rollback, load test, DB backup to MinIO — see [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md) |
 | **Object storage** | `docker-compose/minio/` | Self-hosted MinIO (S3-compatible), `-cpuv1` release tag for CPU compatibility — backup target for the voting app's Postgres dumps |
+| **K3s cluster** | `kubernetes/` | Single-node K3s cluster running alongside the Docker Compose stack, fronted by the same reverse proxy via NodePort, monitored through the same Prometheus/Grafana/Loki stack — see [K3s](docs/K3s.md) |
 
 ## Learning labs (not always-on)
 
@@ -137,6 +139,7 @@ homelab/
 │   ├── Home.md
 │   ├── Architecture-and-Hardware.md
 │   ├── Monitoring-Stack.md
+│   ├── K3s.md
 │   ├── Network-Services.md
 │   ├── Media-Automation.md
 │   ├── Workflow-Automation.md
@@ -152,8 +155,16 @@ homelab/
 │   ├── Voting-App-Monorepo.md
 │   ├── ElasticStack.md
 │   └── images/
+├── kubernetes/
+│   ├── namespace-apps.yaml
+│   ├── limitrange-resourcequota.yaml
+│   ├── kube-state-metrics/
+│   │   └── rbac-patch.yaml
+│   └── backups/
+│       └── k3s-state-backup.sh
 ├── dashboards/
-│   └── gitlab-application-runner-metrics.json
+│   ├── gitlab-application-runner-metrics.json
+│   └── k3s-professional-log-analyzer.json
 └── .gitignore
 ```
 
@@ -199,6 +210,9 @@ Rather than pulling and recreating manually, every always-on stack has a matchin
 - Docker images with `ENTRYPOINT` hardcoded to the tool itself (Trivy, k6, MinIO's `mc`) break GitLab CI's default script invocation with a confusing "unknown command sh" error — fixed the same way every time with `entrypoint: [""]`. See [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md).
 - In Docker-outside-of-Docker setups, bind mounts (`-v host:container`) always resolve against the **host's** filesystem regardless of which container issued the `docker run` command — `docker cp` is the correct way to move a file into a container across that boundary, not a shared path assumption.
 - MinIO's current images require `x86-64-v2` CPU instructions and crash outright on older hardware (Phenom II here) with a `Fatal glibc error` — fixed by pinning to a `-cpuv1` release tag, a legacy compatibility branch MinIO still publishes for exactly this case.
+- `kube-state-metrics`'s default ClusterRole doesn't cover every resource it tries to watch (`NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration`, and more) — surfaces as repeated `reflector.go` `Failed to list ... is forbidden` warnings rather than a single clear error. Fixed with a supplementary `ClusterRole`/`ClusterRoleBinding` granting just the missing `list`/`watch` verbs. See [K3s](docs/K3s.md).
+- A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it.
+- Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it.
 
 ## Roadmap
 

dashboards/kubernetes-dashboards/README.md

@@ -0,0 +1,27 @@
+# Kubernetes Dashboards
+
+Grafana dashboards for the K3s cluster (see [K3s](../../docs/K3s.md)). Two are imported from a community collection, two are hand-built for this homelab specifically.
+
+## Dashboards
+
+### `dashboard-1786902530431.json` — Kubernetes / Views / Pods
+
+Per-pod detail view: CPU/memory usage against requests and limits, network bandwidth/packets/errors, OOM events, container restarts, throttling, QOS and priority class, and unscheduled/problem pods. Imported from [dotdc/grafana-dashboards-kubernetes](https://github.com/dotdc/grafana-dashboards-kubernetes), built for the `kube-prometheus-stack` metric set — works as-is against this cluster's `kube-state-metrics` + cAdvisor scrape targets (see [Monitoring Stack](../../docs/Monitoring-Stack.md)). Data source: Prometheus.
+
+### `dashboard-1786902563692.json` — Kubernetes / Views / K3s Cluster
+
+Cluster-wide overview: total/used CPU and memory gauges, node health count, per-namespace and per-node CPU/memory/network breakdowns, and cluster filesystem usage (the storage panels — Ceph, Longhorn, ZFS ZVOLs — read `N/A` here since none of those are in use; this cluster runs local-path storage only). Also from [dotdc/grafana-dashboards-kubernetes](https://github.com/dotdc/grafana-dashboards-kubernetes). Data source: Prometheus.
+
+![K3s Cluster view — total and cluster usage panels](../../docs/images/k3s-cluster-view.png)
+
+### `dashboard-1786902586059.json` — K3s Unified Application Dashboard
+
+Hand-built, namespace/pod-filtered application view: running vs. problem pod counts (`kube_pod_status_phase`), per-pod CPU and memory usage (`container_cpu_usage_seconds_total`, `container_memory_working_set_bytes`), and a live Loki log stream scoped to `cluster="k3s-homeserver"`. A lighter-weight day-to-day view than the two imported dashboards above — built to answer "is *this* app healthy" rather than "how's the whole cluster." Data sources: Prometheus + Loki.
+
+### `dashboard-1786902623268.json` — K3s Professional Log Analyzer
+
+Hand-built, log-focused: errors-vs-warnings timeline, top error-producing pods, and a filtered live log stream with namespace/pod/container/severity/text filters. LogQL queries carry a `!~` exclusion for known-benign noise (CoreDNS's config-glob warning, `kube-state-metrics`'s `EndpointSlice` deprecation notice) so the error/warning counts reflect real issues — see [Lessons Learned](../../docs/Lessons-Learned.md). Data source: Loki. Already referenced in [Monitoring Stack](../../docs/Monitoring-Stack.md).
+
+## Why two sets
+
+The two imported dashboards (Pods, K3s Cluster) give broad, standard Kubernetes observability out of the box — useful for general cluster health without building anything. The two hand-built ones (Unified Application Dashboard, Professional Log Analyzer) are scoped to this specific setup: single `k3s-homeserver` cluster, Loki log correlation, and noise already filtered for the warnings this homelab actually produces. Kept side by side rather than merged, so the general-purpose community dashboards stay easy to update/re-import from upstream without losing the custom ones.

docs/Home.md

@@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 
 - [Architecture and Hardware](Architecture-and-Hardware.md)
 - [Monitoring Stack](Monitoring-Stack.md)
+- [K3s](K3s.md)
 - [Network Services](Network-Services.md)
 - [Media Automation](Media-Automation.md)
 - [Workflow Automation](Workflow-Automation.md)
@@ -34,6 +35,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 | **Disk health** | Hard Disk Sentinel |
 | **AI Agent** | OpenHands + Ollama |
 | **CI/CD** | Self-hosted **GitLab** (Omnibus) + Docker-executor **Runner** |
+| **Orchestration (K8s)** | Single-node **K3s** cluster, monitored via the same Prometheus/Grafana/Loki stack |
 
 This is a running system, not a demo — screenshots throughout this wiki are from the live dashboards.
 

docs/Lessons-Learned.md

@@ -18,6 +18,19 @@ Standing up Prometheus + Grafana + cAdvisor + node-exporter early, before anythi
 
 Hard Disk Sentinel running continuously catches degrading drives via S.M.A.R.T. data well before failure — reallocated sectors and rising temperatures are visible weeks or months ahead of an actual crash.
 
+## K3s: RBAC gaps look like a wall of errors, not one clear message
+
+`kube-state-metrics`'s bundled ClusterRole doesn't cover every resource type it tries to watch — `NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration` were all missing here. The symptom wasn't one obvious permissions error; it was a continuous stream of `reflector.go` `Failed to list ... is forbidden` lines, one per resource type, repeating every reconcile loop. The fix was a supplementary `ClusterRole` + `ClusterRoleBinding` granting just the missing `list`/`watch` verbs, applied alongside the chart's own RBAC rather than replacing it.
+
+Two things weren't obvious the first time through:
+
+- **Widening the RBAC grant isn't enough on its own.** The running pod's informers had already failed and cached that failure state — they don't retry against newly-granted permissions automatically. `kubectl rollout restart deployment kube-state-metrics -n monitoring` was required before the fix actually took effect.
+- **Old errors don't vanish from Grafana the moment a fix ships.** Loki keeps every line it already ingested; a dashboard's "Errors" panel only reflects what falls inside its time window (e.g. "Last 1 hour"). Right after the restart the panel still showed the old forbidden-access errors, which briefly looked like the fix hadn't worked — it had. Checking `kubectl logs <new-pod> --since=<pod-age>` directly, and watching the panel's error *total* trend downward over a few minutes, confirmed it rather than expecting an instant clean slate.
+
+## Not every WARN/ERROR line is worth alerting on
+
+Two log patterns showed up constantly on the K3s log dashboard without indicating an actual problem: CoreDNS logging a `WARN` every time it globs for an optional custom-config file that doesn't exist, and `kube-state-metrics` logging a `v1 Endpoints is deprecated` notice ahead of a future Kubernetes API removal. Neither needed a cluster-side fix — the dashboard's LogQL queries got a `!~` (negative-match) exclusion for both known-benign patterns instead, so the Errors/Warnings panels reflect things actually worth investigating.
+
 ## What's next
 
 - Self-hosted Gitea for private repos and CI runners

docs/Monitoring-Stack.md

@@ -10,6 +10,7 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi
 - **Grafana** — dashboards on top of Prometheus and Loki data
 - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight
 - **Grafana Alloy** — discovers every running container and ships its logs to Loki
+- **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics
 
 ## Prometheus targets
 
@@ -51,6 +52,17 @@ Self-hosted GitLab (see [GitLab CI/CD](GitLab-CICD.md)) exposes its own Promethe
 
 Visualized on a hand-built Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) rather than GitLab's official community dashboard — that one targets metric names from Omnibus's own bundled Prometheus, which isn't what's running here.
 
+## K3s cluster monitoring
+
+The K3s cluster (see [K3s](K3s.md)) is folded into this same stack rather than running a second, cluster-local Prometheus/Grafana/Loki — cAdvisor here only watches the Docker socket, so it has zero visibility into K8s pods on its own. Two additions close that gap:
+
+- **[`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)** — deployed into the cluster, scraped as a standard Prometheus target, exposing object-level state (replica counts, pod restarts, resource requests vs. limits).
+- **Kubelet's built-in cAdvisor** — already running as part of K3s, scraped at `https://<host>:10250/metrics/cadvisor` for per-pod CPU/memory, authenticated via a scoped bearer-token `ServiceAccount` rather than the open endpoint the Docker-side cAdvisor uses.
+
+Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.source.kubernetes` components stream pod logs into the same Loki instance as the Docker container logs, tagged with `cluster="k3s-homeserver"` to distinguish K3s-origin entries from Docker-origin ones at query time.
+
+**K3s Professional Log Analyzer** (`dashboards/k3s-professional-log-analyzer.json`) is the hand-built Grafana dashboard for this — namespace/pod/container filters, an errors-vs-warnings timeline, top error-producing pods, and a live filtered log stream. Its LogQL queries carry a `!~` exclusion for known-benign noise (CoreDNS's optional-config-file glob warning, `kube-state-metrics`'s `EndpointSlice` deprecation notice) so the Errors/Warnings panels stay focused on things actually worth investigating rather than cosmetic log lines. See [Lessons Learned](Lessons-Learned.md) for how that gap was found and closed.
+
 ## Why this matters
 
 On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.