updated the docs for k3s and new dashboards

docs/Home.md

@@ -6,6 +6,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 
 - [Architecture and Hardware](Architecture-and-Hardware.md)
 - [Monitoring Stack](Monitoring-Stack.md)
+- [K3s](K3s.md)
 - [Network Services](Network-Services.md)
 - [Media Automation](Media-Automation.md)
 - [Workflow Automation](Workflow-Automation.md)
@@ -34,6 +35,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 | **Disk health** | Hard Disk Sentinel |
 | **AI Agent** | OpenHands + Ollama |
 | **CI/CD** | Self-hosted **GitLab** (Omnibus) + Docker-executor **Runner** |
+| **Orchestration (K8s)** | Single-node **K3s** cluster, monitored via the same Prometheus/Grafana/Loki stack |
 
 This is a running system, not a demo — screenshots throughout this wiki are from the live dashboards.
 

docs/Lessons-Learned.md

@@ -18,6 +18,19 @@ Standing up Prometheus + Grafana + cAdvisor + node-exporter early, before anythi
 
 Hard Disk Sentinel running continuously catches degrading drives via S.M.A.R.T. data well before failure — reallocated sectors and rising temperatures are visible weeks or months ahead of an actual crash.
 
+## K3s: RBAC gaps look like a wall of errors, not one clear message
+
+`kube-state-metrics`'s bundled ClusterRole doesn't cover every resource type it tries to watch — `NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration` were all missing here. The symptom wasn't one obvious permissions error; it was a continuous stream of `reflector.go` `Failed to list ... is forbidden` lines, one per resource type, repeating every reconcile loop. The fix was a supplementary `ClusterRole` + `ClusterRoleBinding` granting just the missing `list`/`watch` verbs, applied alongside the chart's own RBAC rather than replacing it.
+
+Two things weren't obvious the first time through:
+
+- **Widening the RBAC grant isn't enough on its own.** The running pod's informers had already failed and cached that failure state — they don't retry against newly-granted permissions automatically. `kubectl rollout restart deployment kube-state-metrics -n monitoring` was required before the fix actually took effect.
+- **Old errors don't vanish from Grafana the moment a fix ships.** Loki keeps every line it already ingested; a dashboard's "Errors" panel only reflects what falls inside its time window (e.g. "Last 1 hour"). Right after the restart the panel still showed the old forbidden-access errors, which briefly looked like the fix hadn't worked — it had. Checking `kubectl logs <new-pod> --since=<pod-age>` directly, and watching the panel's error *total* trend downward over a few minutes, confirmed it rather than expecting an instant clean slate.
+
+## Not every WARN/ERROR line is worth alerting on
+
+Two log patterns showed up constantly on the K3s log dashboard without indicating an actual problem: CoreDNS logging a `WARN` every time it globs for an optional custom-config file that doesn't exist, and `kube-state-metrics` logging a `v1 Endpoints is deprecated` notice ahead of a future Kubernetes API removal. Neither needed a cluster-side fix — the dashboard's LogQL queries got a `!~` (negative-match) exclusion for both known-benign patterns instead, so the Errors/Warnings panels reflect things actually worth investigating.
+
 ## What's next
 
 - Self-hosted Gitea for private repos and CI runners

docs/Monitoring-Stack.md

@@ -10,6 +10,7 @@ Prometheus, Grafana, cAdvisor, and node-exporter running together give full visi
 - **Grafana** — dashboards on top of Prometheus and Loki data
 - **Loki** — stores and indexes logs by label, not full text, keeping it lightweight
 - **Grafana Alloy** — discovers every running container and ships its logs to Loki
+- **kube-state-metrics** — exposes K3s object-level state (deployments, pod restarts, resource requests/limits) as Prometheus metrics
 
 ## Prometheus targets
 
@@ -51,6 +52,17 @@ Self-hosted GitLab (see [GitLab CI/CD](GitLab-CICD.md)) exposes its own Promethe
 
 Visualized on a hand-built Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) rather than GitLab's official community dashboard — that one targets metric names from Omnibus's own bundled Prometheus, which isn't what's running here.
 
+## K3s cluster monitoring
+
+The K3s cluster (see [K3s](K3s.md)) is folded into this same stack rather than running a second, cluster-local Prometheus/Grafana/Loki — cAdvisor here only watches the Docker socket, so it has zero visibility into K8s pods on its own. Two additions close that gap:
+
+- **[`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)** — deployed into the cluster, scraped as a standard Prometheus target, exposing object-level state (replica counts, pod restarts, resource requests vs. limits).
+- **Kubelet's built-in cAdvisor** — already running as part of K3s, scraped at `https://<host>:10250/metrics/cadvisor` for per-pod CPU/memory, authenticated via a scoped bearer-token `ServiceAccount` rather than the open endpoint the Docker-side cAdvisor uses.
+
+Logs follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.source.kubernetes` components stream pod logs into the same Loki instance as the Docker container logs, tagged with `cluster="k3s-homeserver"` to distinguish K3s-origin entries from Docker-origin ones at query time.
+
+**K3s Professional Log Analyzer** (`dashboards/k3s-professional-log-analyzer.json`) is the hand-built Grafana dashboard for this — namespace/pod/container filters, an errors-vs-warnings timeline, top error-producing pods, and a live filtered log stream. Its LogQL queries carry a `!~` exclusion for known-benign noise (CoreDNS's optional-config-file glob warning, `kube-state-metrics`'s `EndpointSlice` deprecation notice) so the Errors/Warnings panels stay focused on things actually worth investigating rather than cosmetic log lines. See [Lessons Learned](Lessons-Learned.md) for how that gap was found and closed.
+
 ## Why this matters
 
 On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.