diff --git a/dashboards/dashboards-README.md b/dashboards/gitlab-dashboards/dashboards-README.md similarity index 100% rename from dashboards/dashboards-README.md rename to dashboards/gitlab-dashboards/dashboards-README.md diff --git a/dashboards/gitlab-application-runner-metrics.json b/dashboards/gitlab-dashboards/gitlab-application-runner-metrics.json similarity index 100% rename from dashboards/gitlab-application-runner-metrics.json rename to dashboards/gitlab-dashboards/gitlab-application-runner-metrics.json diff --git a/docs/K3s.md b/docs/K3s.md new file mode 100644 index 0000000..09b7d99 --- /dev/null +++ b/docs/K3s.md @@ -0,0 +1,116 @@ +# K3s + +A single-node K3s cluster running alongside the existing Docker Compose stack — the first step toward learning Kubernetes hands-on rather than purely through theory, without disrupting the services already running on this host. + +## Why K3s, not kind or full kubeadm + +Full `kubeadm` Kubernetes runs a separate etcd cluster plus independent API server, controller-manager, and scheduler processes — too much overhead for this host's RAM budget. `kind` runs each cluster node as a Docker container, which nests Kubernetes' own networking and iptables rules inside a host that already has Docker managing iptables for 25+ services — more surface area for conflicts, not less, and better suited to disposable CI/test clusters than a persistent homelab service. + +K3s runs as a single systemd-managed binary directly on the host, with SQLite (via `kine`) standing in for etcd on a single-node setup — the lightest option that still behaves like real Kubernetes. + +## Install + +```bash +curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server \ + --disable=traefik \ + --disable=servicelb \ + --flannel-backend=vxlan \ + --write-kubeconfig-mode=644" sh - +``` + +- **`--disable=traefik` / `--disable=servicelb`** — K3s ships with its own ingress controller and LoadBalancer implementation, both of which try to bind ports 80/443. This host's reverse proxy (see [Reverse Proxy](Reverse-Proxy.md)) already owns those ports — disabling K3s's defaults avoids the conflict entirely rather than debugging it after the fact. +- **`--flannel-backend=vxlan`** — explicit rather than relying on the default, and chosen over `host-gw` since VXLAN works across nodes that aren't on the same L2 segment, keeping the door open for a future worker node. +- **`--write-kubeconfig-mode=644`** — makes `/etc/rancher/k3s/k3s.yaml` world-readable so a personal, non-root kubeconfig copy can be made without repeated `sudo`. + +## Exposure pattern: NPM in front, NodePort behind + +Rather than running a second ingress controller inside the cluster (which would mean two reverse proxies stacked for no benefit), each service gets a `Service` of `type: NodePort`. The existing reverse proxy adds one more proxy host pointing at `homeserver:` — identical to how it already fronts every Docker container. From the proxy's perspective, a K8s pod behind a NodePort is indistinguishable from a Docker container; same TLS setup, same cert renewal flow, zero new moving parts in the HTTPS chain. + +## Namespace and resource governance + +All workloads live in a dedicated `apps` namespace (not `default`), governed by a `LimitRange` and `ResourceQuota` sized for this host's RAM budget: + +```yaml +apiVersion: v1 +kind: LimitRange +metadata: + name: apps-limits + namespace: apps +spec: + limits: + - type: Container + default: { cpu: "200m", memory: "256Mi" } + defaultRequest: { cpu: "50m", memory: "64Mi" } + max: { cpu: "500m", memory: "512Mi" } + min: { cpu: "10m", memory: "16Mi" } +--- +apiVersion: v1 +kind: ResourceQuota +metadata: + name: apps-quota + namespace: apps +spec: + hard: + requests.cpu: "1" + requests.memory: "768Mi" + limits.cpu: "1.5" + limits.memory: "1.5Gi" + pods: "10" +``` + +Any container deployed without explicit resource fields automatically inherits the `LimitRange` defaults; no container can exceed the `max` bounds even if it tries; and the whole namespace is hard-capped at 1.5 CPU / 1.5Gi RAM combined — protecting the Docker services and the OS from a runaway K8s workload, regardless of how many pods end up running. + +## Backups + +The cluster's SQLite state (`/var/lib/rancher/k3s/server/db/state.db`) is backed up on a rolling schedule via a systemd timer — deliberately off the OS disk, onto a separate physical drive: + +```bash +sqlite3 "$DB_PATH" ".backup '$BACKUP_FILE'" +gzip "$BACKUP_FILE" +# keep only the 2 most recent, prune the rest +``` + +Runs every 4 days via `OnUnitActiveSec=4d` with `Persistent=true`, so a missed run (host powered off) catches up on next boot rather than silently skipping. This covers cluster state — Deployments, Services, config — not data inside PersistentVolumes; there are no stateful workloads in the cluster yet, so that gap doesn't currently apply. + +## kubectl ergonomics + +- Personal kubeconfig at `~/.kube/config` (copied from `/etc/rancher/k3s/k3s.yaml`), with `KUBECONFIG` explicitly exported in `.zshrc` — avoids the k3s-bundled `kubectl` silently falling back to its own default path and disagreeing with other tools about which config is authoritative. +- `kubectx` / `kubens` for fast context and namespace switching, plus oh-my-zsh's `kubectl` plugin for tab-completion and short aliases (`kgp`, `kgd`, `kaf`, etc.). +- Default namespace set to `apps` via `kubens apps`, so day-to-day commands don't need `-n apps` on every invocation. + +## GUI: FreeLens + +[FreeLens](https://github.com/freelensapp/freelens) — the actively maintained, fully open-source fork of Lens, created after the original Lens IDE moved behind a paid subscription. Same core experience (cluster overview, live workload view, logs, terminal) reading the same kubeconfig, with no account and no license cost — a better fit for a self-hosted learning project than a subscription tool. + +![FreeLens namespaces view](images/freelens-namespaces.png) + +![FreeLens nodes view](images/freelens-nodes.png) + +## Monitoring integration + +Rather than deploying a second Prometheus/Grafana/Loki stack inside K3s, the existing Docker-based monitoring stack (see [Monitoring Stack](Monitoring-Stack.md)) was extended to scrape and tail the K3s cluster as an additional source — avoiding the RAM cost of duplicate infrastructure. + +**The gap this closes:** K3s uses containerd, not Docker, so the existing cAdvisor container (which watches the Docker socket) has no visibility into K8s pods at all. Two things bridge that gap: + +- **[`kube-state-metrics`](https://github.com/kubernetes/kube-state-metrics)** — deployed into the cluster, exposes object-level state (deployment replica counts, pod restarts, resource requests vs. limits) as Prometheus metrics via a NodePort. +- **Kubelet's built-in cAdvisor** — already running as part of K3s, exposes per-pod CPU/memory at `https://:10250/metrics/cadvisor`. Requires a bearer token, unlike the open Docker cAdvisor endpoint — sourced from a scoped `ServiceAccount` created specifically for external Prometheus access. + +Both added as standard Prometheus scrape jobs, authenticated via a `ClusterRole` limited to read-only access on nodes, pods, and services — no broader cluster privileges than the metrics actually require. + +**Logs** follow the same pattern: Grafana Alloy's `discovery.kubernetes` + `loki.source.kubernetes` components stream pod logs through the K8s API (each needing its own `client` block with the same kind of scoped token — the two components don't share credentials even when defined side-by-side), landing in the same Loki instance as the Docker container logs. K3s pod log streams carry a `cluster="k3s-homeserver"` label to distinguish them from Docker-origin entries at query time. + +## Registry access under sanctions + +`registry.k8s.io` (Google-hosted) returns a hard `403 Forbidden` from this network — a harder block than the intermittent Docker Hub flakiness described elsewhere in this wiki, and one that will resurface for any future addon (cert-manager, ingress-nginx, etc.) that defaults to pulling from it. + +**Working pattern when this happens:** +```bash +docker pull # Docker Hub / ghcr.io usually reachable +docker save -o /tmp/image.tar +sudo k3s ctr images import /tmp/image.tar # load directly into containerd +``` +Sidesteps `registry.k8s.io` entirely by pulling through Docker (which has working access) and importing straight into containerd's local store, rather than waiting on network access that isn't coming. + +## Why this matters + +The instinct to keep K3s and the existing Docker stack cleanly separated — one reverse proxy, one monitoring stack, one set of conventions — paid off directly: every new K8s workload is visible in the same Grafana instance, fronted by the same TLS setup, without a second parallel toolchain to maintain. On a four-core host with limited RAM, reusing existing infrastructure instead of duplicating it isn't just tidier — it's the difference between K3s fitting on this hardware at all and it competing with everything else already running here. diff --git a/docs/images/freelens-namespaces.png b/docs/images/freelens-namespaces.png new file mode 100644 index 0000000..6c8a8b1 Binary files /dev/null and b/docs/images/freelens-namespaces.png differ diff --git a/docs/images/freelens-nodes.png b/docs/images/freelens-nodes.png new file mode 100644 index 0000000..a0722c7 Binary files /dev/null and b/docs/images/freelens-nodes.png differ diff --git a/kubernetes/alloy-rbac/alloy-rbac.yaml b/kubernetes/alloy-rbac/alloy-rbac.yaml new file mode 100644 index 0000000..c351e52 --- /dev/null +++ b/kubernetes/alloy-rbac/alloy-rbac.yaml @@ -0,0 +1,36 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: alloy-external + namespace: monitoring +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: alloy-external +rules: +- apiGroups: [""] + resources: ["pods","pods/log","namespaces"] + verbs: ["get","list","watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: alloy-external +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: alloy-external +subjects: +- kind: ServiceAccount + name: alloy-external + namespace: monitoring +--- +apiVersion: v1 +kind: Secret +metadata: + name: alloy-external-token + namespace: monitoring + annotations: + kubernetes.io/service-account.name: alloy-external +type: kubernetes.io/service-account-token diff --git a/kubernetes/kube-state-metrics/kube-state-metrics.yaml b/kubernetes/kube-state-metrics/kube-state-metrics.yaml new file mode 100644 index 0000000..c0074d3 --- /dev/null +++ b/kubernetes/kube-state-metrics/kube-state-metrics.yaml @@ -0,0 +1,88 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: monitoring +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: kube-state-metrics + namespace: monitoring +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: kube-state-metrics +rules: +- apiGroups: [""] + resources: ["configmaps","secrets","nodes","pods","services","resourcequotas","replicationcontrollers","limitranges","persistentvolumeclaims","persistentvolumes","namespaces","endpoints"] + verbs: ["list","watch"] +- apiGroups: ["apps"] + resources: ["statefulsets","daemonsets","deployments","replicasets"] + verbs: ["list","watch"] +- apiGroups: ["batch"] + resources: ["cronjobs","jobs"] + verbs: ["list","watch"] +- apiGroups: ["autoscaling"] + resources: ["horizontalpodautoscalers"] + verbs: ["list","watch"] +- apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["list","watch"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: kube-state-metrics +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: kube-state-metrics +subjects: +- kind: ServiceAccount + name: kube-state-metrics + namespace: monitoring +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: kube-state-metrics + namespace: monitoring +spec: + replicas: 1 + selector: + matchLabels: + app: kube-state-metrics + template: + metadata: + labels: + app: kube-state-metrics + spec: + serviceAccountName: kube-state-metrics + containers: + - name: kube-state-metrics + image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.13.0 + ports: + - containerPort: 8080 + name: metrics + resources: + limits: + cpu: 100m + memory: 128Mi + requests: + cpu: 20m + memory: 32Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: kube-state-metrics + namespace: monitoring +spec: + type: NodePort + selector: + app: kube-state-metrics + ports: + - port: 8080 + targetPort: 8080 + nodePort: 30900 diff --git a/kubernetes/kube-state-metrics/prometheus-rbac.yaml b/kubernetes/kube-state-metrics/prometheus-rbac.yaml new file mode 100644 index 0000000..2785405 --- /dev/null +++ b/kubernetes/kube-state-metrics/prometheus-rbac.yaml @@ -0,0 +1,41 @@ +apiVersion: v1 +kind: ServiceAccount +metadata: + name: prometheus-external + namespace: monitoring +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: prometheus-external +rules: +- apiGroups: [""] + resources: ["nodes","nodes/proxy","nodes/metrics","services","endpoints","pods"] + verbs: ["get","list","watch"] +- apiGroups: [""] + resources: ["configmaps"] + verbs: ["get"] +- nonResourceURLs: ["/metrics"] + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: prometheus-external +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: prometheus-external +subjects: +- kind: ServiceAccount + name: prometheus-external + namespace: monitoring +--- +apiVersion: v1 +kind: Secret +metadata: + name: prometheus-external-token + namespace: monitoring + annotations: + kubernetes.io/service-account.name: prometheus-external +type: kubernetes.io/service-account-token diff --git a/kubernetes/resource-limits/resource-limits.yaml b/kubernetes/resource-limits/resource-limits.yaml new file mode 100644 index 0000000..4d61292 --- /dev/null +++ b/kubernetes/resource-limits/resource-limits.yaml @@ -0,0 +1,38 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: apps +--- +apiVersion: v1 +kind: LimitRange +metadata: + name: apps-limits + namespace: apps +spec: + limits: + - type: Container + default: + cpu: "200m" + memory: "256Mi" + defaultRequest: + cpu: "50m" + memory: "64Mi" + max: + cpu: "500m" + memory: "512Mi" + min: + cpu: "10m" + memory: "16Mi" +--- +apiVersion: v1 +kind: ResourceQuota +metadata: + name: apps-quota + namespace: apps +spec: + hard: + requests.cpu: "1" + requests.memory: "768Mi" + limits.cpu: "1.5" + limits.memory: "1.5Gi" + pods: "10"