From 1c5997246a24fb4a44122488e216a3501305346c Mon Sep 17 00:00:00 2001 From: rezayaghobi-dev Date: Sun, 26 Jul 2026 10:38:26 +0330 Subject: [PATCH] added loki to our monitoring stack and updated the docs --- README.md | 19 +++++++-- .../config.alloy | 40 ++++++++++++++++++ .../docker-compose.yml | 42 +++++++++++++++++-- .../loki.yml | 9 ++++ docs/Home.md | 2 +- docs/Monitoring-Stack.md | 19 +++++++-- 6 files changed, 120 insertions(+), 11 deletions(-) create mode 100644 docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/config.alloy create mode 100644 docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/loki.yml diff --git a/README.md b/README.md index cf8e3f1..9b6f26c 100644 --- a/README.md +++ b/README.md @@ -8,6 +8,7 @@ A self-hosted homelab running on Docker Compose, built and maintained on an Ubun ![Ansible](https://img.shields.io/badge/Ansible-EE0000?style=flat&logo=ansible&logoColor=white) ![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?style=flat&logo=prometheus&logoColor=white) ![Grafana](https://img.shields.io/badge/Grafana-F46800?style=flat&logo=grafana&logoColor=white) +![Loki](https://img.shields.io/badge/Loki-F5A623?style=flat&logo=grafana&logoColor=white) ## Overview @@ -17,7 +18,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s | Service | Folder | Purpose | |---|---|---| -| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics collection, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics | +| **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, plus Loki + Grafana Alloy for centralized container log aggregation — see [Monitoring Stack](docs/Monitoring-Stack.md) | | **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking | | **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host | | **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) | @@ -39,7 +40,7 @@ Not everything in this repo runs permanently. Some folders are kept purely as re | Lab | Folder | Purpose | |---|---|---| -| **Elastic Stack (ELK)** | `docker-compose/elk/` | Elasticsearch + Kibana, security-enabled, sized for this hardware — a modern-workflow (Elastic Agent/Fleet) alternative to an older Filebeat/Metricbeat-based course reference. Not run continuously since Prometheus/Grafana already cover monitoring here — see [Elastic Stack (ELK)](docs/ElasticStack.md) | +| **Elastic Stack (ELK)** | `docker-compose/elk/` | Elasticsearch + Kibana, security-enabled, sized for this hardware — a modern-workflow (Elastic Agent/Fleet) alternative to an older Filebeat/Metricbeat-based course reference. Not run continuously since Loki now covers centralized logging here — see [Elastic Stack (ELK)](docs/ElasticStack.md) | ## Repository structure @@ -47,7 +48,15 @@ Not everything in this repo runs permanently. Some folders are kept purely as re homelab/ ├── docker-compose/ │ ├── monitoring-grafana-promethues-cadvisor-node-exporter/ -│ │ └── docker-compose.yml +│ │ ├── docker-compose.yml +│ │ ├── alloy/ +│ │ │ └── config.alloy +│ │ ├── grafana/ +│ │ │ └── provisioning/ +│ │ │ └── datasources/ +│ │ │ └── loki.yml +│ │ └── prometheus/ +│ │ └── prometheus.yml │ ├── pihole/ │ │ ├── docker-compose.yml │ │ └── .env.example @@ -146,6 +155,7 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - Passwords referenced in compose files are injected via environment variables, never hardcoded. - The AI coding agent (`docker-compose/openhands/`) currently uses a third-party API aggregator as its LLM backend — see [AI Coding Agent](docs/AI-Agent.md) for details and caveats. It has no access to the Docker socket or other containers on the host. - **DockScope** (`docker-compose/dockscope/`) mounts `/var/run/docker.sock`, the same as Portainer — this grants it effectively root-equivalent access to the Docker host, and it currently has no built-in authentication on its web UI, so it's kept LAN-only and not exposed through the reverse proxy. See [DockScope](docs/DockScope.md). +- **Grafana Alloy** (part of the monitoring stack) also mounts `/var/run/docker.sock` read-only, needed to discover containers and tail their logs. ## Lessons learned @@ -156,6 +166,9 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - Wildcard TLS certs (`*.home`) are rejected by browsers on single-label local domains — certificates for local reverse proxies need explicit hostnames listed instead. See [Reverse Proxy](docs/Reverse-Proxy.md). - Grouping all Compose stacks under one `docker-compose/` folder (rather than flat at repo root) kept the structure readable once Ansible playbooks were added as a sibling — a flat repo mixing deployment configs and automation code got confusing fast. - Newer Docker images can silently raise the minimum CPU generation required (base-OS changes, not anything in the compose file itself) — hit this running Elasticsearch 9.x on older hardware. See [Elastic Stack (ELK)](docs/ElasticStack.md). +- **Promtail is end-of-life as of March 2026** and removed outright as of Loki 3.7.3 — any current Loki setup should use Grafana Alloy for log shipping instead. Worth checking a project's actual current-recommended tooling before following an older tutorial, since agent names and workflows in this space have moved fast. +- Bind-mounted container data directories can end up owned by the wrong UID after mixing `sudo` and non-`sudo` commands on the same folder — this silently broke Loki's ability to write its own working directory, surfacing only as a `permission denied` deep in its startup logs. +- Community Grafana dashboards built for older Loki/Promtail label conventions (e.g. `container_name`) don't automatically work against a fresh Alloy-based label schema (`container`, `service_name`) — worth checking a dashboard's expected labels against your actual pipeline's labels before assuming an import error means something's broken on your end. ## Roadmap diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/config.alloy b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/config.alloy new file mode 100644 index 0000000..1dd7296 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/config.alloy @@ -0,0 +1,40 @@ +// Discover all running Docker containers +discovery.docker "containers" { + host = "unix:///var/run/docker.sock" +} + +// Extract a clean "container" label from the container name +discovery.relabel "containers" { + targets = discovery.docker.containers.targets + + rule { + source_labels = ["__meta_docker_container_name"] + regex = "/(.*)" + target_label = "container" + } + + rule { + source_labels = ["__meta_docker_container_name"] + regex = "/(.*)" + target_label = "container_name" + } + + rule { + target_label = "instance" + replacement = constants.hostname + } +} + +// Tail logs from every discovered container +loki.source.docker "containers" { + host = "unix:///var/run/docker.sock" + targets = discovery.relabel.containers.output + forward_to = [loki.write.default.receiver] +} + +// Ship to the local Loki instance +loki.write "default" { + endpoint { + url = "http://loki:3100/loki/api/v1/push" + } +} diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml index a2b3ac4..bb3c480 100644 --- a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/docker-compose.yml @@ -16,6 +16,7 @@ services: networks: - monitoring + grafana: image: grafana/grafana:latest container_name: grafana @@ -24,6 +25,7 @@ services: - "3000:3000" volumes: - ./grafana/data:/var/lib/grafana + - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro networks: - monitoring @@ -54,10 +56,42 @@ services: networks: - monitoring + loki: + image: grafana/loki:3.7.3 + container_name: loki + restart: unless-stopped + command: -config.file=/etc/loki/local-config.yaml + ports: + - "3100:3100" + volumes: + - ./data/loki:/loki + mem_limit: 400m + cpus: 1 + networks: + - monitoring + + alloy: + image: grafana/alloy:v1.17.1 + container_name: alloy + restart: unless-stopped + command: + - "run" + - "--server.http.listen-addr=0.0.0.0:12345" + - "--storage.path=/var/lib/alloy/data" + - "/etc/alloy/config.alloy" + ports: + - "12345:12345" + volumes: + - ./alloy/config.alloy:/etc/alloy/config.alloy:ro + - /var/run/docker.sock:/var/run/docker.sock:ro + - ./data/alloy:/var/lib/alloy/data + mem_limit: 300m + cpus: 0.5 + depends_on: + - loki + networks: + - monitoring + networks: monitoring: driver: bridge - - - - diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/loki.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/loki.yml new file mode 100644 index 0000000..67ce8b7 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/loki.yml @@ -0,0 +1,9 @@ +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://loki:3100 + isDefault: false + editable: false diff --git a/docs/Home.md b/docs/Home.md index bc40d18..8f1a1dd 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -25,7 +25,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This | **Host** | Ubuntu Server | | **Orchestration** | Docker Compose, one stack per service | | **Management** | Portainer, **DockScope** (3D visual dashboard) | -| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter | +| **Monitoring** | Prometheus + Grafana + cAdvisor + node-exporter + **Loki + Alloy** (centralized logs) | | **Network** | Pi-hole (DNS filtering), Samba (file sharing), nginx (proxy managing) | | **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent | | **Disk health** | Hard Disk Sentinel | diff --git a/docs/Monitoring-Stack.md b/docs/Monitoring-Stack.md index d331ffa..9cb45b7 100644 --- a/docs/Monitoring-Stack.md +++ b/docs/Monitoring-Stack.md @@ -1,13 +1,15 @@ # Monitoring Stack -Prometheus, Grafana, cAdvisor, and node-exporter running together give full visibility into both the host and every container on it — essential on modest hardware where knowing what's consuming resources matters. +Prometheus, Grafana, cAdvisor, and node-exporter running together give full visibility into both the host and every container on it — essential on modest hardware where knowing what's consuming resources matters. Loki and Grafana Alloy extend this from metrics into logs, so container output is searchable in the same place instead of living only in `docker logs`. ## Components - **Prometheus** — scrapes and stores time-series metrics from every target below - **node-exporter** — exposes host-level metrics (CPU, memory, disk, network) - **cAdvisor** — exposes per-container resource usage -- **Grafana** — dashboards on top of Prometheus data +- **Grafana** — dashboards on top of Prometheus and Loki data +- **Loki** — stores and indexes logs by label, not full text, keeping it lightweight +- **Grafana Alloy** — discovers every running container and ships its logs to Loki ## Prometheus targets @@ -21,6 +23,17 @@ Host-level metrics — CPU, memory, swap, disk, and network — over a rolling 2 ![Grafana Node Exporter dashboard](images/grafana-node-exporter.png) +## Centralized logging with Loki + Alloy + +Every running container's logs get discovered and tagged automatically — no per-service logging config needed anywhere else in this repo. Alloy watches the Docker socket, picks up new containers as they start, and labels each log stream with `container`, `container_name`, `service_name`, and `instance` before forwarding to Loki. + +Note: this uses **Grafana Alloy**, not Promtail — Promtail reached end-of-life in March 2026 and was removed outright as of Loki 3.7.3, with its functionality folded into Alloy. Any current Loki setup should be built on Alloy rather than following older Promtail-based tutorials. + +Logs are browsable two ways in Grafana: + +- **Explore → Loki**, querying by label directly (e.g. `{container="grafana"}`) +- **Drilldown → Logs**, a zero-config log browser bundled with Grafana — no dashboard needed to start exploring + ## Why this matters -On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. +On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.