# Homelab A self-hosted homelab running on Docker Compose, built and maintained on an Ubuntu Server. This repo documents the actual infrastructure-as-code behind it — deployment configs only, no runtime data or secrets. ![Docker](https://img.shields.io/badge/Docker-2496ED?style=flat&logo=docker&logoColor=white) ![Docker Compose](https://img.shields.io/badge/Docker_Compose-2496ED?style=flat&logo=docker&logoColor=white) ![Ubuntu](https://img.shields.io/badge/Ubuntu_Server-E95420?style=flat&logo=ubuntu&logoColor=white) ![Ansible](https://img.shields.io/badge/Ansible-EE0000?style=flat&logo=ansible&logoColor=white) ![Prometheus](https://img.shields.io/badge/Prometheus-E6522C?style=flat&logo=prometheus&logoColor=white) ![Grafana](https://img.shields.io/badge/Grafana-F46800?style=flat&logo=grafana&logoColor=white) ![Loki](https://img.shields.io/badge/Loki-F5A623?style=flat&logo=grafana&logoColor=white) ![Kubernetes](https://img.shields.io/badge/Kubernetes-326CE5?style=flat&logo=kubernetes&logoColor=white) ![K3s](https://img.shields.io/badge/K3s-FFC61C?style=flat&logo=k3s&logoColor=white) ![Vagrant](https://img.shields.io/badge/Vagrant-1868F2?style=flat&logo=vagrant&logoColor=white) ![GitLab](https://img.shields.io/badge/GitLab-FC6D26?style=flat&logo=gitlab&logoColor=white) ## Overview This is a real, running homelab — not a demo. It's deployed on a home Ubuntu server and managed entirely through Docker Compose, with each service isolated in its own folder, and updated through a shared Ansible role run via Semaphore. Configs here are the actual files running in production, with all secrets, API keys, and runtime state stripped out and replaced with `.env.example` placeholders. ## Services | Service | Folder | Purpose | |---|---|---| | **Monitoring stack** | `docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/` | Prometheus for metrics, Grafana for dashboards, cAdvisor for per-container resource stats, node-exporter for host-level metrics, plus Loki + Grafana Alloy for centralized container log aggregation — see [Monitoring Stack](docs/Monitoring-Stack.md) | | **Pi-hole** | `docker-compose/pihole/` | Network-wide DNS-level ad and tracker blocking | | **Portainer** | `docker-compose/portainer/` | Web UI for managing and monitoring all Docker containers on the host | | **DockScope** | `docker-compose/dockscope/` | 3D visual dashboard mapping containers, networks, and dependencies, with anomaly detection and crash diagnostics — see [DockScope](docs/DockScope.md) | | **n8n** | `docker-compose/n8n/` | Workflow automation engine, backed by Postgres | | **Samba** | `docker-compose/samba/` | Network file sharing across LAN devices | | **Filebrowser** | `docker-compose/filebrowser/` | Web-based file manager for the server's storage | | **Media stack** | `docker-compose/mediastack/` | Plex (media server), Sonarr/Radarr/Prowlarr (media automation and indexing), qBittorrent (download client) | | **Minecraft** | `docker-compose/minecraft/` | Self-hosted game server | | **Sentinel** | `docker-compose/sentinel/` | Disk health monitoring integrated with smartctl, dumps output to container logs | | **smartctl** | `docker-compose/smartctl/` | Disk health monitoring (S.M.A.R.T. data) | | **OpenHands** | `docker-compose/openhands/` | Self-hosted AI coding agent (OpenHands/agent-canvas) backed by Ollama — see [AI Coding Agent](docs/AI-Agent.md) | | **Traefik** | `docker-compose/traefik/` | Reverse proxy with Docker daemon auto-discovery and full config-as-code — replaces Nginx Proxy Manager — see [Traefik](docs/Traefik.md) | | **Vaultwarden** | `docker-compose/vaultwarden/` | Self-hosted Bitwarden-compatible password manager — see [Vaultwarden](docs/Vaultwarden.md) | | **Semaphore** | `docker-compose/semaphore/` | Self-hosted Ansible UI (alternative to AWX) — see [Semaphore Guide](docs/Semaphore.md) | | **Ansible playbooks** | `ansible/update-playbooks/` | Shared update role (pull → recreate → health-check → prune) reused across every stack, run via Semaphore — see [Ansible Playbooks](docs/AnsiblePlaybooks.md) | | **Server provisioning** | `ansible/preparing-playbook/` | Full server preparation playbook (packages, hardening, Docker, Nexus/Traefik) tested against Vagrant first — see [Ansible README](ansible/README.md) | | **GitLab CI/CD** | `docker-compose/gitlab/` | Self-hosted GitLab Omnibus + Docker-executor Runner, tuned for this hardware and integrated with the monitoring stack — see [GitLab CI/CD](docs/GitLab-CICD.md) | | **Custom CI images** | `docker-compose/gitlab/ci-images-reference/` | Self-built, minimal Docker images for CI pipelines, stored in a self-hosted Container Registry, rebuilt biweekly — see [Custom CI Runner Images](docs/CI-Custom-Images.md) | | **Voting app CI/CD** | `voting-app-monorepo` (self-hosted GitLab, referenced here) | Full pipeline: build, test, vulnerability scan, 3-environment deploy with rollback, load test, DB backup to MinIO — see [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md) | | **Object storage** | `docker-compose/minio/` | Self-hosted MinIO (S3-compatible), `-cpuv1` release tag for CPU compatibility — backup target for the voting app's Postgres dumps | | **K3s cluster** | `kubernetes/` | Single-node K3s cluster running alongside the Docker Compose stack, fronted by the same reverse proxy via NodePort, monitored through the same Prometheus/Grafana/Loki stack — see [K3s](docs/K3s.md) | ## Learning labs (not always-on) Not everything in this repo runs permanently. Some folders are kept purely as reference from one-off learning exercises for my DevOps cert coursework — the compose files work, but they're not part of the day-to-day stack above. | Lab | Folder | Purpose | |---|---|---| | **Nginx Proxy Manager** | `docker-compose/nginx-proxy-manager/` | ~~Retired~~ — replaced by Traefik for config-as-code reverse proxying. See [Traefik](docs/Traefik.md) for the migration rationale and [Reverse Proxy](docs/Reverse-Proxy.md) for the original NPM setup | | **Elastic Stack (ELK)** | `docker-compose/ELK/` | Elasticsearch + Kibana, security-enabled, sized for this hardware — a modern-workflow (Elastic Agent/Fleet) alternative to an older Filebeat/Metricbeat-based course reference. Not run continuously since Loki now covers centralized logging here — see [Elastic Stack (ELK)](docs/ElasticStack.md) | | **Vagrant test VM** | `vagrant/` | Disposable Debian 12 VirtualBox VM (`bento/debian-12`) on a private-network IP, for testing provisioning scripts and configuration changes before touching the real host | ## Repository structure ``` homelab/ ├── docker-compose/ │ ├── monitoring-grafana-promethues-cadvisor-node-exporter/ │ │ ├── docker-compose.yml │ │ ├── alloy/ │ │ │ └── config.alloy │ │ ├── grafana/ │ │ │ └── provisioning/ │ │ │ └── datasources/ │ │ │ └── loki.yml │ │ └── prometheus/ │ │ └── prometheus.yml │ ├── pihole/ │ │ ├── docker-compose.yml │ │ └── .env.example │ ├── portainer/ │ │ └── docker-compose.yml │ ├── dockscope/ │ │ └── docker-compose.yml │ ├── n8n/ │ │ ├── docker-compose.yaml │ │ └── .env.example │ ├── samba/ │ │ ├── docker-compose.yml │ │ └── .env.example │ ├── filebrowser/ │ │ └── docker-compose.yml │ ├── mediastack/ │ │ ├── plex/docker-compose.yml │ │ ├── sonarr/docker-compose.yml │ │ ├── radarr/docker-compose.yml │ │ ├── prowlarr/docker-compose.yml │ │ ├── whisparr/docker-compose.yml │ │ └── download-client-qbittorent/docker-compose.yml │ ├── minecraft/ │ │ └── docker-compose.yml │ ├── minio/ │ │ └── docker-compose.yml │ ├── sentinel/ │ │ └── docker-compose.yml │ ├── smartctl/ │ │ └── docker-compose.yml │ ├── openhands/ │ │ └── docker-compose.yaml │ ├── traefik/ │ │ ├── compose.yaml │ │ ├── README.md │ │ ├── certs/ │ │ │ ├── home.server.crt │ │ │ └── home.server.key │ │ └── dynamic/ │ │ ├── tls.yaml │ │ ├── middlewares.yaml │ │ ├── pihole.yaml │ │ ├── plex.yml │ │ ├── sentinel.yml │ │ └── dockscope-auth.yml │ ├── vaultwarden/ │ │ ├── docker-compose.yml │ │ └── vw-data/ │ ├── nginx-proxy-manager/ │ │ └── compose.yaml │ ├── semaphore/ │ │ ├── docker-compose.yml │ │ └── README.md │ ├── gitlab/ │ │ ├── docker-compose.yml │ │ ├── config.toml.example │ │ ├── ci-images-reference/ │ │ │ ├── README.md │ │ │ ├── .gitlab-ci.yml │ │ │ └── images/ │ │ │ └── ssh-alpine/ │ │ │ └── Dockerfile │ │ └── backups/ │ │ └── gitlab-backup.sh │ └── ELK/ │ ├── docker-compose.yml │ └── .env.example ├── ansible/ │ ├── README.md │ ├── update-playbooks/ │ │ ├── roles/ │ │ │ └── docker_compose_update/ │ │ │ └── tasks/main.yml │ │ ├── update-filebrowser.yml │ │ ├── update-n8n.yml │ │ ├── update-npm.yml │ │ ├── update-pihole.yml │ │ ├── update-plex.yml │ │ ├── update-portainer.yml │ │ ├── update-prowlarr.yml │ │ ├── update-radarr.yml │ │ ├── update-samba.yml │ │ ├── update-sentinel.yml │ │ ├── update-smartctl.yml │ │ ├── update-sonarr.yml │ │ ├── update-whisparr.yml │ │ ├── update-agent-canvas.yml │ │ ├── update-grafana.yml │ │ └── ping.yml │ └── preparing-playbook/ │ └── reza-ansible/ │ ├── ansible.cfg │ ├── preparing.yaml │ ├── repo-server.yaml │ ├── inventory/ │ │ ├── host.yaml │ │ ├── group_vars/all/ │ │ └── host_vars/vagrant-test.yaml │ └── roles/ │ ├── preparing_server/ │ ├── docker/ │ └── repo-server/ ├── docs/ │ ├── Home.md │ ├── Architecture-and-Hardware.md │ ├── Monitoring-Stack.md │ ├── K3s.md │ ├── Network-Services.md │ ├── Media-Automation.md │ ├── Workflow-Automation.md │ ├── Storage-Health.md │ ├── AI-Agent.md │ ├── Reverse-Proxy.md │ ├── Lessons-Learned.md │ ├── Semaphore.md │ ├── DockScope.md │ ├── AnsiblePlaybooks.md │ ├── GitLab-CICD.md │ ├── CI-Custom-Images.md │ ├── Voting-App-Monorepo.md │ ├── ElasticStack.md │ └── images/ ├── kubernetes/ │ ├── alloy-rbac/ │ │ └── alloy-rbac.yaml │ ├── kube-state-metrics/ │ │ ├── kube-state-metrics.yaml │ │ └── prometheus-rbac.yaml │ └── resource-limits/ │ └── resource-limits.yaml ├── dashboards/ │ ├── gitlab-dashboards/ │ │ ├── dashboards-README.md │ │ └── gitlab-application-runner-metrics.json │ └── kubernetes-dashboards/ │ ├── README.md │ ├── dashboard-1786902530431.json │ ├── dashboard-1786902563692.json │ ├── dashboard-1786902586059.json │ └── dashboard-1786902623268.json ├── vagrant/ │ ├── Vagrantfile │ ├── bootstrap.sh │ └── README.md └── .gitignore ``` ## Running a service Each folder is a standalone Compose stack: ```bash cd docker-compose/pihole cp .env.example .env # fill in real values docker compose up -d ``` ## Updating a service Rather than pulling and recreating manually, every always-on stack has a matching Ansible playbook that pulls the latest images, recreates containers, verifies they're healthy, and prunes unused images — run through Semaphore's web UI. See [Ansible Playbooks](docs/AnsiblePlaybooks.md) for details on how the shared role works. ## Security notes - All `.env` files, runtime config directories, and databases are excluded via `.gitignore` — only deployment definitions (`docker-compose.yml`) and `.env.example` placeholders are tracked. - Passwords referenced in compose files are injected via environment variables, never hardcoded. Long passwords (30+ chars) are generated and stored in Vaultwarden, referenced via `.env` files on the host. - The AI coding agent (`docker-compose/openhands/`) currently uses a third-party API aggregator as its LLM backend — see [AI Coding Agent](docs/AI-Agent.md) for details and caveats. It has no access to the Docker socket or other containers on the host. - **DockScope** (`docker-compose/dockscope/`) mounts `/var/run/docker.sock`, the same as Portainer — this grants it effectively root-equivalent access to the Docker host. It's protected by a Traefik basicAuth middleware and kept LAN-only. See [DockScope](docs/DockScope.md). - **Grafana Alloy** (part of the monitoring stack) also mounts `/var/run/docker.sock` read-only, needed to discover containers and tail their logs. ## Lessons learned - Splitting each service into its own folder with its own Compose file made it much easier to reason about dependencies and update services independently, instead of one giant Compose file. - Runtime config/state (API keys, session tokens, library databases) needs to be gitignored at the directory level (`**/config/`, `**/data/`) — a plain text/secret grep alone isn't enough, since it silently skips over app-specific binary and XML config files. - Centralized monitoring (Prometheus + Grafana + cAdvisor + node-exporter) made it possible to actually see resource usage per container, which mattered a lot on lower-spec hardware. - Free-tier cloud AI APIs enforce regional access restrictions automatically — worth checking a provider's terms before building around "free tier" as an assumption. See [AI Coding Agent](docs/AI-Agent.md). - Wildcard TLS certs (`*.home`) are rejected by browsers on single-label local domains — certificates for local reverse proxies need explicit hostnames listed instead. See [Reverse Proxy](docs/Reverse-Proxy.md). - Grouping all Compose stacks under one `docker-compose/` folder (rather than flat at repo root) kept the structure readable once Ansible playbooks were added as a sibling — a flat repo mixing deployment configs and automation code got confusing fast. - Newer Docker images can silently raise the minimum CPU generation required (base-OS changes, not anything in the compose file itself) — hit this running Elasticsearch 9.x on older hardware. See [Elastic Stack (ELK)](docs/ElasticStack.md). - **Promtail is end-of-life as of March 2026** and removed outright as of Loki 3.7.3 — any current Loki setup should use Grafana Alloy for log shipping instead. Worth checking a project's actual current-recommended tooling before following an older tutorial, since agent names and workflows in this space have moved fast. - Bind-mounted container data directories can end up owned by the wrong UID after mixing `sudo` and non-`sudo` commands on the same folder — this silently broke Loki's ability to write its own working directory, surfacing only as a `permission denied` deep in its startup logs. - Community Grafana dashboards built for older Loki/Promtail label conventions (e.g. `container_name`) don't automatically work against a fresh Alloy-based label schema (`container`, `service_name`) — worth checking a dashboard's expected labels against your actual pipeline's labels before assuming an import error means something's broken on your end. - GitLab Omnibus's default RAM footprint assumes dedicated hardware — running it alongside an existing stack requires trimming Puma/Sidekiq workers explicitly, and disabling GitLab's own bundled monitoring in favor of the stack already running here. - `registry.gitlab.com` is geo-blocked under U.S. sanctions for several countries at the infrastructure level, independent of any account or token — surfaced as a Docker-executor helper-image pull failure. Fix was sourcing the helper image from Docker Hub instead. See [GitLab CI/CD](docs/GitLab-CICD.md). - Files written by a root-running container into a bind-mounted volume (GitLab's `config/`, `data/backups/`) are root-owned on the host — a backup script touching these needs to run as root and `chown` its own output back to the regular user, or every later interaction with the backup needs `sudo`. - Tailscale actively manages `/etc/resolv.conf` on a host it's installed on, silently overriding local DNS resolvers (Pi-hole) even when `systemd-resolved` is configured correctly — worth checking `/etc/resolv.conf`'s own header before assuming a DNS setup is broken. See [Custom CI Runner Images](docs/CI-Custom-Images.md). - GitLab's Container Registry ties every image path to a real, existing project — unlike Docker Hub, pushing to an arbitrary namespace that has no matching project fails with a generic-looking `denied: requested access` error that reads like an auth problem but isn't one. - Docker images with `ENTRYPOINT` hardcoded to the tool itself (Trivy, k6, MinIO's `mc`) break GitLab CI's default script invocation with a confusing "unknown command sh" error — fixed the same way every time with `entrypoint: [""]`. See [Voting App — Mono-Repo](docs/Voting-App-Monorepo.md). - In Docker-outside-of-Docker setups, bind mounts (`-v host:container`) always resolve against the **host's** filesystem regardless of which container issued the `docker run` command — `docker cp` is the correct way to move a file into a container across that boundary, not a shared path assumption. - MinIO's current images require `x86-64-v2` CPU instructions and crash outright on older hardware (Phenom II here) with a `Fatal glibc error` — fixed by pinning to a `-cpuv1` release tag, a legacy compatibility branch MinIO still publishes for exactly this case. - `kube-state-metrics`'s default ClusterRole doesn't cover every resource it tries to watch (`NetworkPolicy`, `MutatingWebhookConfiguration`, `ValidatingWebhookConfiguration`, and more) — surfaces as repeated `reflector.go` `Failed to list ... is forbidden` warnings rather than a single clear error. Fixed with a supplementary `ClusterRole`/`ClusterRoleBinding` granting just the missing `list`/`watch` verbs. See [K3s](docs/K3s.md). - A pod doesn't pick up a widened RBAC grant on its own — its informers already failed and cached that state. `kubectl rollout restart` is required after any `ClusterRole` change for an already-running pod to actually benefit from it. - Old log lines don't disappear from Grafana/Loki after a fix ships — they just age out of whatever time window the dashboard is querying. A dropping "Errors total" over a few minutes is the real signal that a fix worked, not an empty panel immediately after applying it. - Migrating from a UI-based reverse proxy (Nginx Proxy Manager) to a config-as-code one (Traefik) forced a cleanup of every hardcoded password in every compose file — the migration was the catalyst for adopting Vaultwarden and proper secret management across the stack. See [Traefik](docs/Traefik.md) and [Vaultwarden](docs/Vaultwarden.md). ## Roadmap - [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md) - [ ] Auto-deploy via Watchtower on image updates - [ ] Expand monitoring with alerting rules ## Author **Reza** — Computer Networks student, LPIC-1 [GitHub](https://github.com/rezayaghobi-dev)