diff --git a/README.md b/README.md index 19086df..7e3ff8b 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s | **Semaphore** | `docker-compose/semaphore/` | Self-hosted Ansible UI (alternative to AWX) — see [Semaphore Guide](docs/Semaphore.md) | | **Ansible playbooks** | `ansible/` | Shared update role (pull → recreate → health-check → prune) reused across every stack, run via Semaphore — see [Ansible Playbooks](docs/AnsiblePlaybooks.md) | | **GitLab CI/CD** | `docker-compose/gitlab/` | Self-hosted GitLab Omnibus + Docker-executor Runner, tuned for this hardware and integrated with the monitoring stack — see [GitLab CI/CD](docs/GitLab-CICD.md) | +| **Custom CI images** | `docker-compose/gitlab/ci-images-reference/` | Self-built, minimal Docker images for CI pipelines, stored in a self-hosted Container Registry, rebuilt biweekly — see [Custom CI Runner Images](docs/CI-Custom-Images.md) | ## Learning labs (not always-on) @@ -96,6 +97,12 @@ homelab/ │ ├── gitlab/ │ │ ├── docker-compose.yml │ │ ├── config.toml.example +│ │ ├── ci-images-reference/ +│ │ │ ├── README.md +│ │ │ ├── .gitlab-ci.yml +│ │ │ └── images/ +│ │ │ └── ssh-alpine/ +│ │ │ └── Dockerfile │ │ └── backups/ │ │ └── gitlab-backup.sh │ └── elk/ @@ -137,6 +144,7 @@ homelab/ │ ├── DockScope.md │ ├── AnsiblePlaybooks.md │ ├── GitLab-CICD.md +│ ├── CI-Custom-Images.md │ ├── ElasticStack.md │ └── images/ ├── dashboards/ @@ -181,6 +189,8 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - GitLab Omnibus's default RAM footprint assumes dedicated hardware — running it alongside an existing stack requires trimming Puma/Sidekiq workers explicitly, and disabling GitLab's own bundled monitoring in favor of the stack already running here. - `registry.gitlab.com` is geo-blocked under U.S. sanctions for several countries at the infrastructure level, independent of any account or token — surfaced as a Docker-executor helper-image pull failure. Fix was sourcing the helper image from Docker Hub instead. See [GitLab CI/CD](docs/GitLab-CICD.md). - Files written by a root-running container into a bind-mounted volume (GitLab's `config/`, `data/backups/`) are root-owned on the host — a backup script touching these needs to run as root and `chown` its own output back to the regular user, or every later interaction with the backup needs `sudo`. +- Tailscale actively manages `/etc/resolv.conf` on a host it's installed on, silently overriding local DNS resolvers (Pi-hole) even when `systemd-resolved` is configured correctly — worth checking `/etc/resolv.conf`'s own header before assuming a DNS setup is broken. See [Custom CI Runner Images](docs/CI-Custom-Images.md). +- GitLab's Container Registry ties every image path to a real, existing project — unlike Docker Hub, pushing to an arbitrary namespace that has no matching project fails with a generic-looking `denied: requested access` error that reads like an auth problem but isn't one. ## Roadmap diff --git a/docker-compose/gitlab/ci-images-reference/.gitlab-ci.yml b/docker-compose/gitlab/ci-images-reference/.gitlab-ci.yml new file mode 100644 index 0000000..da97a11 --- /dev/null +++ b/docker-compose/gitlab/ci-images-reference/.gitlab-ci.yml @@ -0,0 +1,24 @@ +stages: + - build + +.build-image: + stage: build + image: docker:27-cli + services: [] # not needed — we're using the host socket, not DinD + before_script: + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" + script: + - docker pull alpine:3.20 + - docker build --pull --no-cache -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" "images/$IMAGE_NAME" + - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" + - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" + +build-ssh-alpine: + extends: .build-image + variables: + IMAGE_NAME: ssh-alpine + +# Scheduled biweekly via GitLab's native Pipeline Schedules (Build > Pipeline +# schedules), cron pattern: 0 3 */14 * * +# Runs on both the schedule and regular pushes — no `rules` restriction — +# so editing a Dockerfile also triggers an immediate rebuild. diff --git a/docker-compose/gitlab/ci-images-reference/README.md b/docker-compose/gitlab/ci-images-reference/README.md new file mode 100644 index 0000000..742312d --- /dev/null +++ b/docker-compose/gitlab/ci-images-reference/README.md @@ -0,0 +1,5 @@ +# ci-images (reference copy) + +This folder mirrors the `.gitlab-ci.yml` and `Dockerfile`(s) from the `ci-images` project, which lives in the self-hosted GitLab instance (not GitHub) — kept here so the work is visible in this repo too. See [docs/CI-Custom-Images.md](../../../docs/CI-Custom-Images.md) for the full write-up. + +The live project has its own git history, pipeline runs, and pushes to the self-hosted Container Registry at `registry.gitlab.home` — this folder is documentation, not a working deployment target. diff --git a/docker-compose/gitlab/ci-images-reference/images/ssh-alpine/Dockerfile b/docker-compose/gitlab/ci-images-reference/images/ssh-alpine/Dockerfile new file mode 100644 index 0000000..673dc90 --- /dev/null +++ b/docker-compose/gitlab/ci-images-reference/images/ssh-alpine/Dockerfile @@ -0,0 +1,17 @@ +FROM alpine:3.20 + +RUN apk add --no-cache openssh-server && \ + ssh-keygen -A && \ + adduser -D -s /bin/ash ciuser && \ + mkdir -p /home/ciuser/.ssh && \ + chmod 700 /home/ciuser/.ssh && \ + chown -R ciuser:ciuser /home/ciuser/.ssh + +# Disable root login and password auth by default — key-based only. +# Override at container runtime if a specific pipeline job genuinely needs +# password auth (it shouldn't). +RUN sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && \ + sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config + +EXPOSE 22 +CMD ["/usr/sbin/sshd", "-D"] diff --git a/docs/CI-Custom-Images.md b/docs/CI-Custom-Images.md new file mode 100644 index 0000000..f01c66d --- /dev/null +++ b/docs/CI-Custom-Images.md @@ -0,0 +1,120 @@ +# Custom CI Runner Images + +Self-built, lightweight Docker images for use inside GitLab CI pipelines, stored in a self-hosted Container Registry and rebuilt on a biweekly schedule so they stay current with upstream security patches — rather than relying on generic public images pulled fresh (and un-vetted) in every job. + +Lives in its own project (`ci-images`) inside the self-hosted GitLab instance — see [GitLab CI/CD](GitLab-CICD.md) for the base GitLab/Runner setup this builds on. Reference copies of the project files are kept in this repo under [`docker-compose/gitlab/ci-images-reference/`](../docker-compose/gitlab/ci-images-reference/), since the live project itself isn't on GitHub. + +## Why a private registry instead of Docker Hub + +Two reasons, one practical and one already learned the hard way elsewhere in this repo: + +- The whole point of the exercise is a self-contained, self-hosted CI pipeline — pushing built images to a third-party registry defeats that. +- `registry.gitlab.com` was already ruled out for pulling *base* images earlier (see the sanctions block in [GitLab CI/CD](GitLab-CICD.md)) — no reason to introduce a similar external dependency here for images this repo controls end to end anyway. + +## Registry setup + +Enabled on the existing GitLab instance, fronted by its own subdomain (`registry.gitlab.home`) rather than a subpath — the standard pattern Docker registries expect, and cleaner to reason about in NPM: + +```yaml +registry['enable'] = true +registry_external_url 'https://registry.gitlab.home' +gitlab_rails['registry_enabled'] = true +gitlab_rails['registry_host'] = "registry.gitlab.home" +gitlab_rails['registry_port'] = "443" +registry_nginx['listen_port'] = 5050 +registry_nginx['listen_https'] = false +``` + +`registry_port = "443"` isn't the container's actual listen port — it's what GitLab tells clients to assume, since NPM terminates HTTPS on 443 and forwards to the container's internal `5050`. TLS itself (via `mkcert`) and the NPM proxy host follow the same pattern as the rest of this repo's [Reverse Proxy](Reverse-Proxy.md) setup. + +### DNS resolution gotcha: Tailscale silently overrides Pi-hole + +The new subdomain resolved fine from every other device on the LAN (all pointed at Pi-hole) but failed with `Could not resolve host` from the GitLab **host itself**. Cause: Tailscale manages `/etc/resolv.conf` directly on this host and overwrites it back to public resolvers (`8.8.8.8`/`1.1.1.1`) regardless of `systemd-resolved` configuration — the file even says so explicitly (`DO NOT EDIT THIS FILE BY HAND — CHANGES WILL BE OVERWRITTEN`). `resolvectl status` showed Pi-hole registered under one scope, but actual lookups still failed because Tailscale's override sat in front of it. + +Rather than fight Tailscale's DNS management host-wide (`tailscale set --accept-dns=false` didn't reliably stick, and trades away MagicDNS resolution for other tailnet devices), the practical fix was a static `/etc/hosts` entry — surgical, doesn't touch system-wide DNS behavior, and doesn't depend on Docker containers inheriting the host's resolver config either: + +``` +192.168.100.6 registry.gitlab.home +``` + +## The image: `ssh-alpine` + +```dockerfile +FROM alpine:3.20 + +RUN apk add --no-cache openssh-server && \ + ssh-keygen -A && \ + adduser -D -s /bin/ash ciuser && \ + mkdir -p /home/ciuser/.ssh && \ + chmod 700 /home/ciuser/.ssh && \ + chown -R ciuser:ciuser /home/ciuser/.ssh + +RUN sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && \ + sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config + +EXPOSE 22 +CMD ["/usr/sbin/sshd", "-D"] +``` + +Alpine base plus `--no-cache` on `apk add` keeps this in the 10–15MB range versus 300MB+ for a Debian/Ubuntu-based equivalent — directly targeting the "keep images small" requirement. Root login and password auth are disabled by default (key-based only), since this is meant to run inside CI jobs, not be a general-purpose SSH box. + +## Build pipeline + +```yaml +stages: + - build + +.build-image: + stage: build + image: docker:27-cli + services: [] # host-socket build, not Docker-in-Docker + before_script: + - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY" + script: + - docker pull alpine:3.20 + - docker build --pull --no-cache -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" "images/$IMAGE_NAME" + - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" + - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" + +build-ssh-alpine: + extends: .build-image + variables: + IMAGE_NAME: ssh-alpine +``` + +Design notes: + +- **Host-socket build, not DinD** — `services: []` is explicit, not an omission. Job containers build against the host's existing Docker daemon via the same socket-mount pattern already used for the Runner itself, avoiding a second nested daemon's RAM overhead. Required adding the socket mount to `[runners.docker].volumes` in `config.toml`, not just the Runner's own container. +- **`--pull --no-cache` is load-bearing, not decoration.** Without it, a scheduled rebuild could silently reuse cached layers and produce byte-identical output even if Alpine shipped a security patch upstream — defeating the entire point of a periodic rebuild. `--pull` forces a fresh base-image check; `--no-cache` forces every `RUN` step (including the `apk add`) to actually execute again. +- **Dual tagging** (`:latest` + `:$CI_COMMIT_SHORT_SHA`) gives a rollback point if a rebuild ever breaks something — pin to the last known-good SHA instead of `:latest`. +- **`.build-image` + `extends`** means adding the next custom image later is a 4-line addition, not a duplicated block. +- **`$CI_REGISTRY*` variables are auto-injected by GitLab** once the registry is enabled for the project — no manual secret configuration needed. + +## Scheduling + +GitLab's native **Pipeline Schedules** (Build → Pipeline schedules), not a host-level cron job — keeps the schedule visible and manageable from inside GitLab itself rather than another config file to remember exists. + +``` +0 3 */14 * * +``` + +Cron has no native "every 2 weeks" unit; `*/14` on the day-of-month field approximates it (drifts slightly across months of different lengths — irrelevant for a background rebuild). No `rules`/`only` restriction was added to the build job, so it runs on both the schedule *and* regular pushes — intentional, since editing the Dockerfile should also trigger a rebuild, not just wait for the next scheduled run. + +## Registry cleanup policy + +Without this, the registry grows unbounded — `:latest` gets overwritten each run, but every `:$CI_COMMIT_SHORT_SHA` tag is unique and permanent by design, and a biweekly schedule running indefinitely means indefinite tag accumulation on a host with limited free disk space. + +- Keep the most recent: **10 tags** +- Remove tags older than: **30 days** +- Keep tags matching: *(intentionally empty — see note below)* +- Remove tags matching: `.*` + +**Gotcha worth flagging**: "Keep tags matching" is a regex field, not a numeric "keep top N" setting (that's the separate "Keep the most recent" field above it). Typing a bare digit like `3` here means "protect any tag whose name contains the character 3" — since commit-SHA tags are hex strings, most of them contain at least one digit 0–9 essentially by chance, which silently protects nearly everything and makes the whole cleanup policy close to a no-op. Left blank unless there's a genuine need to protect specific tag patterns from pruning. + +## Lessons learned + +- GitLab's Container Registry ties every image path to an actual existing project — unlike Docker Hub, you can't push to an arbitrary namespace/path that doesn't correspond to a real project. A `denied: requested access` error on a first push is often this, not a credentials problem. +- Tailscale actively manages and overwrites `/etc/resolv.conf` on a host it's installed on, silently overriding local DNS resolvers like Pi-hole even when `systemd-resolved` is configured correctly — worth checking `/etc/resolv.conf`'s own header comment before assuming a DNS config is wrong. +- A single mistyped character in a TOML config (`1O` instead of `10`, a duplicated `url = ` fragment) crashes the Runner container on every startup with a parse error at the *first* invalid line — the resulting symptom (runner shows "hasn't contacted GitLab in N hours") looks like a networking/DNS problem, not a config typo, unless the container logs are actually checked. +- A Docker build cache will happily produce an identical image on a scheduled rebuild even when the upstream base image has genuinely changed, unless `--pull` and `--no-cache` are both used — "rebuild on a schedule" and "rebuild with actual updates" are not the same thing without this. +- GitLab's registry cleanup "Keep tags matching" field is a regex, not a count — a bare number typed into it matches far more tags than intended and can silently neutralize the cleanup policy. diff --git a/docs/GitLab-CICD.md b/docs/GitLab-CICD.md index d0b7067..f22fc4b 100644 --- a/docs/GitLab-CICD.md +++ b/docs/GitLab-CICD.md @@ -31,7 +31,7 @@ gitlab_exporter['enable'] = false Note: `grafana['enable']` is **not** a valid key on current Omnibus versions — GitLab dropped bundled Grafana entirely (not just disabled it), and setting this key causes a hard `reconfigure` failure (`Reading unsupported config value grafana`) rather than a silent no-op. Worth knowing if following an older guide. -Container Registry is left disabled (`registry['enable'] = false`) — no current need for `docker build`/`push` in CI, and it's a one-line flip later if that changes. +Container Registry was initially left disabled, then enabled once a real use case existed — see [Custom CI Runner Images](CI-Custom-Images.md), which builds and stores self-built Docker images for use inside pipelines rather than relying on generic public images. ![GitLab instance overview](images/gitlab-instance-overview.png) diff --git a/docs/Home.md b/docs/Home.md index d550c16..42ece26 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -17,6 +17,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This - [DockScope](DockScope.md) - [Ansible Playbooks](AnsiblePlaybooks.md) - [GitLab CI/CD](GitLab-CICD.md) +- [Custom CI Runner Images](CI-Custom-Images.md) - [Elastic Stack (ELK) — Lab](ElasticStack.md) ## At a glance