homelab/docs/CI-Custom-Images.md

Custom CI Runner Images

Self-built, lightweight Docker images for use inside GitLab CI pipelines, stored in a self-hosted Container Registry and rebuilt on a biweekly schedule so they stay current with upstream security patches — rather than relying on generic public images pulled fresh (and un-vetted) in every job.

Lives in its own project (ci-images) inside the self-hosted GitLab instance — see GitLab CI/CD for the base GitLab/Runner setup this builds on. Reference copies of the project files are kept in this repo under docker-compose/gitlab/ci-images-reference/, since the live project itself isn't on GitHub.

Why a private registry instead of Docker Hub

Two reasons, one practical and one already learned the hard way elsewhere in this repo:

• The whole point of the exercise is a self-contained, self-hosted CI pipeline — pushing built images to a third-party registry defeats that.
• registry.gitlab.com was already ruled out for pulling base images earlier (see the sanctions block in GitLab CI/CD) — no reason to introduce a similar external dependency here for images this repo controls end to end anyway.

Registry setup

Enabled on the existing GitLab instance, fronted by its own subdomain (registry.gitlab.home) rather than a subpath — the standard pattern Docker registries expect, and cleaner to reason about in NPM:

registry['enable'] = true
registry_external_url 'https://registry.gitlab.home'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitlab.home"
gitlab_rails['registry_port'] = "443"
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false

registry_port = "443" isn't the container's actual listen port — it's what GitLab tells clients to assume, since NPM terminates HTTPS on 443 and forwards to the container's internal 5050. TLS itself (via mkcert) and the NPM proxy host follow the same pattern as the rest of this repo's Reverse Proxy setup.

DNS resolution gotcha: Tailscale silently overrides Pi-hole

The new subdomain resolved fine from every other device on the LAN (all pointed at Pi-hole) but failed with Could not resolve host from the GitLab host itself. Cause: Tailscale manages /etc/resolv.conf directly on this host and overwrites it back to public resolvers (8.8.8.8/1.1.1.1) regardless of systemd-resolved configuration — the file even says so explicitly (DO NOT EDIT THIS FILE BY HAND — CHANGES WILL BE OVERWRITTEN). resolvectl status showed Pi-hole registered under one scope, but actual lookups still failed because Tailscale's override sat in front of it.

Rather than fight Tailscale's DNS management host-wide (tailscale set --accept-dns=false didn't reliably stick, and trades away MagicDNS resolution for other tailnet devices), the practical fix was a static /etc/hosts entry — surgical, doesn't touch system-wide DNS behavior, and doesn't depend on Docker containers inheriting the host's resolver config either:

192.168.100.6   registry.gitlab.home

The image: ssh-alpine

FROM alpine:3.20

RUN apk add --no-cache openssh-server && \
    ssh-keygen -A && \
    adduser -D -s /bin/ash ciuser && \
    mkdir -p /home/ciuser/.ssh && \
    chmod 700 /home/ciuser/.ssh && \
    chown -R ciuser:ciuser /home/ciuser/.ssh

RUN sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && \
    sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]

Alpine base plus --no-cache on apk add keeps this in the 10–15MB range versus 300MB+ for a Debian/Ubuntu-based equivalent — directly targeting the "keep images small" requirement. Root login and password auth are disabled by default (key-based only), since this is meant to run inside CI jobs, not be a general-purpose SSH box.

Build pipeline

stages:
  - build

.build-image:
  stage: build
  image: docker:27-cli
  services: []   # host-socket build, not Docker-in-Docker
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
  script:
    - docker pull alpine:3.20
    - docker build --pull --no-cache -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" "images/$IMAGE_NAME"
    - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest"
    - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"

build-ssh-alpine:
  extends: .build-image
  variables:
    IMAGE_NAME: ssh-alpine

Design notes:

• Host-socket build, not DinD — services: [] is explicit, not an omission. Job containers build against the host's existing Docker daemon via the same socket-mount pattern already used for the Runner itself, avoiding a second nested daemon's RAM overhead. Required adding the socket mount to [runners.docker].volumes in config.toml, not just the Runner's own container.
• --pull --no-cache is load-bearing, not decoration. Without it, a scheduled rebuild could silently reuse cached layers and produce byte-identical output even if Alpine shipped a security patch upstream — defeating the entire point of a periodic rebuild. --pull forces a fresh base-image check; --no-cache forces every RUN step (including the apk add) to actually execute again.
• Dual tagging (:latest + :$CI_COMMIT_SHORT_SHA) gives a rollback point if a rebuild ever breaks something — pin to the last known-good SHA instead of :latest.
• .build-image + extends means adding the next custom image later is a 4-line addition, not a duplicated block.
• $CI_REGISTRY* variables are auto-injected by GitLab once the registry is enabled for the project — no manual secret configuration needed.

Scheduling

GitLab's native Pipeline Schedules (Build → Pipeline schedules), not a host-level cron job — keeps the schedule visible and manageable from inside GitLab itself rather than another config file to remember exists.

0 3 */14 * *

Cron has no native "every 2 weeks" unit; */14 on the day-of-month field approximates it (drifts slightly across months of different lengths — irrelevant for a background rebuild). No rules/only restriction was added to the build job, so it runs on both the schedule and regular pushes — intentional, since editing the Dockerfile should also trigger a rebuild, not just wait for the next scheduled run.

Registry cleanup policy

Without this, the registry grows unbounded — :latest gets overwritten each run, but every :$CI_COMMIT_SHORT_SHA tag is unique and permanent by design, and a biweekly schedule running indefinitely means indefinite tag accumulation on a host with limited free disk space.

• Keep the most recent: 10 tags
• Remove tags older than: 30 days
• Keep tags matching: (intentionally empty — see note below)
• Remove tags matching: .*

Gotcha worth flagging: "Keep tags matching" is a regex field, not a numeric "keep top N" setting (that's the separate "Keep the most recent" field above it). Typing a bare digit like 3 here means "protect any tag whose name contains the character 3" — since commit-SHA tags are hex strings, most of them contain at least one digit 0–9 essentially by chance, which silently protects nearly everything and makes the whole cleanup policy close to a no-op. Left blank unless there's a genuine need to protect specific tag patterns from pruning.

Lessons learned

• GitLab's Container Registry ties every image path to an actual existing project — unlike Docker Hub, you can't push to an arbitrary namespace/path that doesn't correspond to a real project. A denied: requested access error on a first push is often this, not a credentials problem.
• Tailscale actively manages and overwrites /etc/resolv.conf on a host it's installed on, silently overriding local DNS resolvers like Pi-hole even when systemd-resolved is configured correctly — worth checking /etc/resolv.conf's own header comment before assuming a DNS config is wrong.
• A single mistyped character in a TOML config (1O instead of 10, a duplicated url = fragment) crashes the Runner container on every startup with a parse error at the first invalid line — the resulting symptom (runner shows "hasn't contacted GitLab in N hours") looks like a networking/DNS problem, not a config typo, unless the container logs are actually checked.
• A Docker build cache will happily produce an identical image on a scheduled rebuild even when the upstream base image has genuinely changed, unless --pull and --no-cache are both used — "rebuild on a schedule" and "rebuild with actual updates" are not the same thing without this.
• GitLab's registry cleanup "Keep tags matching" field is a regex, not a count — a bare number typed into it matches far more tags than intended and can silently neutralize the cleanup policy.