homelab/docs/CI-Custom-Images.md

# Custom CI Runner Images

Self-built, lightweight Docker images for use inside GitLab CI pipelines, stored in a self-hosted Container Registry and rebuilt on a biweekly schedule so they stay current with upstream security patches — rather than relying on generic public images pulled fresh (and un-vetted) in every job.

Lives in its own project (`ci-images`) inside the self-hosted GitLab instance — see [GitLab CI/CD](GitLab-CICD.md) for the base GitLab/Runner setup this builds on. Reference copies of the project files are kept in this repo under [`docker-compose/gitlab/ci-images-reference/`](../docker-compose/gitlab/ci-images-reference/), since the live project itself isn't on GitHub.

## Why a private registry instead of Docker Hub

Two reasons, one practical and one already learned the hard way elsewhere in this repo:

- The whole point of the exercise is a self-contained, self-hosted CI pipeline — pushing built images to a third-party registry defeats that.
- `registry.gitlab.com` was already ruled out for pulling *base* images earlier (see the sanctions block in [GitLab CI/CD](GitLab-CICD.md)) — no reason to introduce a similar external dependency here for images this repo controls end to end anyway.

## Registry setup

Enabled on the existing GitLab instance, fronted by its own subdomain (`registry.gitlab.home`) rather than a subpath — the standard pattern Docker registries expect, and cleaner to reason about in NPM:

```yaml
registry['enable'] = true
registry_external_url 'https://registry.gitlab.home'
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "registry.gitlab.home"
gitlab_rails['registry_port'] = "443"
registry_nginx['listen_port'] = 5050
registry_nginx['listen_https'] = false
```

`registry_port = "443"` isn't the container's actual listen port — it's what GitLab tells clients to assume, since NPM terminates HTTPS on 443 and forwards to the container's internal `5050`. TLS itself (via `mkcert`) and the NPM proxy host follow the same pattern as the rest of this repo's [Reverse Proxy](Reverse-Proxy.md) setup.

### DNS resolution gotcha: Tailscale silently overrides Pi-hole

The new subdomain resolved fine from every other device on the LAN (all pointed at Pi-hole) but failed with `Could not resolve host` from the GitLab **host itself**. Cause: Tailscale manages `/etc/resolv.conf` directly on this host and overwrites it back to public resolvers (`8.8.8.8`/`1.1.1.1`) regardless of `systemd-resolved` configuration — the file even says so explicitly (`DO NOT EDIT THIS FILE BY HAND — CHANGES WILL BE OVERWRITTEN`). `resolvectl status` showed Pi-hole registered under one scope, but actual lookups still failed because Tailscale's override sat in front of it.

Rather than fight Tailscale's DNS management host-wide (`tailscale set --accept-dns=false` didn't reliably stick, and trades away MagicDNS resolution for other tailnet devices), the practical fix was a static `/etc/hosts` entry — surgical, doesn't touch system-wide DNS behavior, and doesn't depend on Docker containers inheriting the host's resolver config either:

```
192.168.100.6   registry.gitlab.home
```

## The image: `ssh-alpine`

```dockerfile
FROM alpine:3.20

RUN apk add --no-cache openssh-server && \
    ssh-keygen -A && \
    adduser -D -s /bin/ash ciuser && \
    mkdir -p /home/ciuser/.ssh && \
    chmod 700 /home/ciuser/.ssh && \
    chown -R ciuser:ciuser /home/ciuser/.ssh

RUN sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config && \
    sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config

EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]
```

Alpine base plus `--no-cache` on `apk add` keeps this in the 10–15MB range versus 300MB+ for a Debian/Ubuntu-based equivalent — directly targeting the "keep images small" requirement. Root login and password auth are disabled by default (key-based only), since this is meant to run inside CI jobs, not be a general-purpose SSH box.

## Build pipeline

```yaml
stages:
  - build

.build-image:
  stage: build
  image: docker:27-cli
  services: []   # host-socket build, not Docker-in-Docker
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
  script:
    - docker pull alpine:3.20
    - docker build --pull --no-cache -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest" -t "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA" "images/$IMAGE_NAME"
    - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:latest"
    - docker push "$CI_REGISTRY_IMAGE/$IMAGE_NAME:$CI_COMMIT_SHORT_SHA"

build-ssh-alpine:
  extends: .build-image
  variables:
    IMAGE_NAME: ssh-alpine
```

Design notes:

- **Host-socket build, not DinD** — `services: []` is explicit, not an omission. Job containers build against the host's existing Docker daemon via the same socket-mount pattern already used for the Runner itself, avoiding a second nested daemon's RAM overhead. Required adding the socket mount to `[runners.docker].volumes` in `config.toml`, not just the Runner's own container.
- **`--pull --no-cache` is load-bearing, not decoration.** Without it, a scheduled rebuild could silently reuse cached layers and produce byte-identical output even if Alpine shipped a security patch upstream — defeating the entire point of a periodic rebuild. `--pull` forces a fresh base-image check; `--no-cache` forces every `RUN` step (including the `apk add`) to actually execute again.
- **Dual tagging** (`:latest` + `:$CI_COMMIT_SHORT_SHA`) gives a rollback point if a rebuild ever breaks something — pin to the last known-good SHA instead of `:latest`.
- **`.build-image` + `extends`** means adding the next custom image later is a 4-line addition, not a duplicated block.
- **`$CI_REGISTRY*` variables are auto-injected by GitLab** once the registry is enabled for the project — no manual secret configuration needed.

## Scheduling

GitLab's native **Pipeline Schedules** (Build → Pipeline schedules), not a host-level cron job — keeps the schedule visible and manageable from inside GitLab itself rather than another config file to remember exists.

```
0 3 */14 * *
```

Cron has no native "every 2 weeks" unit; `*/14` on the day-of-month field approximates it (drifts slightly across months of different lengths — irrelevant for a background rebuild). No `rules`/`only` restriction was added to the build job, so it runs on both the schedule *and* regular pushes — intentional, since editing the Dockerfile should also trigger a rebuild, not just wait for the next scheduled run.

## Registry cleanup policy

Without this, the registry grows unbounded — `:latest` gets overwritten each run, but every `:$CI_COMMIT_SHORT_SHA` tag is unique and permanent by design, and a biweekly schedule running indefinitely means indefinite tag accumulation on a host with limited free disk space.

- Keep the most recent: **10 tags**
- Remove tags older than: **30 days**
- Keep tags matching: *(intentionally empty — see note below)*
- Remove tags matching: `.*`

**Gotcha worth flagging**: "Keep tags matching" is a regex field, not a numeric "keep top N" setting (that's the separate "Keep the most recent" field above it). Typing a bare digit like `3` here means "protect any tag whose name contains the character 3" — since commit-SHA tags are hex strings, most of them contain at least one digit 0–9 essentially by chance, which silently protects nearly everything and makes the whole cleanup policy close to a no-op. Left blank unless there's a genuine need to protect specific tag patterns from pruning.

## Lessons learned

- GitLab's Container Registry ties every image path to an actual existing project — unlike Docker Hub, you can't push to an arbitrary namespace/path that doesn't correspond to a real project. A `denied: requested access` error on a first push is often this, not a credentials problem.
- Tailscale actively manages and overwrites `/etc/resolv.conf` on a host it's installed on, silently overriding local DNS resolvers like Pi-hole even when `systemd-resolved` is configured correctly — worth checking `/etc/resolv.conf`'s own header comment before assuming a DNS config is wrong.
- A single mistyped character in a TOML config (`1O` instead of `10`, a duplicated `url = ` fragment) crashes the Runner container on every startup with a parse error at the *first* invalid line — the resulting symptom (runner shows "hasn't contacted GitLab in N hours") looks like a networking/DNS problem, not a config typo, unless the container logs are actually checked.
- A Docker build cache will happily produce an identical image on a scheduled rebuild even when the upstream base image has genuinely changed, unless `--pull` and `--no-cache` are both used — "rebuild on a schedule" and "rebuild with actual updates" are not the same thing without this.
- GitLab's registry cleanup "Keep tags matching" field is a regex, not a count — a bare number typed into it matches far more tags than intended and can silently neutralize the cleanup policy.