added gitlab to my setup and configured it

README.md

@@ -33,6 +33,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s
 | **Nginx Proxy Manager** | `docker-compose/nginx-proxy-manager/` | Reverse proxy giving every service a friendly `.home` hostname with locally-trusted HTTPS — see [Reverse Proxy](docs/Reverse-Proxy.md) |
 | **Semaphore** | `docker-compose/semaphore/` | Self-hosted Ansible UI (alternative to AWX) — see [Semaphore Guide](docs/Semaphore.md) |
 | **Ansible playbooks** | `ansible/` | Shared update role (pull → recreate → health-check → prune) reused across every stack, run via Semaphore — see [Ansible Playbooks](docs/AnsiblePlaybooks.md) |
+| **GitLab CI/CD** | `docker-compose/gitlab/` | Self-hosted GitLab Omnibus + Docker-executor Runner, tuned for this hardware and integrated with the monitoring stack — see [GitLab CI/CD](docs/GitLab-CICD.md) |
 
 ## Learning labs (not always-on)
 
@@ -92,6 +93,11 @@ homelab/
 │   ├── semaphore/
 │   │   ├── docker-compose.yml
 │   │   └── README.md
+│   ├── gitlab/
+│   │   ├── docker-compose.yml
+│   │   ├── config.toml.example
+│   │   └── backups/
+│   │       └── gitlab-backup.sh
 │   └── elk/
 │       ├── docker-compose.yml
 │       └── .env.example
@@ -130,8 +136,11 @@ homelab/
 │   ├── Semaphore.md
 │   ├── DockScope.md
 │   ├── AnsiblePlaybooks.md
+│   ├── GitLab-CICD.md
 │   ├── ElasticStack.md
 │   └── images/
+├── dashboards/
+│   └── gitlab-application-runner-metrics.json
 └── .gitignore
 ```
 
@@ -169,10 +178,13 @@ Rather than pulling and recreating manually, every always-on stack has a matchin
 - **Promtail is end-of-life as of March 2026** and removed outright as of Loki 3.7.3 — any current Loki setup should use Grafana Alloy for log shipping instead. Worth checking a project's actual current-recommended tooling before following an older tutorial, since agent names and workflows in this space have moved fast.
 - Bind-mounted container data directories can end up owned by the wrong UID after mixing `sudo` and non-`sudo` commands on the same folder — this silently broke Loki's ability to write its own working directory, surfacing only as a `permission denied` deep in its startup logs.
 - Community Grafana dashboards built for older Loki/Promtail label conventions (e.g. `container_name`) don't automatically work against a fresh Alloy-based label schema (`container`, `service_name`) — worth checking a dashboard's expected labels against your actual pipeline's labels before assuming an import error means something's broken on your end.
+- GitLab Omnibus's default RAM footprint assumes dedicated hardware — running it alongside an existing stack requires trimming Puma/Sidekiq workers explicitly, and disabling GitLab's own bundled monitoring in favor of the stack already running here.
+- `registry.gitlab.com` is geo-blocked under U.S. sanctions for several countries at the infrastructure level, independent of any account or token — surfaced as a Docker-executor helper-image pull failure. Fix was sourcing the helper image from Docker Hub instead. See [GitLab CI/CD](docs/GitLab-CICD.md).
+- Files written by a root-running container into a bind-mounted volume (GitLab's `config/`, `data/backups/`) are root-owned on the host — a backup script touching these needs to run as root and `chown` its own output back to the regular user, or every later interaction with the backup needs `sudo`.
 
 ## Roadmap
 
-- [ ] Add a self-hosted Gitea instance for private repos and CI runners
+- [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md)
 - [ ] Auto-deploy via Watchtower on image updates
 - [ ] Expand monitoring with alerting rules
 

docker-compose/gitlab/config_example.toml

@@ -0,0 +1,35 @@
+concurrent = 4
+check_interval = 0
+shutdown_timeout = 0
+listen_address = ":9252"
+
+[session_server]
+  session_timeout = 1800
+
+[[runners]]
+  name = "docker-runner"
+  url = "http://192.168.100.6:8929"
+  id = 3
+  token = "paste your token here "
+  token_obtained_at = 2026-08-06T16:15:46Z
+  token_expires_at = 0001-01-01T00:00:00Z
+  executor = "docker"
+  [runners.cache]
+    MaxUploadedArchiveSize = 0
+    [runners.cache.s3]
+      AssumeRoleMaxConcurrency = 0
+    [runners.cache.gcs]
+    [runners.cache.azure]
+  [runners.docker]
+    pull_policy = "if-not-present"
+    helper_image = "gitlab/gitlab-runner-helper:x86_64-latest"
+    tls_verify = false
+    image = "alpine:latest"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    disable_cache = false
+    volumes = ["/cache"]
+    volume_keep = false
+    shm_size = 0
+    network_mtu = 0

docker-compose/gitlab/docker-compose.yml

@@ -0,0 +1,51 @@
+services:
+  gitlab:
+    image: gitlab/gitlab-ce:19.2.1-ce.0
+    container_name: gitlab
+    restart: unless-stopped
+    hostname: gitlab.local
+    ports:
+      - "8929:80"
+      - "2224:22"
+      
+    volumes:
+      - ./config:/etc/gitlab
+      - ./logs:/var/log/gitlab
+      - ./data:/var/opt/gitlab
+    shm_size: '256m'
+    environment:
+      GITLAB_OMNIBUS_CONFIG: |
+        gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
+        external_url 'http://192.168.100.6:8929'
+
+        nginx['listen_port'] = 80
+        nginx['listen_https'] = false
+
+        gitlab_rails['gitlab_shell_ssh_port'] = 2224
+
+        # Trim workers for a 4-core / 8GB host
+        puma['worker_processes'] = 2
+        puma['min_threads'] = 1
+        puma['max_threads'] = 4
+        sidekiq['max_concurrency'] = 10
+        postgresql['shared_buffers'] = "256MB"
+
+        # You already have this stack running — turn GitLab's copies off
+        prometheus_monitoring['enable'] = false
+        alertmanager['enable'] = false
+        node_exporter['enable'] = false
+        redis_exporter['enable'] = false
+        postgres_exporter['enable'] = false
+        gitlab_exporter['enable'] = false
+
+        # Registry off for now — flip to true later if you want it
+        registry['enable'] = false
+  gitlab-runner:
+    image: gitlab/gitlab-runner:latest
+    container_name: gitlab-runner
+    restart: unless-stopped
+    ports:
+      - "9252:9252"
+    volumes:
+      - ./runner-config:/etc/gitlab-runner
+      - /var/run/docker.sock:/var/run/docker.sock

docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/prometheus.yml

@@ -0,0 +1,23 @@
+global:
+  scrape_interval: 15s
+
+scrape_configs:
+  - job_name: 'prometheus'
+    static_configs:
+      - targets: ['localhost:9090']
+
+  - job_name: 'cadvisor'
+    static_configs:
+      - targets: ['cadvisor:8080']
+
+  - job_name: 'node_exporter'
+    static_configs:
+      - targets: ['node-exporter:9100']
+
+  - job_name: 'gitlab'
+    metrics_path: '/-/metrics'
+    static_configs:
+      - targets: ['192.168.100.6:8929']
+  - job_name: 'gitlab-runner'
+    static_configs:
+      - targets: ['192.168.100.6:9252'] # Or your runner container name/IP

docs/GitLab-CICD.md

@@ -0,0 +1,122 @@
+# GitLab CI/CD
+
+A self-hosted GitLab instance (Omnibus, Docker) with a Docker-executor Runner, wired into the existing Prometheus/Grafana/Loki stack. Built as a DevOps coursework exercise, tuned to actually run on this hardware rather than GitLab's default sizing assumptions.
+
+## Why self-hosted instead of gitlab.com
+
+The coursework required hands-on CI/CD administration — runners, executors, pipeline internals — which a SaaS instance abstracts away. Self-hosting also surfaced a real constraint the course didn't cover: GitLab is bound by U.S. OFAC sanctions and blocks registry access from several countries, which turned into a genuine debugging exercise (see below).
+
+## Sizing for the hardware
+
+GitLab's own docs recommend 8GB+ RAM for even a minimal instance. This host has 8GB total, shared with the rest of the always-on stack. Omnibus was tuned down rather than run at defaults:
+
+```yaml
+puma['worker_processes'] = 2
+puma['min_threads'] = 1
+puma['max_threads'] = 4
+sidekiq['max_concurrency'] = 10
+postgresql['shared_buffers'] = "256MB"
+```
+
+GitLab Omnibus also bundles its own Prometheus/Alertmanager/exporters — all disabled here, since the host already runs its own monitoring stack:
+
+```yaml
+prometheus_monitoring['enable'] = false
+alertmanager['enable'] = false
+node_exporter['enable'] = false
+redis_exporter['enable'] = false
+postgres_exporter['enable'] = false
+gitlab_exporter['enable'] = false
+```
+
+Note: `grafana['enable']` is **not** a valid key on current Omnibus versions — GitLab dropped bundled Grafana entirely (not just disabled it), and setting this key causes a hard `reconfigure` failure (`Reading unsupported config value grafana`) rather than a silent no-op. Worth knowing if following an older guide.
+
+Container Registry is left disabled (`registry['enable'] = false`) — no current need for `docker build`/`push` in CI, and it's a one-line flip later if that changes.
+
+![GitLab instance overview](images/gitlab-instance-overview.png)
+
+## Runner
+
+Runs as a separate container (`gitlab-runner`) using the Docker executor, sharing the host's Docker daemon via a mounted socket rather than Docker-in-Docker or a dedicated VM — the lightest option that still gives each job its own isolated, ephemeral container.
+
+```yaml
+gitlab-runner:
+  image: gitlab/gitlab-runner:latest
+  ports:
+    - "9252:9252"   # runner's own Prometheus metrics endpoint
+  volumes:
+    - ./runner-config:/etc/gitlab-runner
+    - /var/run/docker.sock:/var/run/docker.sock
+```
+
+`concurrent = 1` in `config.toml` caps job concurrency to one at a time — a deliberate RAM guardrail rather than an oversight, given how little headroom this host has.
+
+![Runner online in GitLab admin](images/gitlab-runner-online.png)
+
+### Registry sanctions block
+
+First test pipeline failed instantly with `denied: error code 1009` pulling GitLab's helper image from `registry.gitlab.com`. This isn't a config or auth issue — GitLab's export-compliance policy explicitly names Iran (along with Cuba, North Korea, Syria, and Crimea) as embargoed under OFAC sanctions, enforced at the registry level regardless of account/token.
+
+Fix: override the helper image to pull from Docker Hub instead, which isn't sanctions-blocked:
+
+```toml
+[runners.docker]
+  helper_image = "gitlab/gitlab-runner-helper:x86_64-latest"
+  pull_policy = "if-not-present"
+```
+
+`pull_policy = "if-not-present"` also means both the helper and job images get cached locally after the first pull, so later jobs don't depend on external registry availability at all.
+
+## Monitoring integration
+
+GitLab's own `/-/metrics` endpoint (Prometheus format) is separate from the bundled monitoring stack disabled above — it's still fully functional and scraped by the existing Prometheus:
+
+```yaml
+- job_name: 'gitlab'
+  metrics_path: '/-/metrics'
+  static_configs:
+    - targets: ['192.168.100.6:8929']
+
+- job_name: 'gitlab-runner'
+  static_configs:
+    - targets: ['192.168.100.6:9252']
+```
+
+By default GitLab only allows `127.0.0.1` to scrape metrics; since Prometheus reaches it over a Docker bridge network (not literally localhost), the whitelist was widened to cover private ranges:
+
+```yaml
+gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
+```
+
+A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics.
+
+Logs flow into the existing Loki/Alloy pipeline automatically, since Alloy discovers all running containers via the Docker socket — no GitLab-specific log config needed.
+
+## HTTPS
+
+Fronted by the existing Nginx Proxy Manager + Pi-hole setup, same pattern as other services in this repo — see [Reverse Proxy](Reverse-Proxy.md). `external_url` in `docker-compose.yml` was updated to match the HTTPS domain, since GitLab uses this value to generate every internal link (clone URLs, CI job links, webhook payloads) — leaving it pointed at the old plain-HTTP IP after adding the proxy would have produced broken/mixed-content links throughout the UI.
+
+## Backups
+
+Weekly, single retained copy, sized for available disk space (~50GB free at time of writing). Script: [`docker-compose/gitlab/backups/gitlab-backup.sh`](../docker-compose/gitlab/backups/gitlab-backup.sh), scheduled via root's crontab (needs root to read GitLab's root-owned config/secrets files):
+
+```
+0 2 * * 0 /home/rezayaghobi/backups/run-backup.sh
+```
+
+Three things get backed up separately since they're owned/generated differently:
+
+1. **GitLab's own backup** (`gitlab-backup create`) — repos, database, uploads. Does *not* include secrets by design.
+2. **`config/`** — `gitlab.rb`, `gitlab-secrets.json`, SSH host keys. Root-owned by the container; without this a restored backup is unreadable.
+3. **Runner `config.toml`** — small, but losing it means re-registering the runner from scratch.
+
+Pruning keeps only the newest file of each type (`find ... | sort -rn | tail -n +2 | xargs rm`) rather than a time-based retention window, to stay predictable on limited disk space. Verified restorable by listing the archive contents (`tar tf`) after the first run, not just assumed from a non-zero exit code.
+
+## Lessons learned
+
+- GitLab Omnibus's RAM recommendations assume dedicated hardware — running it alongside an existing service stack requires trimming Puma/Sidekiq workers explicitly, not just relying on defaults.
+- Newer Omnibus versions removed bundled Grafana as a *feature*, not just disabled it — the `grafana['enable']` key no longer exists and hard-fails `reconfigure` if set, unlike other monitoring toggles which are still valid no-ops.
+- `registry.gitlab.com` access is geo-blocked under U.S. sanctions for several countries at the infrastructure level — no account or token change works around it; the fix is sourcing images from an unaffected registry (Docker Hub, in this case) instead.
+- GitLab's Docker-executor helper image pull happens on the **host's** Docker daemon before the job container's own networking is even relevant — a failure here isn't a job-container network issue, it's a host-level registry pull issue.
+- Community Grafana dashboards built against GitLab's *bundled* Omnibus Prometheus setup don't work against an externally-scraped `/-/metrics` endpoint — the metric names differ. Worth checking a dashboard's expected data source shape before assuming an empty panel means broken config.
+- Files created inside a bind-mounted volume by a root-running container (GitLab's `config/`, `data/backups/`) are root-owned on the host — backup scripts touching these need to run as root and explicitly `chown` their own output back to the regular user afterward, or every later interaction with the backup needs `sudo`.

docs/Home.md

@@ -16,6 +16,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 - [Semaphore](Semaphore.md)
 - [DockScope](DockScope.md)
 - [Ansible Playbooks](AnsiblePlaybooks.md)
+- [GitLab CI/CD](GitLab-CICD.md)
 - [Elastic Stack (ELK) — Lab](ElasticStack.md)
 
 ## At a glance
@@ -30,6 +31,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This
 | **Automation**     | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent |
 | **Disk health** | Hard Disk Sentinel |
 | **AI Agent** | OpenHands + Ollama |
+| **CI/CD** | Self-hosted **GitLab** (Omnibus) + Docker-executor **Runner** |
 
 This is a running system, not a demo — screenshots throughout this wiki are from the live dashboards.
 

docs/Monitoring-Stack.md

@@ -34,6 +34,23 @@ Logs are browsable two ways in Grafana:
 - **Explore → Loki**, querying by label directly (e.g. `{container="grafana"}`)
 - **Drilldown → Logs**, a zero-config log browser bundled with Grafana — no dashboard needed to start exploring
 
+## GitLab & Runner metrics
+
+Self-hosted GitLab (see [GitLab CI/CD](GitLab-CICD.md)) exposes its own Prometheus-format metrics at `/-/metrics`, separate from Omnibus's bundled monitoring stack, which is disabled here in favor of this existing setup. The Runner similarly exposes its own metrics endpoint once `listen_address` is set in `config.toml`. Both are added as standard Prometheus scrape targets:
+
+```yaml
+- job_name: 'gitlab'
+  metrics_path: '/-/metrics'
+  static_configs:
+    - targets: ['192.168.100.6:8929']
+
+- job_name: 'gitlab-runner'
+  static_configs:
+    - targets: ['192.168.100.6:9252']
+```
+
+Visualized on a hand-built Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) rather than GitLab's official community dashboard — that one targets metric names from Omnibus's own bundled Prometheus, which isn't what's running here.
+
 ## Why this matters
 
 On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact.