diff --git a/README.md b/README.md index 9b6f26c..19086df 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,7 @@ This is a real, running homelab — not a demo. It's deployed on a home Ubuntu s | **Nginx Proxy Manager** | `docker-compose/nginx-proxy-manager/` | Reverse proxy giving every service a friendly `.home` hostname with locally-trusted HTTPS — see [Reverse Proxy](docs/Reverse-Proxy.md) | | **Semaphore** | `docker-compose/semaphore/` | Self-hosted Ansible UI (alternative to AWX) — see [Semaphore Guide](docs/Semaphore.md) | | **Ansible playbooks** | `ansible/` | Shared update role (pull → recreate → health-check → prune) reused across every stack, run via Semaphore — see [Ansible Playbooks](docs/AnsiblePlaybooks.md) | +| **GitLab CI/CD** | `docker-compose/gitlab/` | Self-hosted GitLab Omnibus + Docker-executor Runner, tuned for this hardware and integrated with the monitoring stack — see [GitLab CI/CD](docs/GitLab-CICD.md) | ## Learning labs (not always-on) @@ -92,6 +93,11 @@ homelab/ │ ├── semaphore/ │ │ ├── docker-compose.yml │ │ └── README.md +│ ├── gitlab/ +│ │ ├── docker-compose.yml +│ │ ├── config.toml.example +│ │ └── backups/ +│ │ └── gitlab-backup.sh │ └── elk/ │ ├── docker-compose.yml │ └── .env.example @@ -130,8 +136,11 @@ homelab/ │ ├── Semaphore.md │ ├── DockScope.md │ ├── AnsiblePlaybooks.md +│ ├── GitLab-CICD.md │ ├── ElasticStack.md │ └── images/ +├── dashboards/ +│ └── gitlab-application-runner-metrics.json └── .gitignore ``` @@ -169,10 +178,13 @@ Rather than pulling and recreating manually, every always-on stack has a matchin - **Promtail is end-of-life as of March 2026** and removed outright as of Loki 3.7.3 — any current Loki setup should use Grafana Alloy for log shipping instead. Worth checking a project's actual current-recommended tooling before following an older tutorial, since agent names and workflows in this space have moved fast. - Bind-mounted container data directories can end up owned by the wrong UID after mixing `sudo` and non-`sudo` commands on the same folder — this silently broke Loki's ability to write its own working directory, surfacing only as a `permission denied` deep in its startup logs. - Community Grafana dashboards built for older Loki/Promtail label conventions (e.g. `container_name`) don't automatically work against a fresh Alloy-based label schema (`container`, `service_name`) — worth checking a dashboard's expected labels against your actual pipeline's labels before assuming an import error means something's broken on your end. +- GitLab Omnibus's default RAM footprint assumes dedicated hardware — running it alongside an existing stack requires trimming Puma/Sidekiq workers explicitly, and disabling GitLab's own bundled monitoring in favor of the stack already running here. +- `registry.gitlab.com` is geo-blocked under U.S. sanctions for several countries at the infrastructure level, independent of any account or token — surfaced as a Docker-executor helper-image pull failure. Fix was sourcing the helper image from Docker Hub instead. See [GitLab CI/CD](docs/GitLab-CICD.md). +- Files written by a root-running container into a bind-mounted volume (GitLab's `config/`, `data/backups/`) are root-owned on the host — a backup script touching these needs to run as root and `chown` its own output back to the regular user, or every later interaction with the backup needs `sudo`. ## Roadmap -- [ ] Add a self-hosted Gitea instance for private repos and CI runners +- [x] ~~Add a self-hosted Gitea instance for private repos and CI runners~~ — done via self-hosted GitLab instead, see [GitLab CI/CD](docs/GitLab-CICD.md) - [ ] Auto-deploy via Watchtower on image updates - [ ] Expand monitoring with alerting rules diff --git a/dashboards/gitlab-application-runner-metrics.json b/dashboards/gitlab-application-runner-metrics.json new file mode 100644 index 0000000..6ad8df7 --- /dev/null +++ b/dashboards/gitlab-application-runner-metrics.json @@ -0,0 +1,1724 @@ +{ + "apiVersion": "dashboard.grafana.app/v2", + "kind": "Dashboard", + "metadata": { + "name": "gitlab-app-metricss", + "namespace": "default", + "uid": "e1ea2b6d-f036-4f5e-bdb5-5c0fdfef1679", + "resourceVersion": "1786041594900991", + "generation": 4, + "creationTimestamp": "2026-08-06T18:02:09Z", + "labels": { + "grafana.app/deprecatedInternalID": "259585787068416" + }, + "annotations": { + "grafana.app/createdBy": "user:dfbtp4cn32dj4f", + "grafana.app/folder": "", + "grafana.app/saved-from-ui": "Grafana v13.1.1 (a9cee6e172)", + "grafana.app/updatedBy": "user:dfbtp4cn32dj4f", + "grafana.app/updatedTimestamp": "2026-08-06T18:39:54Z" + } + }, + "spec": { + "annotations": [ + { + "kind": "AnnotationQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "grafana", + "version": "v0", + "spec": {} + }, + "enable": true, + "hide": true, + "iconColor": "", + "name": "Annotations & Alerts", + "builtIn": true + } + } + ], + "cursorSync": "Crosshair", + "editable": true, + "elements": { + "panel-1": { + "kind": "Panel", + "spec": { + "id": 1, + "title": "Rails Request Queue Latency", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum(rate(gitlab_rails_queue_duration_seconds_bucket{job=\"gitlab\"}[5m])) by (le))", + "legendFormat": "p95 Queue Duration", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_rails_queue_duration_seconds_sum{job=\"gitlab\"}[5m])) / sum(rate(gitlab_rails_queue_duration_seconds_count{job=\"gitlab\"}[5m]))", + "legendFormat": "Avg Queue Duration", + "range": true + } + }, + "refId": "B", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "smooth", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-10": { + "kind": "Panel", + "spec": { + "id": 10, + "title": "GitLab Runner Memory Footprint", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "process_resident_memory_bytes{job=\"gitlab-runner\"}", + "legendFormat": "RAM Usage", + "range": true + } + }, + "refId": "A", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "bytes", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-12": { + "kind": "Panel", + "spec": { + "id": 12, + "title": "logs", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "loki", + "version": "v0", + "datasource": { + "name": "P8E80F9AEF21F6940" + }, + "spec": { + "direction": "backward", + "editorMode": "code", + "expr": "{service_name=\"gitlab\"}", + "queryType": "range" + } + }, + "refId": "A", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "logs", + "version": "13.1.1", + "spec": { + "options": { + "dedupStrategy": "none", + "enableInfiniteScrolling": false, + "enableLogDetails": true, + "prettifyLogMessage": true, + "showControls": false, + "showFieldSelector": false, + "showLevel": true, + "showTime": false, + "sortOrder": "Descending", + "timestampResolution": "ms", + "unwrappedColumns": false, + "wrapLogMessage": false + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + } + } + } + } + }, + "panel-2": { + "kind": "Panel", + "spec": { + "id": 2, + "title": "Puma Memory Watchdog Limit", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_memwd_max_memory_limit{job=\"gitlab\"}", + "legendFormat": "Puma Max Memory Limit ({{pid}})", + "range": true + } + }, + "refId": "A", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "bytes", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-3": { + "kind": "Panel", + "spec": { + "id": 3, + "title": "PostgreSQL Connection Pool Status", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_database_connection_pool_busy{job=\"gitlab\"}", + "legendFormat": "Busy Connections ({{db_config_name}})", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_database_connection_pool_connections{job=\"gitlab\"}", + "legendFormat": "Active Connections ({{db_config_name}})", + "range": true + } + }, + "refId": "B", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_database_connection_pool_size{job=\"gitlab\"}", + "legendFormat": "Max Pool Size ({{db_config_name}})", + "range": true + } + }, + "refId": "C", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-4": { + "kind": "Panel", + "spec": { + "id": 4, + "title": "Redis Client Operations / sec", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_redis_client_requests_total{job=\"gitlab\"}[5m])) by (storage)", + "legendFormat": "{{storage}}", + "range": true + } + }, + "refId": "A", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "table", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "reqps", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-5": { + "kind": "Panel", + "spec": { + "id": 5, + "title": "GitLab Application Cache Activity", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_cache_operations_total{job=\"gitlab\"}[5m])) by (operation)", + "legendFormat": "Ops: {{operation}}", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_cache_misses_total{job=\"gitlab\"}[5m]))", + "legendFormat": "Cache Misses Total", + "range": true + } + }, + "refId": "B", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "ops", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-6": { + "kind": "Panel", + "spec": { + "id": 6, + "title": "ActionCable & Active CI Jobs", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "action_cable_active_connections{job=\"gitlab\"}", + "legendFormat": "ActionCable Active Connections ({{pid}})", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_ci_active_jobs_count{job=\"gitlab\"}", + "legendFormat": "Active CI Jobs", + "range": true + } + }, + "refId": "B", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-7": { + "kind": "Panel", + "spec": { + "id": 7, + "title": "Runner Active Jobs vs Limit", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(gitlab_runner_jobs{job=\"gitlab-runner\", state=\"running\"})", + "legendFormat": "Running Jobs", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "gitlab_runner_concurrent{job=\"gitlab-runner\"}", + "legendFormat": "Max Concurrent Limit", + "range": true + } + }, + "refId": "B", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "short", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-8": { + "kind": "Panel", + "spec": { + "id": 8, + "title": "Runner Job Execution Duration", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum(rate(gitlab_runner_job_duration_seconds_bucket{job=\"gitlab-runner\"}[5m])) by (le))", + "legendFormat": "p95 Job Duration", + "range": true + } + }, + "refId": "A", + "hidden": false + } + }, + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_runner_job_duration_seconds_sum{job=\"gitlab-runner\"}[5m])) / sum(rate(gitlab_runner_job_duration_seconds_count{job=\"gitlab-runner\"}[5m]))", + "legendFormat": "Avg Job Duration", + "range": true + } + }, + "refId": "B", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "s", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + }, + "panel-9": { + "kind": "Panel", + "spec": { + "id": 9, + "title": "Runner Job Errors / sec", + "description": "", + "links": [], + "data": { + "kind": "QueryGroup", + "spec": { + "queries": [ + { + "kind": "PanelQuery", + "spec": { + "query": { + "kind": "DataQuery", + "group": "prometheus", + "version": "v0", + "datasource": { + "name": "${DS_PROMETHEUS}" + }, + "spec": { + "editorMode": "code", + "expr": "sum(rate(gitlab_runner_errors_total{job=\"gitlab-runner\"}[5m])) by (runner)", + "legendFormat": "Errors ({{runner}})", + "range": true + } + }, + "refId": "A", + "hidden": false + } + } + ], + "transformations": [], + "queryOptions": {} + } + }, + "vizConfig": { + "kind": "VizConfig", + "group": "timeseries", + "version": "13.1.1", + "spec": { + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "mode": "single", + "sort": "none" + } + }, + "fieldConfig": { + "defaults": { + "unit": "errps", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "value": 0, + "color": "green" + }, + { + "value": 80, + "color": "red" + } + ] + }, + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 5, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + } + }, + "overrides": [] + } + } + } + } + } + }, + "layout": { + "kind": "RowsLayout", + "spec": { + "rows": [ + { + "kind": "RowsLayoutRow", + "spec": { + "title": "GitLab Rails & Puma Overview", + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-1" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 12, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-2" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 8, + "width": 24, + "height": 6, + "element": { + "kind": "ElementReference", + "name": "panel-12" + } + } + } + ] + } + } + } + }, + { + "kind": "RowsLayoutRow", + "spec": { + "title": "Database & Redis Performance", + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-3" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 12, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-4" + } + } + } + ] + } + } + } + }, + { + "kind": "RowsLayoutRow", + "spec": { + "title": "Cache & CI Metrics", + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-5" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 12, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-6" + } + } + } + ] + } + } + } + }, + { + "kind": "RowsLayoutRow", + "spec": { + "title": "GitLab Runner Metrics", + "collapse": false, + "layout": { + "kind": "GridLayout", + "spec": { + "items": [ + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-7" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 12, + "y": 0, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-8" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 0, + "y": 8, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-9" + } + } + }, + { + "kind": "GridLayoutItem", + "spec": { + "x": 12, + "y": 8, + "width": 12, + "height": 8, + "element": { + "kind": "ElementReference", + "name": "panel-10" + } + } + } + ] + } + } + } + } + ] + } + }, + "links": [], + "liveNow": false, + "preload": false, + "tags": [ + "gitlab", + "prometheus", + "runner" + ], + "timeSettings": { + "timezone": "browser", + "from": "now-1h", + "to": "now", + "autoRefresh": "10s", + "autoRefreshIntervals": [ + "5s", + "10s", + "30s", + "1m", + "5m", + "15m", + "30m", + "1h", + "2h", + "1d" + ], + "hideTimepicker": false, + "fiscalYearStartMonth": 0 + }, + "title": "GitLab Application & Runner Metrics", + "variables": [ + { + "kind": "DatasourceVariable", + "spec": { + "name": "DS_PROMETHEUS", + "pluginId": "prometheus", + "refresh": "onDashboardLoad", + "regex": "", + "current": { + "text": "Prometheus", + "value": "Prometheus" + }, + "options": [], + "multi": false, + "includeAll": false, + "label": "Data Source", + "hide": "dontHide", + "skipUrlSync": false, + "allowCustomValue": true + } + } + ] + } +} \ No newline at end of file diff --git a/docker-compose/gitlab/config_example.toml b/docker-compose/gitlab/config_example.toml new file mode 100644 index 0000000..8c57ca0 --- /dev/null +++ b/docker-compose/gitlab/config_example.toml @@ -0,0 +1,35 @@ +concurrent = 4 +check_interval = 0 +shutdown_timeout = 0 +listen_address = ":9252" + +[session_server] + session_timeout = 1800 + +[[runners]] + name = "docker-runner" + url = "http://192.168.100.6:8929" + id = 3 + token = "paste your token here " + token_obtained_at = 2026-08-06T16:15:46Z + token_expires_at = 0001-01-01T00:00:00Z + executor = "docker" + [runners.cache] + MaxUploadedArchiveSize = 0 + [runners.cache.s3] + AssumeRoleMaxConcurrency = 0 + [runners.cache.gcs] + [runners.cache.azure] + [runners.docker] + pull_policy = "if-not-present" + helper_image = "gitlab/gitlab-runner-helper:x86_64-latest" + tls_verify = false + image = "alpine:latest" + privileged = false + disable_entrypoint_overwrite = false + oom_kill_disable = false + disable_cache = false + volumes = ["/cache"] + volume_keep = false + shm_size = 0 + network_mtu = 0 diff --git a/docker-compose/gitlab/docker-compose.yml b/docker-compose/gitlab/docker-compose.yml new file mode 100644 index 0000000..f7c3f2a --- /dev/null +++ b/docker-compose/gitlab/docker-compose.yml @@ -0,0 +1,51 @@ +services: + gitlab: + image: gitlab/gitlab-ce:19.2.1-ce.0 + container_name: gitlab + restart: unless-stopped + hostname: gitlab.local + ports: + - "8929:80" + - "2224:22" + + volumes: + - ./config:/etc/gitlab + - ./logs:/var/log/gitlab + - ./data:/var/opt/gitlab + shm_size: '256m' + environment: + GITLAB_OMNIBUS_CONFIG: | + gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'] + external_url 'http://192.168.100.6:8929' + + nginx['listen_port'] = 80 + nginx['listen_https'] = false + + gitlab_rails['gitlab_shell_ssh_port'] = 2224 + + # Trim workers for a 4-core / 8GB host + puma['worker_processes'] = 2 + puma['min_threads'] = 1 + puma['max_threads'] = 4 + sidekiq['max_concurrency'] = 10 + postgresql['shared_buffers'] = "256MB" + + # You already have this stack running — turn GitLab's copies off + prometheus_monitoring['enable'] = false + alertmanager['enable'] = false + node_exporter['enable'] = false + redis_exporter['enable'] = false + postgres_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Registry off for now — flip to true later if you want it + registry['enable'] = false + gitlab-runner: + image: gitlab/gitlab-runner:latest + container_name: gitlab-runner + restart: unless-stopped + ports: + - "9252:9252" + volumes: + - ./runner-config:/etc/gitlab-runner + - /var/run/docker.sock:/var/run/docker.sock diff --git a/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/prometheus.yml b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/prometheus.yml new file mode 100644 index 0000000..dc4b4c2 --- /dev/null +++ b/docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/prometheus.yml @@ -0,0 +1,23 @@ +global: + scrape_interval: 15s + +scrape_configs: + - job_name: 'prometheus' + static_configs: + - targets: ['localhost:9090'] + + - job_name: 'cadvisor' + static_configs: + - targets: ['cadvisor:8080'] + + - job_name: 'node_exporter' + static_configs: + - targets: ['node-exporter:9100'] + + - job_name: 'gitlab' + metrics_path: '/-/metrics' + static_configs: + - targets: ['192.168.100.6:8929'] + - job_name: 'gitlab-runner' + static_configs: + - targets: ['192.168.100.6:9252'] # Or your runner container name/IP diff --git a/docs/GitLab-CICD.md b/docs/GitLab-CICD.md new file mode 100644 index 0000000..d7572db --- /dev/null +++ b/docs/GitLab-CICD.md @@ -0,0 +1,122 @@ +# GitLab CI/CD + +A self-hosted GitLab instance (Omnibus, Docker) with a Docker-executor Runner, wired into the existing Prometheus/Grafana/Loki stack. Built as a DevOps coursework exercise, tuned to actually run on this hardware rather than GitLab's default sizing assumptions. + +## Why self-hosted instead of gitlab.com + +The coursework required hands-on CI/CD administration — runners, executors, pipeline internals — which a SaaS instance abstracts away. Self-hosting also surfaced a real constraint the course didn't cover: GitLab is bound by U.S. OFAC sanctions and blocks registry access from several countries, which turned into a genuine debugging exercise (see below). + +## Sizing for the hardware + +GitLab's own docs recommend 8GB+ RAM for even a minimal instance. This host has 8GB total, shared with the rest of the always-on stack. Omnibus was tuned down rather than run at defaults: + +```yaml +puma['worker_processes'] = 2 +puma['min_threads'] = 1 +puma['max_threads'] = 4 +sidekiq['max_concurrency'] = 10 +postgresql['shared_buffers'] = "256MB" +``` + +GitLab Omnibus also bundles its own Prometheus/Alertmanager/exporters — all disabled here, since the host already runs its own monitoring stack: + +```yaml +prometheus_monitoring['enable'] = false +alertmanager['enable'] = false +node_exporter['enable'] = false +redis_exporter['enable'] = false +postgres_exporter['enable'] = false +gitlab_exporter['enable'] = false +``` + +Note: `grafana['enable']` is **not** a valid key on current Omnibus versions — GitLab dropped bundled Grafana entirely (not just disabled it), and setting this key causes a hard `reconfigure` failure (`Reading unsupported config value grafana`) rather than a silent no-op. Worth knowing if following an older guide. + +Container Registry is left disabled (`registry['enable'] = false`) — no current need for `docker build`/`push` in CI, and it's a one-line flip later if that changes. + +![GitLab instance overview](images/gitlab-instance-overview.png) + +## Runner + +Runs as a separate container (`gitlab-runner`) using the Docker executor, sharing the host's Docker daemon via a mounted socket rather than Docker-in-Docker or a dedicated VM — the lightest option that still gives each job its own isolated, ephemeral container. + +```yaml +gitlab-runner: + image: gitlab/gitlab-runner:latest + ports: + - "9252:9252" # runner's own Prometheus metrics endpoint + volumes: + - ./runner-config:/etc/gitlab-runner + - /var/run/docker.sock:/var/run/docker.sock +``` + +`concurrent = 1` in `config.toml` caps job concurrency to one at a time — a deliberate RAM guardrail rather than an oversight, given how little headroom this host has. + +![Runner online in GitLab admin](images/gitlab-runner-online.png) + +### Registry sanctions block + +First test pipeline failed instantly with `denied: error code 1009` pulling GitLab's helper image from `registry.gitlab.com`. This isn't a config or auth issue — GitLab's export-compliance policy explicitly names Iran (along with Cuba, North Korea, Syria, and Crimea) as embargoed under OFAC sanctions, enforced at the registry level regardless of account/token. + +Fix: override the helper image to pull from Docker Hub instead, which isn't sanctions-blocked: + +```toml +[runners.docker] + helper_image = "gitlab/gitlab-runner-helper:x86_64-latest" + pull_policy = "if-not-present" +``` + +`pull_policy = "if-not-present"` also means both the helper and job images get cached locally after the first pull, so later jobs don't depend on external registry availability at all. + +## Monitoring integration + +GitLab's own `/-/metrics` endpoint (Prometheus format) is separate from the bundled monitoring stack disabled above — it's still fully functional and scraped by the existing Prometheus: + +```yaml +- job_name: 'gitlab' + metrics_path: '/-/metrics' + static_configs: + - targets: ['192.168.100.6:8929'] + +- job_name: 'gitlab-runner' + static_configs: + - targets: ['192.168.100.6:9252'] +``` + +By default GitLab only allows `127.0.0.1` to scrape metrics; since Prometheus reaches it over a Docker bridge network (not literally localhost), the whitelist was widened to cover private ranges: + +```yaml +gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'] +``` + +A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics. + +Logs flow into the existing Loki/Alloy pipeline automatically, since Alloy discovers all running containers via the Docker socket — no GitLab-specific log config needed. + +## HTTPS + +Fronted by the existing Nginx Proxy Manager + Pi-hole setup, same pattern as other services in this repo — see [Reverse Proxy](Reverse-Proxy.md). `external_url` in `docker-compose.yml` was updated to match the HTTPS domain, since GitLab uses this value to generate every internal link (clone URLs, CI job links, webhook payloads) — leaving it pointed at the old plain-HTTP IP after adding the proxy would have produced broken/mixed-content links throughout the UI. + +## Backups + +Weekly, single retained copy, sized for available disk space (~50GB free at time of writing). Script: [`docker-compose/gitlab/backups/gitlab-backup.sh`](../docker-compose/gitlab/backups/gitlab-backup.sh), scheduled via root's crontab (needs root to read GitLab's root-owned config/secrets files): + +``` +0 2 * * 0 /home/rezayaghobi/backups/run-backup.sh +``` + +Three things get backed up separately since they're owned/generated differently: + +1. **GitLab's own backup** (`gitlab-backup create`) — repos, database, uploads. Does *not* include secrets by design. +2. **`config/`** — `gitlab.rb`, `gitlab-secrets.json`, SSH host keys. Root-owned by the container; without this a restored backup is unreadable. +3. **Runner `config.toml`** — small, but losing it means re-registering the runner from scratch. + +Pruning keeps only the newest file of each type (`find ... | sort -rn | tail -n +2 | xargs rm`) rather than a time-based retention window, to stay predictable on limited disk space. Verified restorable by listing the archive contents (`tar tf`) after the first run, not just assumed from a non-zero exit code. + +## Lessons learned + +- GitLab Omnibus's RAM recommendations assume dedicated hardware — running it alongside an existing service stack requires trimming Puma/Sidekiq workers explicitly, not just relying on defaults. +- Newer Omnibus versions removed bundled Grafana as a *feature*, not just disabled it — the `grafana['enable']` key no longer exists and hard-fails `reconfigure` if set, unlike other monitoring toggles which are still valid no-ops. +- `registry.gitlab.com` access is geo-blocked under U.S. sanctions for several countries at the infrastructure level — no account or token change works around it; the fix is sourcing images from an unaffected registry (Docker Hub, in this case) instead. +- GitLab's Docker-executor helper image pull happens on the **host's** Docker daemon before the job container's own networking is even relevant — a failure here isn't a job-container network issue, it's a host-level registry pull issue. +- Community Grafana dashboards built against GitLab's *bundled* Omnibus Prometheus setup don't work against an externally-scraped `/-/metrics` endpoint — the metric names differ. Worth checking a dashboard's expected data source shape before assuming an empty panel means broken config. +- Files created inside a bind-mounted volume by a root-running container (GitLab's `config/`, `data/backups/`) are root-owned on the host — backup scripts touching these need to run as root and explicitly `chown` their own output back to the regular user afterward, or every later interaction with the backup needs `sudo`. diff --git a/docs/Home.md b/docs/Home.md index 8f1a1dd..d550c16 100644 --- a/docs/Home.md +++ b/docs/Home.md @@ -16,6 +16,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This - [Semaphore](Semaphore.md) - [DockScope](DockScope.md) - [Ansible Playbooks](AnsiblePlaybooks.md) +- [GitLab CI/CD](GitLab-CICD.md) - [Elastic Stack (ELK) — Lab](ElasticStack.md) ## At a glance @@ -30,6 +31,7 @@ Documentation for a self-hosted homelab running entirely on Docker Compose. This | **Automation** | n8n + **Semaphore** (Ansible Web UI) + **Ansible** (update playbooks) || **Media** | Plex + Sonarr/Radarr/Prowlarr + qBittorrent | | **Disk health** | Hard Disk Sentinel | | **AI Agent** | OpenHands + Ollama | +| **CI/CD** | Self-hosted **GitLab** (Omnibus) + Docker-executor **Runner** | This is a running system, not a demo — screenshots throughout this wiki are from the live dashboards. diff --git a/docs/Monitoring-Stack.md b/docs/Monitoring-Stack.md index 9cb45b7..a0f4312 100644 --- a/docs/Monitoring-Stack.md +++ b/docs/Monitoring-Stack.md @@ -34,6 +34,23 @@ Logs are browsable two ways in Grafana: - **Explore → Loki**, querying by label directly (e.g. `{container="grafana"}`) - **Drilldown → Logs**, a zero-config log browser bundled with Grafana — no dashboard needed to start exploring +## GitLab & Runner metrics + +Self-hosted GitLab (see [GitLab CI/CD](GitLab-CICD.md)) exposes its own Prometheus-format metrics at `/-/metrics`, separate from Omnibus's bundled monitoring stack, which is disabled here in favor of this existing setup. The Runner similarly exposes its own metrics endpoint once `listen_address` is set in `config.toml`. Both are added as standard Prometheus scrape targets: + +```yaml +- job_name: 'gitlab' + metrics_path: '/-/metrics' + static_configs: + - targets: ['192.168.100.6:8929'] + +- job_name: 'gitlab-runner' + static_configs: + - targets: ['192.168.100.6:9252'] +``` + +Visualized on a hand-built Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) rather than GitLab's official community dashboard — that one targets metric names from Omnibus's own bundled Prometheus, which isn't what's running here. + ## Why this matters On a lower-spec host, resource headroom is limited. Having per-container and host-level metrics from day one means a runaway container or a creeping memory leak shows up on a dashboard before it takes the box down — instead of finding out from an outage. Centralized logs close the other half of that gap: when a container *does* misbehave, its logs are searchable and correlated against the same timeline as its resource usage, rather than requiring a separate `docker logs` per container after the fact. diff --git a/docs/images/gitlab-instance-overview.png b/docs/images/gitlab-instance-overview.png new file mode 100644 index 0000000..b9aee2d Binary files /dev/null and b/docs/images/gitlab-instance-overview.png differ diff --git a/docs/images/gitlab-runner-online.png b/docs/images/gitlab-runner-online.png new file mode 100644 index 0000000..743ae0b Binary files /dev/null and b/docs/images/gitlab-runner-online.png differ