From 8eb3670585ce6ae373d4ec048c6096a66c285a5a Mon Sep 17 00:00:00 2001 From: rezayaghobi-dev Date: Fri, 7 Aug 2026 23:59:10 +0330 Subject: [PATCH] added readme for my first dashboard --- dashboards/dashboards-README.md | 28 ++++++++++++++++++++++++++++ docs/GitLab-CICD.md | 2 +- 2 files changed, 29 insertions(+), 1 deletion(-) create mode 100644 dashboards/dashboards-README.md diff --git a/dashboards/dashboards-README.md b/dashboards/dashboards-README.md new file mode 100644 index 0000000..9c22640 --- /dev/null +++ b/dashboards/dashboards-README.md @@ -0,0 +1,28 @@ +# Dashboards + +Grafana dashboard exports, kept as JSON so they're versioned alongside the infrastructure they visualize instead of only existing inside Grafana's own database. + +## `gitlab-application-runner-metrics.json` + +Custom-built dashboard for the self-hosted GitLab instance and its Runner — see [GitLab CI/CD](../docs/GitLab-CICD.md) for the full write-up of why this is hand-built rather than an imported community dashboard. + +**Data sources required:** Prometheus (metrics) and Loki (logs), both already provisioned by the [monitoring stack](../docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/). + +### Rows + +| Row | Panels | +|---|---| +| **GitLab Rails & Puma Overview** | Request queue latency (p95 / avg), Puma memory watchdog limit, live nginx/workhorse logs (Loki) correlated against the same timeline | +| **Database & Redis Performance** | Postgres and Redis metrics | +| **Cache & CI Metrics** | Rails cache performance, CI pipeline metrics | +| **GitLab Runner Metrics** | Active jobs vs. concurrency limit, job execution duration, job errors/sec, Runner process memory footprint | + +Pulling live logs into the same row as request latency (rather than a separate Loki-only dashboard) was a deliberate choice — when Puma queue latency spikes, the correlated nginx/workhorse log lines are visible in the same view without switching to Explore. + +### Importing + +**Grafana → Dashboards → New → Import → Upload JSON file**, select `gitlab-application-runner-metrics.json`, map the `DS_PROMETHEUS` variable to your Prometheus data source when prompted. + +### Metric naming note + +Panels query GitLab's own `/-/metrics` endpoint directly (e.g. `gitlab_rails_queue_duration_seconds_bucket`, `process_resident_memory_bytes{job="gitlab-runner"}`) — these metric names come from an *externally scraped* GitLab instance with its bundled Omnibus monitoring disabled. If reusing this dashboard against a GitLab instance running its own internal Prometheus (Omnibus monitoring enabled), the metric names and job labels won't match and panels will show no data. diff --git a/docs/GitLab-CICD.md b/docs/GitLab-CICD.md index d7572db..d0b7067 100644 --- a/docs/GitLab-CICD.md +++ b/docs/GitLab-CICD.md @@ -88,7 +88,7 @@ By default GitLab only allows `127.0.0.1` to scrape metrics; since Prometheus re gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'] ``` -A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics. +A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics — plus a live Loki logs panel embedded alongside the Rails/Puma metrics, so a latency spike and its corresponding nginx/workhorse log lines are visible in the same view. See [`dashboards/README.md`](../dashboards/README.md) for the full panel breakdown. Logs flow into the existing Loki/Alloy pipeline automatically, since Alloy discovers all running containers via the Docker socket — no GitLab-specific log config needed.