added readme for my first dashboard
This commit is contained in:
28
dashboards/dashboards-README.md
Normal file
28
dashboards/dashboards-README.md
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
# Dashboards
|
||||||
|
|
||||||
|
Grafana dashboard exports, kept as JSON so they're versioned alongside the infrastructure they visualize instead of only existing inside Grafana's own database.
|
||||||
|
|
||||||
|
## `gitlab-application-runner-metrics.json`
|
||||||
|
|
||||||
|
Custom-built dashboard for the self-hosted GitLab instance and its Runner — see [GitLab CI/CD](../docs/GitLab-CICD.md) for the full write-up of why this is hand-built rather than an imported community dashboard.
|
||||||
|
|
||||||
|
**Data sources required:** Prometheus (metrics) and Loki (logs), both already provisioned by the [monitoring stack](../docker-compose/monitoring-grafana-promethues-cadvisor-node-exporter/).
|
||||||
|
|
||||||
|
### Rows
|
||||||
|
|
||||||
|
| Row | Panels |
|
||||||
|
|---|---|
|
||||||
|
| **GitLab Rails & Puma Overview** | Request queue latency (p95 / avg), Puma memory watchdog limit, live nginx/workhorse logs (Loki) correlated against the same timeline |
|
||||||
|
| **Database & Redis Performance** | Postgres and Redis metrics |
|
||||||
|
| **Cache & CI Metrics** | Rails cache performance, CI pipeline metrics |
|
||||||
|
| **GitLab Runner Metrics** | Active jobs vs. concurrency limit, job execution duration, job errors/sec, Runner process memory footprint |
|
||||||
|
|
||||||
|
Pulling live logs into the same row as request latency (rather than a separate Loki-only dashboard) was a deliberate choice — when Puma queue latency spikes, the correlated nginx/workhorse log lines are visible in the same view without switching to Explore.
|
||||||
|
|
||||||
|
### Importing
|
||||||
|
|
||||||
|
**Grafana → Dashboards → New → Import → Upload JSON file**, select `gitlab-application-runner-metrics.json`, map the `DS_PROMETHEUS` variable to your Prometheus data source when prompted.
|
||||||
|
|
||||||
|
### Metric naming note
|
||||||
|
|
||||||
|
Panels query GitLab's own `/-/metrics` endpoint directly (e.g. `gitlab_rails_queue_duration_seconds_bucket`, `process_resident_memory_bytes{job="gitlab-runner"}`) — these metric names come from an *externally scraped* GitLab instance with its bundled Omnibus monitoring disabled. If reusing this dashboard against a GitLab instance running its own internal Prometheus (Omnibus monitoring enabled), the metric names and job labels won't match and panels will show no data.
|
||||||
@@ -88,7 +88,7 @@ By default GitLab only allows `127.0.0.1` to scrape metrics; since Prometheus re
|
|||||||
gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
|
gitlab_rails['monitoring_whitelist'] = ['127.0.0.1', '10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16']
|
||||||
```
|
```
|
||||||
|
|
||||||
A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics.
|
A custom Grafana dashboard (`dashboards/gitlab-application-runner-metrics.json`) was built by hand rather than importing GitLab's official community dashboard (ID `10990`) — that dashboard targets metric names from Omnibus's *own* bundled Prometheus setup, which was disabled above, so the metric names don't match what this external Prometheus actually scrapes. Panels cover Rails/Puma request queue latency, Sidekiq/Redis/Postgres performance, and Runner job/memory metrics — plus a live Loki logs panel embedded alongside the Rails/Puma metrics, so a latency spike and its corresponding nginx/workhorse log lines are visible in the same view. See [`dashboards/README.md`](../dashboards/README.md) for the full panel breakdown.
|
||||||
|
|
||||||
Logs flow into the existing Loki/Alloy pipeline automatically, since Alloy discovers all running containers via the Docker socket — no GitLab-specific log config needed.
|
Logs flow into the existing Loki/Alloy pipeline automatically, since Alloy discovers all running containers via the Docker socket — no GitLab-specific log config needed.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user