diff --git a/README.md b/README.md new file mode 100644 index 0000000..8afbb0a --- /dev/null +++ b/README.md @@ -0,0 +1,131 @@ +# DevOps practical test + +A small Python web service, containerized with Docker Compose, built and deployed by a GitLab CI/CD pipeline, monitored with Prometheus, Alertmanager and Grafana, and protected with iptables rules. + +The project covers four areas: **containerization**, **CI/CD**, **monitoring and alerting**, and **network security with restart policies**. Each area has its own detailed guide in [`docs/`](docs/). + +## Documentation + +| Guide | What you will find | +|---|---| +| [Web application and containerization](docs/webapp.md) | The FastAPI service, tests, Dockerfile line by line, Compose file, health check, restart policy and the crash-recovery demonstration | +| [Monitoring and alerting](docs/monitoring.md) | Prometheus, Alertmanager and Grafana: every configuration file explained, the three alert rules, how to trigger and read an alert | +| [GitLab CI/CD pipeline](docs/gitlab.md) | The pipeline stage by stage, the runner setup, how to trigger it, failure recovery | +| [Network security with iptables](docs/iptables.md) | Why Docker needs `DOCKER-USER` rules, the rule set, the script, the before/after test and counter evidence | + +## Requirements and where they are covered + +| # | Requirement | Implementation | Guide | Evidence | +|---|---|---|---|---| +| 1 | Containerize the service with Docker Compose | `webapp/Dockerfile`, `webapp/compose.yml` | [webapp.md](docs/webapp.md) | [Service started via Compose](docs/screenshots/web-app.png) | +| 2 | GitLab CI/CD pipeline that builds, tests and deploys | `gitlab/.gitlab-ci.yml` (included by the root `.gitlab-ci.yml`) | [gitlab.md](docs/gitlab.md) | [Passing pipeline](docs/screenshots/full-green-pipline.png), [test report](docs/screenshots/test-pipline.png) | +| 3 | Prometheus monitoring and alerting | `monitoring/` (Prometheus, Alertmanager, Grafana configs) | [monitoring.md](docs/monitoring.md) | [Targets up](docs/screenshots/scraps.png), [rules](docs/screenshots/alerts.png), [alert firing](docs/screenshots/web-app-down.png), [Grafana dashboard](docs/screenshots/dashboard-webapp.png) | +| 4 | iptables rules restricting external access | `iptables/firewall.sh` | [iptables.md](docs/iptables.md) | [Test from a second machine](docs/screenshots/iptabels-afterrule.png), [rule counters](docs/screenshots/firewall-rule.png) | +| 5 | Restart policy and automatic restart after a kill | `restart: unless-stopped` in `webapp/compose.yml` | [webapp.md, section 7](docs/webapp.md#7-restart-policy-and-crash-recovery) | [Restart after kill](docs/screenshots/restart-proof.png) | + +## Architecture + +``` + Developer laptop Home server + +------------------------------+ git push +-----------------------------------+ + | webapp + monitoring stack | ----------------> | GitLab CE (behind Traefik, TLS) | + | - webapp :8000 | | | | + | - Prometheus :9090 | | v | + | - Alertmanager :9093 | | GitLab runner (Docker executor) | + | - Grafana :3000 | | test -> build -> deploy | + | iptables firewall | | | | + +------------------------------+ | v | + | webapp container :8000 | + +-----------------------------------+ +``` + +The webapp exposes `/metrics`; Prometheus collects them, evaluates the alert rules and passes firing alerts to Alertmanager, which sends them to a webhook receiver. Grafana draws the data. The pipeline tests the code, builds the image and deploys it, and only reports success when the container is healthy. + +## Repository layout + +``` +. +├── README.md this file +├── .gitlab-ci.yml includes gitlab/.gitlab-ci.yml +├── docs/ the four guides and docs/screenshots/ +├── webapp/ FastAPI service, tests, Dockerfile, compose.yml +├── monitoring/ compose.yml, prometheus/, alertmanager/, grafana/ +├── gitlab/ .gitlab-ci.yml, compose.yml of the GitLab stack, runner notes +└── iptables/ firewall.sh +``` + +## Setup + +**Requirements:** Docker with the Compose plugin, Linux. For the firewall: `iptables` and root access. + +```bash +# 1. A network shared by the two stacks (once) +docker network create monitoring-net + +# 2. The web service +cd webapp +docker compose up -d --build + +# 3. Monitoring +cd ../monitoring +docker compose up -d +``` + +Check that everything is running: + +```bash +docker ps +curl localhost:8000/health # {"status":"ok"} +``` + +| Service | URL | Notes | +|---|---|---| +| Webapp | http://localhost:8000 | `/health`, `/metrics`, `/error` (returns 500 on purpose) | +| Prometheus | http://localhost:9090 | Status → Targets, Alerts | +| Alertmanager | http://localhost:9093 | | +| Grafana | http://localhost:3000 | User `admin`, password `admin` (set `GRAFANA_PASSWORD` to change); dashboard "Webapp Overview" | + +Apply the firewall (optional, requires root; undo with `down`): + +```bash +sudo ./iptables/firewall.sh up +sudo ./iptables/firewall.sh status +sudo ./iptables/firewall.sh down +``` + +Stop everything: + +```bash +(cd monitoring && docker compose down) +(cd webapp && docker compose down) +``` + +## Trigger the pipeline + +1. Push the repository to the GitLab project: `git push origin master`. +2. GitLab starts a pipeline automatically. Follow it under **Build → Pipelines**. +3. The stages are **test** (pytest), **build** (Docker image), an optional **publish** and **deploy** (default branch only). The deploy succeeds only when the container reports `healthy`. +4. To run it again without a new commit: **Build → Pipelines → Run pipeline**. + +Details, the runner setup and the pipeline file are in [docs/gitlab.md](docs/gitlab.md). + +## Failure recovery + +| Situation | What to do | +|---|---| +| The webapp container crashed | Nothing: `restart: unless-stopped` restarts it automatically. Check with `docker ps` and `docker logs webapp` | +| The webapp was stopped by hand (`docker stop`) | Start it again: `docker start webapp` (or `docker compose up -d` in `webapp/`) | +| The container is `unhealthy` | `docker logs webapp`, fix the cause, then `docker compose up -d --build` | +| An alert is firing | Read it at http://localhost:9090/alerts and `docker logs alert-logger`; check the Grafana dashboard and `docker logs webapp` | +| Prometheus rules or config changed | Validate with `docker exec prometheus promtool check config /etc/prometheus/prometheus.yml`, then reload with `curl -X POST localhost:9090/-/reload` | +| A pipeline job is stuck "pending" | The runner is offline or not enabled for the project: `docker logs gitlab-runner` and the GitLab runners page | +| A deploy broke the service | Revert the commit and push; the pipeline redeploys the previous version | +| The firewall blocks something unexpected | `sudo ./iptables/firewall.sh down` removes all rules at once | + +More specific tables are at the end of each guide. + +## Notes + +- The Compose files are named `compose.yml`, the current default name that Docker Compose looks for (`docker-compose.yml` also works). +- Screenshots in `docs/screenshots/` were taken during the actual runs described in the guides. +- This is a demonstration setup. Each guide ends with a short list of what would change in production (authentication, TLS, real alert receivers, isolated CI runners, IPv6 firewall rules, and so on). diff --git a/docs/gitlab.md b/docs/gitlab.md new file mode 100644 index 0000000..6deb450 --- /dev/null +++ b/docs/gitlab.md @@ -0,0 +1,242 @@ +# GitLab CI/CD pipeline + +[← Back to the main README](../README.md) + +This document explains the automation around the code: where the repository lives, how the pipeline is defined, what each stage does, how the runner is set up, and how to fix the usual problems. + +> **In plain words:** every time code is pushed, GitLab starts a robot (the *runner*) that tests the code, builds the Docker image and starts the new version on the server, and fails loudly if anything is wrong. Nobody has to remember to do those steps by hand. + +**Contents** + +1. [The environment](#1-the-environment) +2. [How one pipeline run flows](#2-how-one-pipeline-run-flows) +3. [The pipeline file](#3-the-pipeline-file) +4. [Stage by stage](#4-stage-by-stage) +5. [Results](#5-results) +6. [Trigger the pipeline](#6-trigger-the-pipeline) +7. [The runner](#7-the-runner) +8. [Failure recovery](#8-failure-recovery) +9. [Security notes and improvements](#9-security-notes-and-improvements) + +--- + +## 1. The environment + +| Piece | Details | +|---|---| +| GitLab | Self-hosted GitLab CE, reached at `https://gitlab.home.server`, running in Docker behind a Traefik reverse proxy (TLS) | +| Project | `root/devops-test`, branch `master` | +| Container registry | Available at `registry.home.server` (not used by the pipeline, see the `publish` stage) | +| Runner | `gitlab-runner` container with the **Docker executor**, on the same Docker network as GitLab (`web_net`) | +| Deploy target | The same server: the pipeline starts the webapp on the server's Docker | + +The `gitlab/` folder of this repository contains the pipeline file, a copy of the Compose file of the GitLab stack (`compose.yml`, with no secrets) and a short README on registering a runner. + +## 2. How one pipeline run flows + +``` + git push GitLab (gitlab.home.server) + ---------> repository ---------------> creates a pipeline from .gitlab-ci.yml + | + v + runner picks up the jobs, one by one + | + +---------------------+----------------------+----------------------+ + v v v v + [ test ] -------> [ build ] -------> [ publish ] -------> [ deploy ] + pytest in a clean docker build, optional, docker compose up, + Python container tag with commit off by default wait until healthy +``` + +Each job runs inside a fresh, disposable container, so a run never depends on leftovers from a previous one. If a stage fails, the next stages do not run. + +## 3. The pipeline file + +GitLab only reads `.gitlab-ci.yml` from the repository root. To keep CI files in the `gitlab/` folder, the root file just includes the real one: + +```yaml +# .gitlab-ci.yml (repository root) +include: + - local: gitlab/.gitlab-ci.yml +``` + +The real pipeline, `gitlab/.gitlab-ci.yml`: + +```yaml +stages: + - test + - build + - publish + - deploy + +variables: + GIT_DEPTH: "20" + PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip" + +# ---------- 1. TEST ---------- +test: + stage: test + image: python:3.12-slim + cache: + key: pip-cache + paths: + - .cache/pip + script: + - cd webapp + - pip install -r requirements-dev.txt + - pytest -v --junitxml=report.xml + artifacts: + when: always + reports: + junit: webapp/report.xml + +# ---------- 2. BUILD ---------- +build: + stage: build + script: + - docker build -t webapp:latest -t webapp:$CI_COMMIT_SHORT_SHA webapp + - docker image ls webapp + +# ---------- 3. PUBLISH (opt-in: set CI/CD variable PUSH_TO_REGISTRY=true) ---------- +publish: + stage: publish + rules: + - if: '$PUSH_TO_REGISTRY == "true"' + script: + - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY" + - docker tag webapp:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE/webapp:$CI_COMMIT_SHORT_SHA + - docker tag webapp:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE/webapp:latest + - docker push $CI_REGISTRY_IMAGE/webapp:$CI_COMMIT_SHORT_SHA + - docker push $CI_REGISTRY_IMAGE/webapp:latest + +# ---------- 4. DEPLOY (default branch only) ---------- +deploy: + stage: deploy + rules: + - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH' + script: + - docker network inspect monitoring-net >/dev/null 2>&1 || docker network create monitoring-net + - docker compose -f webapp/compose.yml up -d --no-build + - | + for i in $(seq 1 12); do + status=$(docker inspect -f '{{.State.Health.Status}}' webapp) + echo "health: $status" + [ "$status" = "healthy" ] && exit 0 + sleep 5 + done + docker logs webapp --tail 50 + exit 1 + environment: + name: production +``` + +## 4. Stage by stage + +### `test` + +- Runs in the image `python:3.12-slim`, a clean Python environment. +- Installs the test dependencies and runs `pytest`. The pip download cache is kept between runs to save time. +- Writes a JUnit XML report. GitLab reads it and shows every test in the pipeline's **Tests** tab. +- `artifacts: when: always` keeps the report even if a test fails, which is exactly when it is needed. + +### `build` + +- Builds the Docker image from `webapp/Dockerfile`. +- Tags it twice: `webapp:latest` (the newest) and `webapp:` (for example `webapp:8bca986f`), so every image can be traced back to the exact code that made it. +- The runner talks to the server's Docker through the mounted Docker socket (see [section 7](#7-the-runner)), so the image ends up on the same Docker that the deploy stage uses. That is why `deploy` does not need to rebuild. + +### `publish` (optional) + +Pushes the image to the project's container registry. It only runs when the CI/CD variable `PUSH_TO_REGISTRY` is set to `true` (**Settings → CI/CD → Variables**). It is switched off because the task does not need a registry and the server's Docker would first have to trust the registry's certificate. This is why the pipeline graph below shows three stages, not four. + +### `deploy` + +- Runs only on the default branch, so work on other branches is tested and built but never deployed. +- Makes sure the shared network `monitoring-net` exists. +- Starts the service with `docker compose up -d --no-build`, using the image the previous stage built. +- **Verifies the result:** it asks Docker for the container's health status every 5 seconds, up to 12 times (one minute). If the container becomes `healthy`, the job succeeds. If not, the job prints the container logs and fails. A deploy that "ran the command" but left a broken service therefore turns the pipeline red. + +## 5. Results + +![Pipeline #33 passed: test, build and deploy all green](screenshots/full-green-pipline.png) + +*Pipeline #33 for commit `8bca986f` on `master`: the three stages **test**, **build** and **deploy** passed in sequence. The optional `publish` stage does not appear because its rule did not match.* + +![Pipeline test report: 4 tests, 0 failures, 100% success](screenshots/test-pipline.png) + +*The **Tests** tab, fed by the JUnit report: 4 tests, 0 failures, 0 errors, 100% success rate.* + +After a green pipeline, the service is running on the server. Check it from any machine on the home network: + +```bash +curl http://:8000/health +``` + +## 6. Trigger the pipeline + +| How | When | +|---|---| +| `git push` | The normal way. Any push to any branch creates a pipeline; only the default branch deploys | +| **Build → Pipelines → Run pipeline** in GitLab | Re-run without a new commit | +| The retry button on a pipeline or job | After fixing a temporary problem such as a network error | + +First-time setup of the repository connection: + +```bash +git remote add origin ssh://git@gitlab.home.server:2224/root/devops-test.git +git push -u origin master +``` + +(SSH on this GitLab uses port 2224. Your public key must be added under **Preferences → SSH Keys**. As an alternative, use the HTTPS URL with an access token.) + +## 7. The runner + +The runner is the worker that executes the jobs. It runs as the container `gitlab-runner` next to GitLab. Its key settings: + +| Setting | Value | Why | +|---|---|---| +| Executor | `docker` | Every job runs in its own container | +| GitLab URL | `http://gitlab:80` | Direct, internal address over the Docker network; no DNS or certificate needed | +| Network mode | `web_net` | Lets job containers reach GitLab to clone the code | +| Default image | `docker:cli` | Contains the `docker` command and the Compose plugin, used by `build` and `deploy` | +| Volume | `/var/run/docker.sock` | Gives jobs access to the server's Docker engine | +| Helper image | `gitlab/gitlab-runner-helper:x86_64-latest` | Pulled from Docker Hub, because `registry.gitlab.com` (the default source) is not reachable from this network | + +Registering a runner (create it first in **Admin → CI/CD → Runners → New instance runner**, then use its token): + +```bash +docker exec gitlab-runner gitlab-runner register --non-interactive \ + --url http://gitlab:80 --token \ + --executor docker --docker-image docker:cli \ + --docker-network-mode web_net \ + --docker-volumes /var/run/docker.sock:/var/run/docker.sock \ + --clone-url http://gitlab:80 +``` + +Then add the helper image in the runner's `config.toml`, under `[runners.docker]`: + +```toml +helper_image = "gitlab/gitlab-runner-helper:x86_64-latest" +``` + +> Never paste a runner token (`glrt-...`) into chats, tickets or the repository. If one leaks, delete the runner in the GitLab admin area and register a new one. + +## 8. Failure recovery + +| Symptom | Cause and fix | +|---|---| +| Job stays **pending** with "no active runners can run this job" | The runner is offline, paused, not enabled for the project, or does not accept untagged jobs. Check **Admin → CI/CD → Runners** and the project's **Settings → CI/CD → Runners**; read `docker logs gitlab-runner` | +| Runner log shows `connection refused` to GitLab | The runner's URL is outdated (this happened after GitLab was put behind Traefik). Register the runner with `http://gitlab:80` as above | +| Job fails pulling `registry.gitlab.com/.../gitlab-runner-helper` | Set `helper_image` to the Docker Hub image (section 7) | +| Job fails pulling `python:3.12-slim` or `docker:cli` | Docker Hub is unreachable from the server. Pre-pull the images on the server: `docker pull python:3.12-slim && docker pull docker:cli` | +| `git push` says `Permission denied (publickey)` | Add your SSH public key in GitLab, or switch the remote to HTTPS | +| `test` fails | Open the job log, then reproduce locally with `cd webapp && pytest -v` | +| `deploy` fails at the health check | The job prints the container logs. Fix the problem, push again. On the server: `docker logs webapp` | +| Pipeline did not start at all | Open **Build → Pipeline editor**; it shows YAML or include errors | +| A bad version was deployed | Revert the commit (`git revert `) and push; the pipeline redeploys the previous behavior | + +## 9. Security notes and improvements + +- **Docker socket mount.** A job that can use the Docker socket can control the whole host. That is acceptable on a private home server with trusted code. In production, use isolated runners, Docker-in-Docker, or a daemonless builder such as Kaniko, and run deploy jobs only from protected branches. +- **Secrets** (registry or deploy credentials) belong in masked, protected CI/CD variables, never in the repository. +- Possible improvements: push images to the registry and deploy by pulling a tagged version (enables rollback to a known tag), add an image vulnerability scan stage, add a lint stage (`ruff`, `hadolint`), and run deploys against a staging environment first. diff --git a/docs/iptables.md b/docs/iptables.md new file mode 100644 index 0000000..9fa6ba6 --- /dev/null +++ b/docs/iptables.md @@ -0,0 +1,242 @@ +# Network security with iptables + +[← Back to the main README](../README.md) + +This document explains the firewall rules that restrict external access, why Docker needs special handling, how to apply and remove the rules, and the evidence that they work. + +> **In plain words:** a firewall is a security guard at the door of the computer. The guard has a list of rules ("let replies in", "only people from the home network may open door 8000", "everyone else is turned away"). `iptables` is the tool that writes that list on Linux. + +**Contents** + +1. [Goal](#1-goal) +2. [Concepts you need](#2-concepts-you-need) +3. [The rules](#3-the-rules) +4. [Use the script](#4-use-the-script) +5. [Test plan and results](#5-test-plan-and-results) +6. [Evidence from the rule counters](#6-evidence-from-the-rule-counters) +7. [The script](#7-the-script) +8. [Limitations](#8-limitations) +9. [Troubleshooting](#9-troubleshooting) + +--- + +## 1. Goal + +The machine runs four services that publish ports: the webapp (8000), Prometheus (9090), Alertmanager (9093) and Grafana (3000). The requirement is to restrict external access: + +| Port | Service | Who may connect | +|---|---|---| +| 8000 | Webapp | Computers on the local network | +| 9090 | Prometheus | Only this machine (localhost) | +| 9093 | Alertmanager | Only this machine | +| 3000 | Grafana | Only this machine | + +Everything else arriving from the network is dropped, except normal replies to the machine's own connections. + +## 2. Concepts you need + +**Chains and rules.** iptables has *chains*, ordered lists of rules. A packet is compared with the rules from top to bottom, and the first matching rule decides: `ACCEPT` (let it through), `DROP` (silently discard) or `RETURN` (no decision here, go back to the calling chain). + +**Connection tracking (`conntrack`).** Linux remembers connections. A packet that belongs to a conversation the machine started (`ESTABLISHED`) or is closely related to one (`RELATED`) can safely be allowed; this is how web pages you request can come back through a strict firewall. + +**Why Docker needs special handling.** When a container publishes a port (`-p 9090:9090`), Docker rewrites the destination of incoming packets to the container's own address (this is called DNAT). Those packets are then *forwarded* to the container instead of being delivered to the host, so they never pass through the `INPUT` chain that most firewall guides use. The place Docker leaves for your own rules is the **`DOCKER-USER`** chain, which is checked before Docker's own rules. + +``` + from the network + | + [ PREROUTING: Docker rewrites the destination (DNAT) ] + | + +--> addressed to the host itself ------> INPUT ........ chain DEVOPS-FW (this project) + | + +--> published container port ----------> FORWARD -> DOCKER-USER ... our container rules +``` + +Because the destination was rewritten, the rules use `--ctorigdstport`, which looks up the **original** port the client asked for. + +## 3. The rules + +Two sets of rules are created. Both only look at traffic arriving on the **external network interface** (for example `enp47s0`), so the machine's own traffic, Docker's internal networks and loopback are not affected. + +### 3.1 Container ports: chain `DOCKER-USER` + +| # | Rule | Meaning | +|---|---|---| +| 1 | `-i EXT_IF -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN` | Replies to connections the machine opened are fine | +| 2 | `-i EXT_IF -s LAN -p tcp --ctorigdstport 8000 --ctdir ORIGINAL -j RETURN` | The local network may open port 8000 | +| 3 | `-i EXT_IF -j DROP` | Anything else arriving from outside towards a container is dropped (this blocks 9090, 9093 and 3000) | +| 4 | `-j RETURN` | Everything else (traffic between containers, outgoing traffic) continues to Docker's normal rules | + +Connections from the machine itself (`localhost:9090`) do not arrive on the external interface, so they keep working. + +### 3.2 The host itself: chain `DEVOPS-FW` (jumped to from `INPUT`) + +| # | Rule | Meaning | +|---|---|---| +| 1 | `! -i EXT_IF -j RETURN` | Not our business if it did not arrive on the external interface (loopback, Docker bridges and VPN stay untouched) | +| 2 | `-m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT` | Replies are allowed | +| 3 | `-p icmp -j ACCEPT` | Ping and network diagnostics are allowed | +| 4 | `-p udp --sport 67 --dport 68 -j ACCEPT` | DHCP answers, so the machine can still get its network address | +| 5 | `-p tcp --dport 22 -s LAN -j ACCEPT` | SSH only from the local network | +| 6 | `-j DROP` | Everything else unsolicited is dropped | + +A dedicated chain was used on purpose. Other software (here: Tailscale) adds its own rules to `INPUT`, and a separate chain can be added and removed without touching them. + +## 4. Use the script + +The script is `iptables/firewall.sh`. It needs root. + +```bash +cd iptables +sudo ./firewall.sh up # apply the rules and print them +sudo ./firewall.sh status # show the rules with their packet counters +sudo ./firewall.sh down # remove everything, back to the default state +``` + +By default the script detects the external interface from the default route and the local network from that interface's address. Override them if needed: + +```bash +sudo EXT_IF=enp47s0 LAN=192.168.100.0/24 ./firewall.sh up +``` + +After `up` it prints, for example: `External interface: enp47s0 | LAN: 192.168.100.15/24`. + +`down` is always safe to run: it removes the `DEVOPS-FW` chain and the jump to it, and resets `DOCKER-USER` to its default single `RETURN` rule. + +## 5. Test plan and results + +The test needs a **second machine** on the network, because connections from the machine to itself are never blocked. Here the home server (`192.168.100.6`) acted as the outside client, and the laptop (`192.168.100.15`) ran the services and the firewall. + +The test, run on the second machine (it prints the HTTP status code, `000` means the connection timed out after 3 seconds): + +```bash +for p in 8000 9090 9093 3000 +do + printf "port $p: " + curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://192.168.100.15:$p/ +done +``` + +| Port | Service | Before the firewall | After `firewall.sh up` | +|---|---|---|---| +| 8000 | Webapp | `200` | `200` (allowed from the LAN) | +| 9090 | Prometheus | `302` (redirect to its UI) | `000` (blocked) | +| 9093 | Alertmanager | `200` | `000` (blocked) | +| 3000 | Grafana | `302` (redirect to the login page) | `000` (blocked) | + +![The firewall is applied on the laptop (left); the second machine tests all four ports (right)](screenshots/iptabels-afterrule.png) + +*Left: `sudo ./firewall.sh up` reports the external interface and the LAN and prints the rules. Right: the test from the server shows port 8000 answering `200` while 9090, 9093 and 3000 time out (`000`).* + +The services remain fully usable on the machine itself: + +```bash +curl -s localhost:9090/-/healthy # Prometheus Server is Healthy. +curl -s localhost:3000/api/health # Grafana answers {"database": "ok", ...} +``` + +## 6. Evidence from the rule counters + +`iptables -L -v` shows how many packets each rule has handled. Counters that are above zero on the DROP rules prove that the rules are actually blocking traffic. + +![firewall.sh status with packet counters after the test](screenshots/firewall-rule.png) + +*The `status` output after the test:* + +| Where | Rule | Packets | What it shows | +|---|---|---|---| +| `INPUT` | #1 jump to `DEVOPS-FW` | 4226 | `DEVOPS-FW` is checked first, before Tailscale's rules (`ts-input`) | +| `DEVOPS-FW` | #2 `RELATED,ESTABLISHED` | 3753 | Replies to the machine's own connections are allowed through | +| `DEVOPS-FW` | #6 `DROP` | 190 | Unsolicited inbound packets nobody allowed were dropped | +| `DOCKER-USER` | #2 LAN to port 8000 | 1 | The allowed connection to the webapp passed | +| `DOCKER-USER` | #3 `DROP` | 9 | The connection attempts to the blocked ports 9090, 9093 and 3000 (TCP retries the first packet) | + +## 7. The script + +
+Show iptables/firewall.sh + +```bash +#!/usr/bin/env bash +# Restrict external access to this host and to Docker-published ports. +# Usage: sudo ./firewall.sh up | down | status +set -euo pipefail + +[ "$EUID" -eq 0 ] || { echo "Run with sudo"; exit 1; } + +EXT_IF="${EXT_IF:-$(ip route show default | awk '/default/ {print $5; exit}')}" +[ -n "$EXT_IF" ] || { echo "No default route found, set EXT_IF="; exit 1; } +LAN="${LAN:-$(ip -o -4 addr show dev "$EXT_IF" | awk '{print $4; exit}')}" +APP_PORT=8000 +CHAIN=DEVOPS-FW + +down() { + while iptables -D INPUT -j "$CHAIN" 2>/dev/null; do :; done + iptables -F "$CHAIN" 2>/dev/null || true + iptables -X "$CHAIN" 2>/dev/null || true + iptables -F DOCKER-USER 2>/dev/null || true + iptables -A DOCKER-USER -j RETURN 2>/dev/null || true +} + +up() { + down + echo "External interface: $EXT_IF | LAN: $LAN" + + # Host traffic (INPUT): only packets arriving on the external interface are filtered + iptables -N "$CHAIN" + iptables -A "$CHAIN" ! -i "$EXT_IF" -j RETURN + iptables -A "$CHAIN" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT + iptables -A "$CHAIN" -p icmp -j ACCEPT + iptables -A "$CHAIN" -p udp --sport 67 --dport 68 -j ACCEPT + iptables -A "$CHAIN" -p tcp --dport 22 -s "$LAN" -j ACCEPT + iptables -A "$CHAIN" -j DROP + iptables -I INPUT 1 -j "$CHAIN" + + # Container ports (DOCKER-USER): after DNAT only the original destination port tells us what was requested + iptables -F DOCKER-USER + iptables -A DOCKER-USER -i "$EXT_IF" -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN + iptables -A DOCKER-USER -i "$EXT_IF" -s "$LAN" -p tcp -m conntrack --ctorigdstport "$APP_PORT" --ctdir ORIGINAL -j RETURN + iptables -A DOCKER-USER -i "$EXT_IF" -j DROP + iptables -A DOCKER-USER -j RETURN +} + +status() { + echo "== INPUT (top) =="; iptables -L INPUT -n -v --line-numbers | head -5 + echo "== $CHAIN =="; iptables -L "$CHAIN" -n -v --line-numbers 2>/dev/null || echo "not active" + echo "== DOCKER-USER =="; iptables -L DOCKER-USER -n -v --line-numbers +} + +case "${1:-}" in + up) up; status ;; + down) down; status ;; + status) status ;; + *) echo "Usage: sudo $0 up|down|status"; exit 1 ;; +esac +``` + +
+ +![The script as written on the laptop, and the laptop's network interfaces](screenshots/laptop-before-iptabels.png) + +*The terminal shows the script as saved on the laptop, and `ip -4 -br addr`, which lists the interfaces: `enp47s0` (the wired network, `192.168.100.15/24`), `tailscale0` (VPN), and the Docker bridges.* + +## 8. Limitations + +| Limitation | Detail | +|---|---| +| **Not persistent** | The rules disappear after a reboot (and `down` removes them on purpose). To keep them, run the script at boot (for example from a systemd unit) or use `iptables-persistent` or nftables | +| **IPv4 only** | Docker also publishes ports on IPv6 (`[::]:8000`). A production setup needs the same rules in `ip6tables`, or IPv6 disabled on the host | +| **One external interface** | Only the interface detected as `EXT_IF` is filtered. Traffic arriving through another interface, such as a VPN (`tailscale0`) or a second network card, is not touched by these rules | +| **Port 8000 is fixed** | Change `APP_PORT` in the script if the webapp port changes | +| **`DOCKER-USER` is replaced** | `up` empties the chain before adding its rules, so any other custom rule in it would be lost | +| **Not a complete host firewall** | Other services listening directly on the host (not in containers) are only covered by the `DEVOPS-FW` rules | + +## 9. Troubleshooting + +| Symptom | Cause and fix | +|---|---| +| The second machine cannot reach even port 8000, and `ping` fails, **before** the firewall | A VPN is routing the replies another way. This happened with Tailscale: the laptop accepted the home network as a "subnet route" and answered through the tunnel instead of the cable. `ip route get ` shows the interface used. Fix for the test: `sudo tailscale set --accept-routes=false` (turn it back on afterwards with `=true`) | +| Locked out or something broke | `sudo ./firewall.sh down` removes everything immediately | +| `iptables: No chain/target/match by that name` for `DOCKER-USER` | Docker is not running. Start Docker first | +| Rules seem ignored | Another firewall manager may be active: `sudo ufw status` should say `inactive` | +| `Run with sudo` | The script needs root to change firewall rules | +| Wrong interface detected | Pass it explicitly: `sudo EXT_IF= ./firewall.sh up` (list names with `ip -br addr`) | diff --git a/docs/monitoring.md b/docs/monitoring.md new file mode 100644 index 0000000..ed366d8 --- /dev/null +++ b/docs/monitoring.md @@ -0,0 +1,320 @@ +# Monitoring and alerting + +[← Back to the main README](../README.md) + +This document explains how the web service is monitored: how numbers are collected (Prometheus), how problems are detected (alert rules), how people are told (Alertmanager), and how the data is shown (Grafana). + +> **In plain words:** Prometheus is a collector that visits the web service every 15 seconds and writes down its numbers. Alert rules are questions Prometheus asks over and over ("is the service down?"). When the answer is yes for long enough, Alertmanager sends a message. Grafana draws the numbers as charts. + +**Contents** + +1. [Architecture](#1-architecture) +2. [Files in `monitoring/`](#2-files-in-monitoring) +3. [How each part is configured](#3-how-each-part-is-configured) +4. [Start the stack](#4-start-the-stack) +5. [Verify it works](#5-verify-it-works) +6. [Trigger and read an alert](#6-trigger-and-read-an-alert) +7. [Grafana](#7-grafana) +8. [Troubleshooting](#8-troubleshooting) +9. [What would change in production](#9-what-would-change-in-production) + +--- + +## 1. Architecture + +``` + scrapes /metrics every 15 s + +-----------+ <------------------------------- +------------+ + | webapp | | Prometheus | evaluates the alert rules + | :8000 | | :9090 | every 15 s + +-----------+ +-----+------+ + | sends alerts that are firing + v + +---------------+ webhook +--------------+ + | Alertmanager | ---------> | alert-logger | + | :9093 | | prints them | + +---------------+ +--------------+ + + +-----------+ asks Prometheus questions (PromQL) + | Grafana | ----------------------------------> Prometheus + | :3000 | + +-----------+ +``` + +All containers share the Docker network `monitoring-net`, so they find each other by name (`webapp:8000`, `prometheus:9090`, `alertmanager:9093`). Only the ports in the diagram are published on the host. + +| Component | Image | Role | +|---|---|---| +| Prometheus | `prom/prometheus:v2.53.0` | Collects and stores metrics, evaluates alert rules | +| Alertmanager | `prom/alertmanager:v0.27.0` | Groups alerts and delivers them to a receiver | +| alert-logger | `mendhak/http-https-echo` | A tiny receiver that prints every notification it gets. It stands in for e-mail or Slack, so the project needs no external account | +| Grafana | `grafana/grafana:11.1.0` | Dashboards | + +## 2. Files in `monitoring/` + +``` +monitoring/ +├── compose.yml starts the four containers +├── prometheus/ +│ ├── prometheus.yml what to scrape, where to send alerts +│ └── alert_rules.yml the alert rules +├── alertmanager/ +│ └── alertmanager.yml how alerts are routed +└── grafana/ + ├── provisioning/ + │ ├── datasources/prometheus.yml connects Grafana to Prometheus automatically + │ └── dashboards/dashboards.yml tells Grafana to load dashboards from a folder + └── dashboards/ + └── webapp.json the "Webapp Overview" dashboard +``` + +Everything is configured by files, so the whole setup can be recreated from the repository without clicking in any web interface. + +## 3. How each part is configured + +### 3.1 `compose.yml` + +Four services (`prometheus`, `alertmanager`, `alert-logger`, `grafana`), all with `restart: unless-stopped`, all on the external network `monitoring-net`. Configuration files are mounted read-only into the containers (`:ro`). Three named volumes (`prometheus-data`, `alertmanager-data`, `grafana-data`) keep the data when containers are recreated. Prometheus keeps 7 days of history (`--storage.tsdb.retention.time=7d`) and has `--web.enable-lifecycle` on, which allows reloading its configuration without a restart. + +### 3.2 `prometheus/prometheus.yml` + +```yaml +global: + scrape_interval: 15s + evaluation_interval: 15s + +alerting: + alertmanagers: + - static_configs: + - targets: ["alertmanager:9093"] + +rule_files: + - /etc/prometheus/alert_rules.yml + +scrape_configs: + - job_name: prometheus + static_configs: + - targets: ["localhost:9090"] + + - job_name: webapp + metrics_path: /metrics + static_configs: + - targets: ["webapp:8000"] +``` + +- **scrape_interval**: how often targets are visited. **evaluation_interval**: how often alert rules are checked. +- **alerting**: where firing alerts are sent. +- **scrape_configs**: two targets. Prometheus monitors itself, and the `webapp` job reads `http://webapp:8000/metrics`. + +### 3.3 `prometheus/alert_rules.yml` + +```yaml +groups: + - name: webapp + rules: + - alert: WebappDown + expr: up{job="webapp"} == 0 + for: 30s + labels: + severity: critical + annotations: + summary: "Webapp is down" + description: "Prometheus cannot scrape the webapp for more than 30 seconds." + + - alert: HighErrorRate + expr: sum(rate(http_requests_total{status=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) > 0.05 + for: 1m + labels: + severity: warning + annotations: + summary: "More than 5% of requests are failing" + description: "The 5xx ratio has been above 5% for 1 minute." + + - alert: HighLatencyP95 + expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) > 0.5 + for: 2m + labels: + severity: warning + annotations: + summary: "p95 latency above 500ms" + description: "95% of requests take longer than 0.5s." +``` + +What the three rules ask, in plain words: + +| Alert | The question | Fires after | +|---|---|---| +| `WebappDown` | `up` is 1 when Prometheus could read the target and 0 when it could not. "Is the webapp unreachable?" | 30 s | +| `HighErrorRate` | `rate(...[1m])` is "per second, averaged over the last minute". Errors per second divided by all requests per second is the share of failures. "Are more than 5% failing?" | 1 min | +| `HighLatencyP95` | `histogram_quantile(0.95, ...)` is the time that 95 out of 100 requests beat. "Are the slowest 5% taking longer than half a second?" | 2 min | + +The `for:` delay prevents false alarms from a single bad reading. An alert goes through three states: **inactive** (all fine), **pending** (condition true, waiting out the `for:` time) and **firing** (alert sent). + +The rules alert on what users feel (down, failing, slow), not on internal causes such as CPU usage. + +### 3.4 `alertmanager/alertmanager.yml` + +```yaml +route: + receiver: logger + group_by: ["alertname"] + group_wait: 10s + group_interval: 1m + repeat_interval: 1h + +receivers: + - name: logger + webhook_configs: + - url: http://alert-logger:8080/ + send_resolved: true +``` + +- **group_by**: alerts with the same name are bundled into one message. +- **group_wait: 10s**: after the first alert arrives, wait 10 s to bundle related alerts before sending. +- **group_interval: 1m**: minimum time between messages about changes in the same group. +- **repeat_interval: 1h**: re-send a still-firing alert every hour. +- **send_resolved: true**: also notify when the problem is over. + +### 3.5 Grafana provisioning + +- `datasources/prometheus.yml` registers Prometheus (`http://prometheus:9090`) as the default data source. +- `dashboards/dashboards.yml` tells Grafana to load every dashboard file in `/var/lib/grafana/dashboards`. +- `dashboards/webapp.json` is the **Webapp Overview** dashboard with four panels: webapp up (1 = yes), request rate by path, 5xx error ratio, and p95 latency. + +## 4. Start the stack + +The webapp stack must exist first, because both stacks use the same network: + +```bash +docker network create monitoring-net # only once +cd webapp && docker compose up -d --build +cd ../monitoring && docker compose up -d +docker compose ps +``` + +![Pulling the images and starting the four containers](screenshots/monitoring-cli.png) + +*The first `docker compose pull` hit a temporary "403 Forbidden" while downloading one layer from Docker Hub's content network; running it again succeeded. Then `docker compose up -d` creates the three volumes and starts all four containers.* + +Check the configuration for mistakes before or after starting: + +```bash +docker exec prometheus promtool check config /etc/prometheus/prometheus.yml +``` + +## 5. Verify it works + +| What | Where | +|---|---| +| Prometheus targets | http://localhost:9090/targets | +| Alert rules | http://localhost:9090/alerts | +| Alertmanager | http://localhost:9093 | +| Grafana | http://localhost:3000 (user `admin`, password `admin` unless `GRAFANA_PASSWORD` is set) | + +**Targets.** Both `prometheus` and `webapp` must be **UP**. + +![Prometheus targets page with both targets UP](screenshots/scraps.png) + +*`webapp` is scraped at `http://webapp:8000/metrics` (the container name resolves over the shared network). The scrape takes about 2 ms.* + +**Alert rules.** All three rules are loaded and inactive while everything is healthy. + +![Prometheus alerts page: three rules loaded, all inactive](screenshots/alerts.png) + +*The page shows each rule's expression, its `for` time, labels and annotations, which match `alert_rules.yml`.* + +**Grafana data source.** It was created by configuration, not by hand. + +![Grafana data source page showing "Provisioned data source" and the Prometheus URL](screenshots/promethues-datasource.png) + +*Grafana says "This data source was added by config and cannot be modified using the UI", which proves provisioning works. The URL is `http://prometheus:9090`.* + +## 6. Trigger and read an alert + +### Test 1: the webapp goes down + +`docker stop` is a deliberate stop, so the restart policy does not bring the container back, and the target stays down: + +```bash +docker stop webapp +``` + +Watch http://localhost:9090/alerts. After the `for: 30s` delay, `WebappDown` turns red: + +![Prometheus alerts page: WebappDown is FIRING](screenshots/web-app-down.png) + +*`WebappDown` is FIRING with the labels `instance=webapp:8000`, `job=webapp`, `severity=critical`. The other two rules stay inactive. "Active Since" shows the moment Prometheus first saw the target down.* + +Alertmanager then delivers the notification to the receiver. Read it with: + +```bash +docker logs alert-logger --tail 40 +``` + +The log of the test run contained (shortened): + +``` +"status": "firing", +"labels": { "alertname": "WebappDown", "instance": "webapp:8000", "job": "webapp", "severity": "critical" }, +"annotations": { "summary": "Webapp is down", "description": "Prometheus cannot scrape the webapp for more than 30 seconds." }, +"startsAt": "2026-10-03T16:12:27.768Z", +... +::ffff:172.18.0.4 - - [03/Oct/2026:16:12:37 +0000] "POST / HTTP/1.1" 200 2729 "-" "Alertmanager/0.27.0" +``` + +**The timeline matches the configuration:** + +| Time (UTC) | Event | +|---|---| +| 16:11:57 | Prometheus sees the webapp down; the alert is *pending* ("Active Since" in the screenshot) | +| 16:12:27 | 30 s later (`for: 30s`) the alert starts *firing* (`startsAt`) | +| 16:12:37 | 10 s later (`group_wait: 10s`) Alertmanager delivers it, and the receiver answers `200` | + +Bring the service back; a "resolved" notification follows because of `send_resolved: true`: + +```bash +docker start webapp +``` + +**Alertmanager's own page** at http://localhost:9093 lists alerts that are active right now: + +![Alertmanager web interface](screenshots/alert-ui.png) + +*This capture was taken when nothing was firing, which is why it says "No alert groups found". During an incident the firing alert appears here, grouped by `alertname`.* + +### Test 2: many requests fail + +```bash +for i in $(seq 1 600); do curl -s localhost:8000/error > /dev/null; sleep 0.2; done +``` + +`/error` returns 500 every time, so the share of failed requests rises above 5%. After one minute `HighErrorRate` starts firing. The Grafana panel **5xx error ratio** shows the spike. + +## 7. Grafana + +Open http://localhost:3000, sign in, then go to **Dashboards → Webapp Overview**. It refreshes every 10 seconds. + +![Grafana dashboard Webapp Overview](screenshots/dashboard-webapp.png) + +*Top left: the webapp is up (1). Top right: requests per second, split by path. The two lines are `/health` (Docker's health check) and `/metrics` (Prometheus's own scrapes). Bottom right: the p95 latency is around 5 ms. Bottom left: "No data" is correct, because no 5xx errors had happened yet; the panel fills in once `/error` is called.* + +> The default password `admin` is acceptable on a laptop only. Set a real one with the environment variable `GRAFANA_PASSWORD` before exposing Grafana anywhere. + +## 8. Troubleshooting + +| Symptom | What to check | +|---|---| +| `docker compose pull` fails with `403 Forbidden` | The download network refused one layer. Run the command again; if it keeps failing, pull the images on another connection or use a registry mirror | +| Target `webapp` is DOWN | Is the webapp container running (`docker ps`)? Is it on the shared network (`docker network inspect monitoring-net`)? Does `curl localhost:8000/metrics` work? | +| Alerts page is empty | The rules file did not load. Run `promtool check config` (above) and read `docker logs prometheus` | +| Changed a rule or config | Reload without restarting: `curl -X POST localhost:9090/-/reload` | +| Alert fires but nothing arrives | Is Alertmanager reachable (`docker logs alertmanager`)? Is `alert-logger` running? Check http://localhost:9090/status for the alertmanager list | +| Grafana panels show "No data" | Wait a minute for data; check the time range (top right); confirm the data source is healthy on the data sources page | + +## 9. What would change in production + +- Real receivers (e-mail, Slack, PagerDuty) instead of the logging webhook, with secrets kept out of the repository. +- Authentication and TLS in front of Prometheus, Alertmanager and Grafana (they are open on their ports here). +- A strong Grafana password and pinned image versions everywhere (`alert-logger` uses the `latest` tag in this project). +- A longer retention or remote storage, and Alertmanager high availability. +- Alerts that also cover the host (disk, memory) and the Docker daemon, for example with `node_exporter` and `cAdvisor`. diff --git a/docs/screenshots/alert-ui.png b/docs/screenshots/alert-ui.png index 1cfcbd9..481ca81 100644 Binary files a/docs/screenshots/alert-ui.png and b/docs/screenshots/alert-ui.png differ diff --git a/docs/screenshots/alerts.png b/docs/screenshots/alerts.png index be65309..2c066b5 100644 Binary files a/docs/screenshots/alerts.png and b/docs/screenshots/alerts.png differ diff --git a/docs/screenshots/dashboard-webapp.png b/docs/screenshots/dashboard-webapp.png index aca75da..df4c109 100644 Binary files a/docs/screenshots/dashboard-webapp.png and b/docs/screenshots/dashboard-webapp.png differ diff --git a/docs/screenshots/firewall-rule.png b/docs/screenshots/firewall-rule.png index 8dd8367..d47958f 100644 Binary files a/docs/screenshots/firewall-rule.png and b/docs/screenshots/firewall-rule.png differ diff --git a/docs/screenshots/full-green-pipline.png b/docs/screenshots/full-green-pipline.png index d06e59c..496fcbe 100644 Binary files a/docs/screenshots/full-green-pipline.png and b/docs/screenshots/full-green-pipline.png differ diff --git a/docs/screenshots/iptabels-afterrule.png b/docs/screenshots/iptabels-afterrule.png index 294f664..874fa2e 100644 Binary files a/docs/screenshots/iptabels-afterrule.png and b/docs/screenshots/iptabels-afterrule.png differ diff --git a/docs/screenshots/laptop-before-iptabels.png b/docs/screenshots/laptop-before-iptabels.png index 6f807ab..8a9c200 100644 Binary files a/docs/screenshots/laptop-before-iptabels.png and b/docs/screenshots/laptop-before-iptabels.png differ diff --git a/docs/screenshots/monitoring-cli.png b/docs/screenshots/monitoring-cli.png index 36198b3..b40ab97 100644 Binary files a/docs/screenshots/monitoring-cli.png and b/docs/screenshots/monitoring-cli.png differ diff --git a/docs/screenshots/promethues-datasource.png b/docs/screenshots/promethues-datasource.png index a07eb82..61b3a7c 100644 Binary files a/docs/screenshots/promethues-datasource.png and b/docs/screenshots/promethues-datasource.png differ diff --git a/docs/screenshots/restart-proof.png b/docs/screenshots/restart-proof.png index 791132f..fb86113 100644 Binary files a/docs/screenshots/restart-proof.png and b/docs/screenshots/restart-proof.png differ diff --git a/docs/screenshots/web-app-down.png b/docs/screenshots/web-app-down.png index 30cec15..044b2fe 100644 Binary files a/docs/screenshots/web-app-down.png and b/docs/screenshots/web-app-down.png differ diff --git a/docs/screenshots/web-app.png b/docs/screenshots/web-app.png index 1966524..568f94e 100644 Binary files a/docs/screenshots/web-app.png and b/docs/screenshots/web-app.png differ diff --git a/docs/webapp.md b/docs/webapp.md new file mode 100644 index 0000000..d7fdcd0 --- /dev/null +++ b/docs/webapp.md @@ -0,0 +1,354 @@ +# The web application and its container + +[← Back to the main README](../README.md) + +This document explains the small Python web service that the rest of the project deploys, builds, monitors and protects: what it does, how it is tested, and how the Dockerfile and Compose file turn it into a container. + +> **In plain words:** the webapp is a tiny web server with a health page and a page that counts things, so the monitoring can watch it. Docker packs it into a box (a *container*) together with everything it needs to run, so the same box runs the same way on any machine. + +**Contents** + +1. [What the service does](#1-what-the-service-does) +2. [The application code](#2-the-application-code) +3. [The tests](#3-the-tests) +4. [The Dockerfile, line by line](#4-the-dockerfile-line-by-line) +5. [The Compose file](#5-the-compose-file) +6. [The health check](#6-the-health-check) +7. [Restart policy and crash recovery](#7-restart-policy-and-crash-recovery) +8. [Run and verify](#8-run-and-verify) +9. [Troubleshooting](#9-troubleshooting) +10. [What would change in production](#10-what-would-change-in-production) + +--- + +## 1. What the service does + +The service is a **FastAPI** application served by **uvicorn** on port **8000**. It is deliberately small, but it exposes everything the other parts of the test need: a health endpoint for Docker, a metrics endpoint for Prometheus, and an endpoint that always fails so the error-rate alert can be triggered on purpose. + +| Method | Path | Response | Why it exists | +|---|---|---|---| +| `GET` | `/` | `{"message": "Hello from the DevOps test service"}` | Proves the service is reachable | +| `GET` | `/health` | `{"status": "ok"}` | Read by Docker's health check | +| `GET` | `/error` | HTTP `500` | Generates 5xx responses to trigger the `HighErrorRate` alert | +| `GET` | `/metrics` | Prometheus text format | Scraped by Prometheus every 15 s | + +Every request is also measured. Two metrics are exported: + +| Metric | Type | Labels | Meaning | +|---|---|---|---| +| `http_requests_total` | Counter | `method`, `path`, `status` | How many requests, split by outcome | +| `http_request_duration_seconds` | Histogram | `path` | How long requests take, as a distribution | + +The histogram is what lets Prometheus answer "are the slowest 5% of requests taking too long?" in `docs/monitoring.md`. + +The files that make up the service: + +``` +webapp/ +├── app/ +│ └── main.py the whole application +├── tests/ +│ └── test_app.py four pytest tests +├── requirements.txt runtime dependencies +├── requirements-dev.txt test dependencies +├── pytest.ini lets pytest import the app +├── Dockerfile builds the image +├── compose.yml runs the container +└── .dockerignore keeps junk out of the image +``` + +## 2. The application code + +The entire application is `webapp/app/main.py`: + +```python +import time + +from fastapi import FastAPI, HTTPException, Request, Response +from prometheus_client import CONTENT_TYPE_LATEST, Counter, Histogram, generate_latest + +app = FastAPI(title="DevOps Test Service") + +REQUESTS = Counter( + "http_requests_total", "Total HTTP requests", ["method", "path", "status"] +) +LATENCY = Histogram( + "http_request_duration_seconds", "Request latency in seconds", ["path"] +) + + +@app.middleware("http") +async def track_metrics(request: Request, call_next): + start = time.perf_counter() + response = await call_next(request) + path = request.url.path + REQUESTS.labels(request.method, path, response.status_code).inc() + LATENCY.labels(path).observe(time.perf_counter() - start) + return response + + +@app.get("/") +def index(): + return {"message": "Hello from the DevOps test service"} + + +@app.get("/health") +def health(): + return {"status": "ok"} + + +@app.get("/error") +def error(): + # Used to generate 5xx responses and trigger the high-error-rate alert + raise HTTPException(status_code=500, detail="simulated failure") + + +@app.get("/metrics") +def metrics(): + return Response(generate_latest(), media_type=CONTENT_TYPE_LATEST) +``` + +- The two metric objects are created **once**, at import time. Prometheus's client keeps a global registry, so they exist as soon as the process starts. +- The **middleware** wraps every request: it records the start time, lets the request run, then increments the counter with the method, the path and the real status code, and records the duration in the histogram. This is why the response code of `/error` (`500`) shows up in the metrics even though the handler raises an exception. +- `/error` raises `HTTPException(500)`. FastAPI turns that into a proper `500` JSON response, so it is a controlled failure, not a crash. +- `/metrics` returns the generated exposition text with the right content type, so Prometheus can read it directly. + +## 3. The tests + +There are four tests in `webapp/tests/test_app.py`, using FastAPI's `TestClient`: + +```python +from fastapi.testclient import TestClient + +from app.main import app + +client = TestClient(app) + + +def test_index(): + r = client.get("/") + assert r.status_code == 200 + + +def test_health(): + r = client.get("/health") + assert r.status_code == 200 + assert r.json() == {"status": "ok"} + + +def test_error_endpoint_returns_500(): + assert client.get("/error").status_code == 500 + + +def test_metrics_exposed(): + client.get("/health") + r = client.get("/metrics") + assert r.status_code == 200 + assert "http_requests_total" in r.text +``` + +| Test | What it proves | +|---|---| +| `test_index` | The app starts and serves a request | +| `test_health` | The health endpoint returns exactly `{"status": "ok"}` — the contract Docker's health check relies on | +| `test_error_endpoint_returns_500` | `/error` really fails with `500`, so the alert can be triggered | +| `test_metrics_exposed` | Metrics are generated and contain the request counter | + +`pytest.ini` contains `pythonpath = .`, which is what lets the tests `from app.main import app` without installing the package. The test-only dependencies (`pytest`, `httpx`) are in `requirements-dev.txt`, kept separate from the runtime `requirements.txt` so the production image stays small. + +Run them locally with: + +```bash +cd webapp +pip install -r requirements-dev.txt +pytest -v +``` + +The CI pipeline runs exactly this (see `docs/gitlab.md`), and GitLab shows the four tests in the pipeline's **Tests** tab. + +## 4. The Dockerfile, line by line + +```dockerfile +FROM python:3.12-slim + +ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY app ./app + +RUN useradd --system --uid 1001 appuser +USER appuser + +EXPOSE 8000 +HEALTHCHECK --interval=15s --timeout=3s --retries=3 \ + CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1 + +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +| Line | What it does and why | +|---|---| +| `FROM python:3.12-slim` | Starts from a small official Python image. `slim` omits compilers and docs the service does not need | +| `ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1` | No `.pyc` files inside the container, and logs appear immediately instead of being buffered (important for `docker logs`) | +| `WORKDIR /app` | All following commands run in `/app` | +| `COPY requirements.txt .` then `RUN pip install` | Dependencies are copied **before** the app code. Docker caches this layer, so changing a source file does not reinstall packages | +| `--no-cache-dir` | Does not keep pip's download cache in the image, saving space | +| `COPY app ./app` | Copies only the application code | +| `RUN useradd …` + `USER appuser` | The container runs as a non-root user (uid 1001). If the app is ever compromised, it does not run as root | +| `EXPOSE 8000` | Documents the port. Publishing it is done by Compose, not here | +| `HEALTHCHECK …` | Every 15 s Docker calls `/health`; after 3 failures in a row the container is marked `unhealthy` (see [section 6](#6-the-health-check)) | +| `CMD ["uvicorn", …]` | The exec form (`["…"]`) makes uvicorn PID 1, so it receives stop signals directly | + +`.dockerignore` excludes `.venv/`, `__pycache__/`, `*.pyc` and `.pytest_cache/`, so those never end up in the image. + +## 5. The Compose file + +```yaml +services: + webapp: + build: . + image: webapp:latest + container_name: webapp + restart: unless-stopped + ports: + - "8000:8000" + networks: + - monitoring-net + +networks: + monitoring-net: + external: true +``` + +| Setting | Meaning | +|---|---| +| `build: .` | Build the image from the Dockerfile in this folder | +| `image: webapp:latest` | Name the built image, so `deploy` can start it with `--no-build` | +| `container_name: webapp` | A fixed name, which the pipeline's health check and the docs refer to | +| `restart: unless-stopped` | Restart automatically after a crash (see [section 7](#7-restart-policy-and-crash-recovery)) | +| `ports: 8000:8000` | Publish the port to the host, so the service is reachable and Prometheus/curl can reach it | +| `networks: monitoring-net` | Join the shared network, so Prometheus and Grafana find `webapp` by name | + +`monitoring-net` is declared `external: true`: Compose expects the network to already exist instead of creating it. That is why the setup and the deploy stage run `docker network create monitoring-net` first (see `docs/monitoring.md` and `docs/gitlab.md`). Marking it external is what lets the webapp stack and the monitoring stack share one network even though they are separate Compose projects. + +## 6. The health check + +The health check has two halves that meet at `/health`: + +- **The endpoint** — `app/main.py` returns `{"status": "ok"}` only when the app itself is running and serving requests. +- **The check** — the Dockerfile's `HEALTHCHECK` runs `python -c "…urlopen('http://localhost:8000/health')"` every 15 s, with a 3 s timeout and 3 retries. + +Docker tracks the result and gives the container a state: + +| State | When | +|---|---| +| `starting` | Right after start, before the first successful check | +| `healthy` | The check returns success | +| `unhealthy` | Three consecutive failures | + +This state is not decoration. The CI **deploy** stage reads it with `docker inspect -f '{{.State.Health.Status}}' webapp` and fails the job if it never becomes `healthy` (see `docs/gitlab.md`). So the health check is what turns "the container started" into "the service works". + +Check it by hand: + +```bash +docker inspect -f '{{.State.Health.Status}}' webapp +docker inspect -f '{{json .State.Health}}' webapp | python3 -m json.tool +``` + +## 7. Restart policy and crash recovery + +The Compose file sets `restart: unless-stopped`. Docker has several policies: + +| Policy | Restarts after a crash | Restarts after `docker stop` | After a host reboot | +|---|---|---|---| +| `no` (default) | no | no | no | +| `on-failure` | yes (only non-zero exits) | no | no | +| `always` | yes | yes — even something you stopped by hand | yes | +| `unless-stopped` | yes | no | yes (unless you had stopped it) | + +`unless-stopped` is the right middle ground: a crash is recovered automatically, but a container you stopped **on purpose** stays stopped. That distinction matters for the monitoring test in `docs/monitoring.md`, where `docker stop webapp` is used to trigger the `WebappDown` alert and must *not* be undone by the restart policy. + +**The crash-recovery demonstration.** `docker kill` is not a graceful stop: it sends `SIGKILL`, the container dies immediately with a non-zero exit code, and the policy brings it back. + +```bash +# 1. Note the current container and its restart count +docker ps --format '{{.Names}}\t{{.Status}}' | grep webapp +docker inspect -f '{{.RestartCount}}' webapp + +# 2. Kill it hard +docker kill webapp + +# 3. A moment later it is back — same container name, RestartCount increased +docker ps --format '{{.Names}}\t{{.Status}}' | grep webapp +docker inspect -f '{{.RestartCount}}' webapp +``` + +![Container automatically restarted after docker kill](screenshots/restart-proof.png) + +*After `docker kill webapp` the container disappears for a moment and Docker starts it again on its own: `docker ps` shows `webapp` up again with a higher `RestartCount`, and no command was typed to bring it back.* + +Two things this deliberately does **not** cover: + +- A container stopped with `docker stop` is considered a deliberate stop and stays down. That is the documented behavior, used on purpose in the monitoring test. +- `restart: unless-stopped` restarts the **container**, not the whole stack. If the Docker daemon itself is restarted, the policy is re-applied at daemon startup — but the network `monitoring-net` must still exist. + +## 8. Run and verify + +The webapp stack must be started before the monitoring stack, because both use the same network: + +```bash +docker network create monitoring-net # only once +cd webapp +docker compose up -d --build +``` + +![The service started with docker compose up](screenshots/web-app.png) + +*`docker compose up -d --build` builds the image, creates the container and starts it detached. `docker ps` then shows `webapp` up, on port `8000`, with its health status.* + +Confirm it works from the outside: + +```bash +docker ps # webapp is Up +curl localhost:8000/health # {"status":"ok"} +curl localhost:8000/ # {"message":"Hello from the DevOps test service"} +curl -s localhost:8000/metrics | head -3 # Prometheus exposition text +curl -s -o /dev/null -w "%{http_code}\n" localhost:8000/error # 500, on purpose +``` + +Watch the health state turn `healthy`: + +```bash +docker inspect -f '{{.State.Health.Status}}' webapp +``` + +Stop the stack (this keeps the data volumes and the image): + +```bash +docker compose down +``` + +## 9. Troubleshooting + +| Symptom | Cause and fix | +|---|---| +| `docker compose up` fails with `network monitoring-net … not found` | The network is declared external. Create it once: `docker network create monitoring-net` | +| Port `8000` is already in use | Another process or an old container holds it. Find it with `ss -ltnp \| grep 8000` or `docker ps`, then stop it | +| The container is `unhealthy` | The app is not answering `/health`. Read `docker logs webapp`; test locally with `cd webapp && pytest -v` | +| `docker compose up` reuses an old image | The build was skipped or cached. Rebuild with `docker compose up -d --build` | +| A `docker stop webapp` did **not** restart it | Expected: `unless-stopped` treats a deliberate stop as final. Use `docker start webapp` (or `docker compose up -d`) | +| `docker kill webapp` did not restart it | Check the policy with `docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' webapp`; it must be `unless-stopped` | +| Changes to `main.py` do not appear | The running container still uses the old image. Rebuild: `docker compose up -d --build` | +| `pytest` cannot import `app` | Run it from `webapp/` so `pytest.ini` (`pythonpath = .`) applies | + +## 10. What would change in production + +- **Pin the base image and dependencies.** `python:3.12-slim` floats; in production pin a digest and pin the versions in `requirements.txt` for reproducible builds. +- **Read-only filesystem and dropped capabilities.** The app needs no write access and no privileges; add `read_only: true` and `cap_drop: ["ALL"]` to harden the container. +- **Run uvicorn with multiple workers.** A single process is fine for a demonstration, but production would use `--workers` (or Gunicorn with uvicorn workers) and CPU/memory limits. +- **TLS and authentication in front of the service.** Here it is served in plain HTTP; a real deployment terminates TLS at a reverse proxy. +- **A real health/readiness split.** Docker's `HEALTHCHECK` doubles as liveness and readiness; production often separates `/healthz` and `/readyz`. +- **Image scanning and a registry.** The image is built locally and deployed by tag; production would push to a registry and scan for vulnerabilities (see the `publish` stage in `docs/gitlab.md`).