Kubernetes monitoring in practice almost always means the same stack: Prometheus collects metrics, Grafana displays them, Alertmanager sends alerts once something goes off the rails. The fastest way there is the kube-prometheus-stack Helm chart, which rolls out all three components together with the exporters they need and sensible defaults, instead of you wiring up every piece by hand.
I run Kubernetes clusters in production and wrote the Kubernetes Practical Guide published by Rheinwerk Computing (2024). Monitoring is not a nice to have for me, it is the precondition for knowing whether a cluster is healthy before a customer notices otherwise. You can find all my articles on running Kubernetes clusters on the Kubernetes page.
It helps to keep three jobs apart that everyday language happily merges into "monitoring". Collection pulls metrics in and stores them, and that is Prometheus alone, through its scrape configuration. Alerting evaluates that stored data against rules and reports violations, which is what the alerting rules in Prometheus and the Alertmanager do. Visualization only reads the stored data back out, and that is Grafana's role. Grafana is therefore not part of the scrape logic at all: if Grafana goes down, Prometheus keeps collecting and alerts keep firing.
What Kubernetes monitoring needs to cover
Three layers need attention, and all three end up in the same time series database in a good monitoring setup. The node layer shows you CPU, memory, disk and network of the underlying machines. The Kubernetes layer shows the state of the objects themselves: how many replicas of a deployment are actually running, how often a Pod restarts, how many nodes are reachable. The application layer, finally, shows what matters to your users, things like response times, error rates or the length of a queue.
Prometheus covers all three layers through different sources. Node Exporter delivers hardware and operating system metrics for each machine. Kube State Metrics translates the state of Kubernetes objects such as deployments, Pods and nodes into metrics Prometheus can understand. Your own applications supply their own metrics whenever they expose a Prometheus compatible format over an HTTP endpoint, usually under the path /metrics.
Rolling out kube-prometheus-stack via Helm
Rather than installing Prometheus, Grafana, Alertmanager, Node Exporter and Kube State Metrics separately and wiring them together, I use the community kube-prometheus-stack Helm chart in every one of my clusters. It ships sensible defaults, installs the Prometheus Operator along with everything else, and already includes a set of standard dashboards and alert rules for the cluster itself.
# values.yaml for kube-prometheus-stack
grafana:
adminPassword: "overridden-by-a-secret"
prometheus:
prometheusSpec:
retention: 15d
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
memory: 1Gi
storageSpec:
volumeClaimTemplate:
spec:
resources:
requests:
storage: 20Gi
alertmanager:
alertmanagerSpec:
resources:
requests:
cpu: 50m
memory: 64Mi
If you roll this chart out through ArgoCD instead of a manual helm install, this values.yaml lives in Git and syncs automatically with every change. The Grafana password belongs in a secret encrypted with SOPS, never in plain text in the values file. After the rollout, kubectl get pods -n monitoring shows you the running components, and a port forward to the Grafana service immediately shows the bundled dashboards for nodes, namespaces and individual workloads.
Hooking up your own applications with a ServiceMonitor
The Prometheus Operator that comes with kube-prometheus-stack replaces manually editing the Prometheus configuration with a dedicated Kubernetes object, the ServiceMonitor. Instead of editing a config map and restarting the Prometheus Pod, you create one of these objects for every application you want monitored, and the operator makes sure Prometheus picks up the right scrape configuration.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: shop-api
namespace: shop
labels:
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app: shop-api
endpoints:
- port: metrics
path: /metrics
interval: 30s
The release label matters here, it has to match the name of your Helm installation: the chart sets the serviceMonitorSelector of the Prometheus instance it creates to exactly this label by default, so without a matching value your ServiceMonitor is simply never selected. The selector points at the labels of the Kubernetes service in front of your application, not at the Pod directly, and endpoints.port has to match the named port from your service manifest. You can confirm Prometheus is actually scraping the application under Status and Targets in the Prometheus UI, where your ServiceMonitor shows up with its last scrape status. A ServiceMonitor changes collection only; it does not produce a dashboard, which you build afterwards in Grafana on top of the metrics that are now available.
The alerts that actually matter in production
A fresh kube-prometheus-stack install already ships plenty of alert rules for the cluster itself, things like failed nodes or an etcd storage volume filling up. For your own applications, it pays to add a few well chosen rules rather than monitoring everything that can be measured. In my clusters, these cases have proven genuinely worth an alert: a deployment running fewer replicas than desired for several minutes, a Pod restarting repeatedly, a namespace approaching its resource limit, and an error rate crossing a clearly defined threshold.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: shop-api-alerts
namespace: shop
labels:
release: kube-prometheus-stack
spec:
groups:
- name: shop-api
rules:
- alert: ShopApiReplicasLow
expr: kube_deployment_status_replicas_available{deployment="shop-api"} < 2
for: 5m
labels:
severity: warning
annotations:
summary: "Fewer than two running replicas for shop-api"
Every rule needs a for clause so a brief blip does not trigger an alert instantly, and a clear summary that already states what is wrong right in the alert, without anyone having to open a dashboard first. Too many alerts with too low a threshold bury the real problems in noise, and that is the most common mistake I see in freshly set up monitoring.
Wiring Alertmanager to mail or chat
Alertmanager does not send alerts on its own, it needs a receiver that defines where a notification actually goes. Smaller teams often get by with a single chat webhook, larger environments benefit from splitting by severity, so a plain heads up does not land in the same channel as an alert firing in the middle of the night.
route:
receiver: chat-default
routes:
- matchers:
- severity="critical"
receiver: chat-urgent
receivers:
- name: chat-default
webhook_configs:
- url: "https://example-webhook.invalid/default"
- name: chat-urgent
webhook_configs:
- url: "https://example-webhook.invalid/urgent"
This configuration lives under the alertmanager.config key in the kube-prometheus-stack values.yaml, and the webhook URLs again belong in an encrypted secret. Test the wiring with a harmless test alert after every change before you rely on it, an Alertmanager that quietly does nothing is more dangerous than no monitoring at all, because it creates a false sense of security.
Resource sizing and retention on small clusters
Prometheus keeps its time series in memory before writing them to disk, so its memory footprint grows with the number of metrics and labels, not just with cluster size. On my smaller Hetzner clusters, 512 mebibytes of requested memory and roughly one gigabyte of limit are usually enough for Prometheus, and that grows accordingly once you monitor a lot more applications, something you should observe through the monitoring itself rather than guess at.
The retention setting in the values.yaml determines how long Prometheus keeps data locally before it gets overwritten. Two to four weeks cover most operational questions, and for longer trend analysis it is worth connecting an object storage compatible long term archive instead of simply raising local retention over and over. Storage space for the time series is tied directly to this setting and should be sized generously rather than tightly, since a full volume simply stops Prometheus from writing new data at all.
Frequently asked questions
How much memory does Prometheus need?
That depends heavily on the number of metrics and labels, not just on cluster size. Smaller clusters with a modest number of applications often get by on a few hundred megabytes, while larger environments with many labels and short scrape intervals need considerably more. Watch the actual usage of the Prometheus Pod itself through its own dashboard.
How long should I keep metrics around?
Two to four weeks is usually enough for day to day operations. If you want to compare seasonal trends over months, an external long term store is a better fit than pushing local retention very high, which mostly just costs storage and resources.
Should I run Prometheus myself or use a hosted option?
Both have their place. Running it yourself keeps every metric inside your own cluster, which matters to customers with data protection requirements, but it costs operational effort for updates and storage. A hosted option takes that effort off your hands but ties you to another external service and its pricing model.
How do I monitor my own applications on top of the cluster?
Through a ServiceMonitor pointing at your application's service, provided the application exposes metrics itself in a Prometheus compatible format over an HTTP endpoint. For applications that cannot do that on their own, matching exporters handle the translation.
Do I need logging alongside metrics monitoring?
Yes, metrics tell you something is wrong, logs are usually what tells you why. I combine Prometheus with a logging stack in every cluster, more on that in Kubernetes Logging with Loki and Grafana.
Where to go next
Once monitoring is running, it is worth looking at automatic scaling: in Kubernetes Autoscaling with HPA I show how to use the metrics you just collected for automatic scaling decisions too. And if you want to know whether your applications even start up healthy before Prometheus ever measures them, Liveness and Readiness Probes Explained will help.
My suggestion: install kube-prometheus-stack first with its bundled defaults in a test namespace, look at the dashboards that come with it, and only then add a ServiceMonitor for your first real application. Details on the chart itself are in the kube-prometheus-stack GitHub repository, and the fundamentals of PromQL and the Prometheus architecture are in the official Prometheus documentation.
In full detail, from Node Exporter to building your own Grafana dashboard, this is covered in chapter 8.4 of my Kubernetes Practical Guide (Rheinwerk Computing).
