Kubernetes Logging with Loki and Grafana

Kubernetes logging with Loki and Grafana pulls the container logs of your whole cluster into one place: a log shipper reads them on every node, Loki stores them lean and sorted by labels, and you search all of it through Grafana with its own query language called LogQL. That replaces jumping between individual kubectl logs calls with one central search across the whole cluster, even for a Pod that is long gone.

The Loki stack deliberately works differently from classic logging solutions. Loki does not index the full log text, only the labels around it, more on that in a moment. That makes the operation cheaper and simpler, but it demands discipline in how you choose your labels, otherwise your searches turn slow instead of fast.

I run production-grade Kubernetes clusters and use Loki together with Grafana for exactly this kind of logging in my own production cluster. You can find all my articles on running Kubernetes clusters collected on the Kubernetes page.

Why Loki instead of ELK for Kubernetes logging

The classic approach to centralized logging is ELK or EFK: Elasticsearch, Logstash or Fluentd, Kibana. Every log line ends up fully in an inverted full-text index there, every word becomes searchable. That is powerful, but it costs storage and compute that grows with your cluster's log volume, often faster than you would like.

Loki takes the opposite approach and borrows its thinking from Prometheus, just for logs instead of metrics. Instead of indexing every word, Loki only remembers the combination of labels a log line arrived with, such as namespace, application and Pod. The actual text lands compressed in chunks that you only search after selecting the matching labels. That keeps the index small even as your log volume grows over months.

	Loki	Elasticsearch (ELK/EFK)
What gets indexed	only labels like namespace, app, Pod	the full log text of every line
Storage footprint	small, compressed chunks in object storage	large, inverted full-text index
Query language	LogQL, modeled on PromQL	its own query syntax with full-text search
Operational effort	few components, a single binary at small scale	manage indexing, sharding and replicas yourself
Strength	search a lot of logs cheaply, narrow down by labels	free full-text search across arbitrary fields

For you, that plays out very concretely: if your searches usually start with a namespace, an application or a Pod, and you filter by text from there, Loki fits your day-to-day work well. If instead you need to search freely across arbitrary fields, say across every application for a customer number without narrowing down first, a full-text index like Elasticsearch has the edge. I use Loki because my queries almost always start at a namespace or an application.

Loki's architecture in five sentences

Loki consists of a write path and a read path that run as a single process in the simple setup and scale as separate components in the larger setup. Incoming log lines are grouped into streams, where a stream is a fixed combination of labels, and within a stream they are packed into compressed chunks. The index only holds the mapping from a label combination to a chunk, not the content itself, and that is the key difference to full-text systems. Chunks and index land in object storage, which keeps Loki itself largely stateless and easy to replace. A query first picks the matching streams by labels and only then filters or parses the actual text, which is why your choice of labels decides whether your queries stay fast or turn sluggish.

Installing Loki on the cluster with Helm

For most clusters the monolithic mode of Loki is enough: one instance handles writing, reading and compaction together. Only once your log volume grows noticeably does switching to the mode with separate components, which you scale independently, pay off. I always start new clusters in monolithic mode and only switch once I actually see it in the resource curves.

Installation runs through Grafana's official Helm repository, like most charts:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki -n logging --create-namespace -f loki-values.yaml

In loki-values.yaml you decide where Loki writes its chunks and how long they should stay around. On my Hetzner clusters I use an S3-compatible object storage for this, the same approach I use for backups:

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  storage:
    type: s3
    bucketNames:
      chunks: loki-chunks
    s3:
      endpoint: https://fsn1.your-objectstorage.com
      region: fsn1
      s3ForcePathStyle: true
  limits_config:
    retention_period: 720h
singleBinary:
  replicas: 1
  persistence:
    size: 20Gi

storage.type and the s3 block tell Loki where the chunks go, replication_factor: 1 fits a single-binary setup without multiple replicas. The limits_config block sets how long logs stick around, more on that below. After the rollout, kubectl get pods -n logging shows you the running Pod, and kubectl Commands: The Essentials covers how to keep an eye on Loki's own state and logs afterwards.

Alloy instead of Promtail: shipping the logs

Loki only stores what reaches it, and collecting the logs on every node is a separate agent's job. For a long time that was Promtail, built specifically for Loki. Promtail is no longer under active development, and its successor is called Alloy, which covers logs alongside metrics and traces with the same configuration language. If you are starting a new Loki stack, go straight for Alloy. Moving an existing Promtail setup over is worth it at the latest once you are already touching the log pipeline anyway.

Alloy runs as a DaemonSet just like Promtail did, one Pod per node that reads that node's container logs and enriches them with Kubernetes labels. How a DaemonSet works in general and when you need one is covered in Kubernetes DaemonSet Explained. Alloy does not describe its configuration in YAML but in its own declarative language made of named components that pass targets on to each other:

discovery.kubernetes "pods" {
  role = "pod"
}

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.write.production.receiver]
}

loki.write "production" {
  endpoint {
    url = "http://loki-gateway.logging.svc/loki/api/v1/push"
  }
}

The discovery.kubernetes component finds every Pod in the cluster through the Kubernetes API, loki.source.kubernetes reads their container logs and passes them to loki.write, which sends them to Loki's push endpoint. Each component hands its output to the next, similar to a pipe in the shell, just as a graph that keeps running instead of a one-off command.

One detail you have to account for with a DaemonSet: discovery.kubernetes finds every Pod in the whole cluster, not just the ones on its own node. Without a filter, every Alloy instance ships all of it, and you end up storing the same lines in Loki once per node. The fix is a discovery.relabel step between discovery and the log source that trims the targets down to the local node:

discovery.relabel "own_node" {
  targets = discovery.kubernetes.pods.targets

  rule {
    source_labels = ["__meta_kubernetes_pod_node_name"]
    regex         = sys.env("NODE_NAME")
    action        = "keep"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.own_node.output
  forward_to = [loki.write.production.receiver]
}

Discovery attaches the meta label __meta_kubernetes_pod_node_name to every target, and the keep rule retains only those whose node name matches the local one. The Pod gets its own node name through the downward API into the NODE_NAME environment variable, which in the DaemonSet means an env entry with a fieldRef on spec.nodeName. In the Alloy configuration, sys.env reads that value back. From then on, loki.source.kubernetes no longer points at discovery directly but at the filtered output. The available meta labels and the component syntax are documented in the Alloy documentation for discovery.kubernetes.

LogQL: the queries you need every day

LogQL always starts by selecting a stream through labels in curly braces, followed by optional filters and parsers. Three queries cover a large part of daily work:

# Every error in a namespace over the last hour
{namespace="checkout"} |= "error"

# Pull response times over 500ms out of structured JSON logs
{app="checkout-api"} | json | duration > 500ms

# Error rate per minute as a number instead of raw text
sum(rate({namespace="checkout"} |= "error" [1m]))

The first query shows how close LogQL is to grep: labels pick the stream, |= "error" then filters the text. The second query parses every line as JSON and filters on a field from it, which assumes your application logs in a structured format. The third query turns log lines into a time series, just like a metric, and you can drop it straight into a Grafana panel or an alert. If you already know Prometheus's label model, LogQL feels familiar almost immediately.

Retention and the object storage backend

Logs grow faster than almost any other kind of data in a cluster, which is why retention decides your costs directly. Loki removes expired chunks through its own component, the compactor, which regularly walks through object storage and deletes everything older than the configured retention period:

loki:
  limits_config:
    retention_period: 720h
  compactor:
    retention_enabled: true
    delete_request_store: s3
    working_directory: /loki/compactor

retention_period sets how long a chunk is kept, retention_enabled is what actually turns on deletion in the compactor. Without an enabled compactor you keep accumulating chunks without limit, and your object storage grows unnoticed and unbounded. I like staggering retention by environment: development and test clusters need much shorter retention than the production cluster, where I sometimes want to look back weeks.

Labels: the most common source of trouble

The biggest mistake with Loki is almost always the same one: too many or too variable labels. Every unique combination of labels creates its own stream, and every stream costs index entries. Put a request ID, a user ID or a timestamp into a label instead of the log text, and the number of streams explodes, the index turns huge, and exactly the queries that should be fast become painfully slow.

The rule behind this is simple: labels describe where a log line came from, such as namespace, application, Pod or node, not what is written in it. Anything that keeps changing within a Pod belongs in the log text and gets parsed at query time with json or logfmt, not carried as its own label. Keep that separation from the start and your queries stay fast even after months of logs.

Frequently asked questions

Loki or Elasticsearch for Kubernetes logging?

Loki is a good fit when your searches almost always start with labels like namespace, application or Pod and you filter by text from there, and it keeps operations leaner and cheaper. If you need free full-text search across arbitrary fields without a fixed label schema, say across every application at once, Elasticsearch is the better choice.

Promtail or Alloy: which one should I use?

For a fresh setup I would go straight for Alloy, it is the actively maintained successor and covers logs alongside metrics and traces with the same language. If Promtail is already running stably for you, you do not have to switch immediately, but you should plan the move for whenever you touch the log pipeline anyway.

How long should I keep Kubernetes logs?

That depends on your environment and any compliance requirements you have, there is no universal number. I run shorter retention in development and test clusters and much longer retention in production, controlled through retention_period and an enabled compactor.

Why are my Loki queries slow?

In almost every case it comes down to too many or too variable labels, which inflate the number of streams and therefore the index. Check first whether request IDs, user IDs or timestamps are accidentally landing as labels instead of in the log text, that is the most common cause of sluggish queries.

Where to go next

If you think about logging and monitoring together, it is worth looking at Kubernetes Monitoring with Prometheus, since the two stacks often share the same labels and the same Grafana instance in practice. For everyday work with kubectl around Loki, Alloy and the applications themselves, kubectl Commands: The Essentials will help, and if it is not yet clear how a DaemonSet like Alloy even gets onto every node, that is covered in Kubernetes DaemonSet Explained.

My suggestion for getting started: install Loki in monolithic mode in a test namespace, roll out Alloy as the shipper, and put together your first query in Grafana from a single namespace before you attempt more complex LogQL expressions.

My Kubernetes Practical Guide (Rheinwerk Computing) dedicates chapter 8 to running Kubernetes in production, with its own section on monitoring. I do not cover logging as its own topic there, since it is closely tied to monitoring and gets more room here as a separate article instead. You can find all the information about the book on my Kubernetes page. The details of Loki's architecture are in the official Loki documentation, and Kubernetes's own logging concepts are covered in the Kubernetes documentation on logging architecture.