2026
Loki: Log Aggregation for Modern Infrastructure
Every service writes logs. The challenge is finding the right lines across dozens of containers running on multiple nodes. Loki collects log streams from your infrastructure, stores them cheaply by indexing only labels and lets you query everything through LogQL.
Every service writes logs. Every request that fails, every exception that gets caught, every job that finishes leaves a line somewhere on disk. The question is never whether the information exists. The question is whether you can find it when you need it.
On a single machine that is manageable. You know where the logs are, you can tail the file. Once you have dozens of containers running across multiple nodes, all rotating their own log files, finding anything useful becomes a different kind of problem. You need somewhere to send all of it, a way to search across it and enough context to understand what you are looking at.
Loki is Grafana Labs' answer to that problem. It is a log aggregation system designed to collect log streams from across your infrastructure, store them efficiently and let you query them with a syntax that will feel familiar if you have spent time with Prometheus. The design philosophy is deliberately different from older log aggregation tools: Loki indexes as little as possible, which keeps storage costs low and ingestion fast at the expense of some query flexibility.
Why Loki indexes labels, not content
Tools like Elasticsearch index every word in every log line. That makes full-text search very fast but it also makes ingestion expensive, storage large and the operational overhead significant. For many teams the tradeoff works fine. For others it becomes a problem at scale.
Loki takes a different approach. It only indexes the labels attached to each log stream. The actual log lines are stored as compressed chunks without any word-level indexing. When you run a query, Loki uses the labels to identify which chunks to fetch, then scans through those chunks looking for matches. The label index is small. The chunks are just compressed text.
The practical consequence is that ingestion is cheaper, storage is smaller and operations are simpler. The cost is that queries which cannot narrow down using labels have to scan more data. If your label cardinality is low and your queries start with label selectors, Loki performs well. If you need to search arbitrary strings across all logs without any label to narrow the scope first, it will be slower than a fully indexed system.
Core components
A Loki deployment has three moving parts. They are designed to work together but each has a distinct job. Understanding what each one does makes it easier to reason about where things can go wrong.
Promtail
The agent that runs on each host and ships log lines to Loki. It watches log files on disk, reads from the systemd journal and can also receive logs over HTTP. Before sending, it attaches labels based on the file path, pod metadata or anything else you configure in the scrape rules.
Loki
The storage and query backend. It receives log streams from agents, indexes only the labels attached to each stream and stores the raw log content compressed in object storage. When a query comes in, Loki uses the labels to locate the right chunks, then runs the filter or search against the actual log lines.
LogQL
The query language. It works in two modes: log queries return the matching log lines themselves, metric queries turn those lines into numeric time series. You use it interactively in Grafana or when writing alerting rules.
Labels and log streams
In Loki, a log stream is a sequence of log lines that all share the same set of labels. Labels are key-value pairs attached at ingestion time. They are the only thing Loki indexes, so they are also the primary way you narrow down a query.
The same rules that apply to Prometheus labels apply here. Labels should reflect low-cardinality dimensions: application name, environment, namespace, host. A label whose value can be one of millions of unique things, like a request ID, will create millions of streams. That blows up the index and defeats the storage efficiency that makes Loki worth using.
{app="api", env="production", namespace="default"}
Every log line from this stream carries these three labels. The stream is the unit of storage in Loki.
{job="nginx", host="web-01"}
Scraped from a log file on a specific host. The job and host labels came from the Promtail scrape config.
{container="postgres", pod="postgres-0", namespace="db"}
Scraped from Kubernetes. These labels were attached automatically from the pod metadata.
How logs get from source to storage
Logs move from wherever they originate to long-term storage through a pipeline that involves the agent, the Loki ingester and the backing store. Each step is straightforward on its own. Together they handle the pressure of continuous high-volume writes without losing data.
Agent scrapes log source
Promtail (or another agent) watches log files, systemd journals or HTTP endpoints and reads new lines as they appear
Labels are attached
The scrape configuration maps metadata to labels. File path, pod name, namespace, environment. These become the index
Lines are batched and compressed
The agent groups lines into chunks, compresses them and ships them over HTTP to the Loki ingester
Stored in object storage
Compressed chunks land in S3, GCS or local disk. The label index is written separately so queries can locate chunks without scanning everything
LogQL
LogQL is the query language you use to talk to Loki. Every query starts with a stream selector in curly braces that picks which log streams to look at. From there you can filter lines, extract fields from structured logs and, if you need numbers rather than lines, turn log volume into a time series.
LogQL examples
{app="api", env="production"}
All log lines from the production API. Just the stream selector with no filter.
{app="api"} |= "error"
Lines from the API stream that contain the string "error". Case sensitive.
{app="api"} | json | level="error"
Parse each line as JSON, then filter to lines where the level field equals "error". Works well with structured logging.
rate({app="api"} |= "error" [5m])
Error line rate per second over the last five minutes. This is a metric query. The result is a time series you can graph or alert on.
sum by (namespace) (rate({job="kubernetes"} [5m]))
Total log line ingestion rate across Kubernetes, broken down by namespace. Useful for spotting which service is suddenly very chatty.
The pipeline stages after the stream selector chain together. You can parse JSON with | json, extract fields from unstructured text with | pattern or | regexp, then filter on those extracted fields further down the pipeline. Queries that would previously have required multiple passes through a log file can often be expressed as a single LogQL expression in Grafana.
Loki in Kubernetes
Kubernetes is where Loki tends to show up most. Containers write to stdout and Kubernetes captures that output into log files on the node. Promtail runs as a DaemonSet, one instance per node, watching those files and shipping the lines to Loki with labels pulled from the pod metadata.
Promtail scrape config for Kubernetes pods
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
The relabeling configuration is where labels get attached. Promtail discovers pods through the Kubernetes API, reads their metadata and maps fields like namespace, app label and pod name onto the log stream. Once those labels are in place you can query for all logs from a specific namespace or narrow down to a single pod without knowing which node it is running on.
Alerting on logs
Loki supports alerting rules written in LogQL. The same metric query syntax you use to build a graph in Grafana can be used to fire an alert when a threshold is crossed. Rules are evaluated on a schedule by the Loki ruler component and fired alerts are routed through Alertmanager, the same path Prometheus alerts take.
Example alerting rule
groups:
- name: application
rules:
- alert: HighErrorRate
expr: |
sum(rate({app="api"} |= "error" [5m])) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "API error rate above threshold"
Log-based alerting fills a gap that metrics alone cannot cover. Prometheus tells you that error rate is high. A Loki alert on specific log patterns can tell you about things that never made it into a metric: a panic message buried in application output, a specific exception class that your instrumentation does not expose as a counter, a security event recorded only in an access log.
Storage
Loki is built to use object storage as its backing store. S3, GCS and Azure Blob Storage are all supported. Object storage is cheap at scale and durable by default, which makes it a reasonable choice for log data that you need to keep for weeks or months but query infrequently once it is more than a few days old.
For smaller deployments or local testing, Loki can store everything on local disk using the filesystem backend. The tradeoff is that you lose durability guarantees and horizontal scalability. Most production setups move to object storage once volume grows past what a single node handles comfortably.
Retention is configured at the store level. You define how long chunks are kept before deletion. Unlike a database, there is no need to run periodic cleanup jobs manually. The compactor component handles old chunk removal on a schedule once data passes the configured retention window.
Where it fits
Loki sits in the logs corner of the observability triangle. Prometheus handles metrics. Tempo handles distributed traces. Loki handles logs. All three feed into Grafana, which gives you a single interface to move between them during an incident. When a Prometheus alert fires you can jump directly from the alert into Loki and look at what the relevant service was logging at that moment, without switching tools.
The label model is the key thing to get right early. Labels that are too broad make queries slow because they pull in more streams than needed. Labels with too many unique values make the index large and ingestion expensive. The right cardinality sits somewhere between those two problems and the answer is usually to index the dimensions you actually filter on: app, environment, namespace, host.
Getting started is straightforward. The Loki Helm chart for Kubernetes deploys Loki, Promtail and Grafana together and has Kubernetes log scraping preconfigured. Within an hour you typically have logs flowing from every pod in the cluster, searchable by namespace and application name in Grafana. The harder work comes later: building useful dashboards, writing alerting rules that catch real problems and deciding how long different categories of logs actually need to be kept.