Centralized Logging in Kubernetes with Grafana Loki
As Kubernetes environments grow, relying on kubectl logs for individual pods stops working. Pods get replaced, logs disappear, and finding a specific error across dozens of services becomes a time-consuming hunt. Centralized logging solves this by aggregating logs from every container into a single searchable store.
This guide walks through setting up the Grafana Loki stack on Kubernetes using Helm, covering installation, verification, and the most common issues you will encounter in production.
The Architecture
The Loki stack uses three components working together in a simple pipeline:
- Promtail runs as a DaemonSet, meaning one instance per node. It watches the container log files on each node and ships them to Loki.
- Loki receives and indexes log streams. Unlike Elasticsearch, it indexes only metadata labels rather than full log content, which keeps storage costs low.
- Grafana provides the query interface. You write LogQL queries to filter, search, and visualize logs from any service across your cluster.
Installation with Helm
Add the Grafana Helm repository and install the Loki stack:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install loki grafana/loki-stack \
--namespace monitoring \
--create-namespace \
--set grafana.enabled=true \
--set promtail.enabled=true
Wait for all pods to reach Running status before proceeding:
kubectl get pods -n monitoring
You should see pods for Loki, Promtail (one per node), and Grafana all running.
Accessing Grafana
Port-forward to the Grafana service to access the UI locally:
kubectl port-forward -n monitoring svc/loki-grafana 3000:80
Get the auto-generated admin password:
kubectl get secret -n monitoring loki-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode
Open http://localhost:3000, log in with admin and the password above, then navigate to Configuration > Data Sources to confirm Loki is listed and the connection test passes.
Running Your First LogQL Query
In Grafana, open the Explore view and select Loki as the data source. A basic query to see all logs from a specific namespace looks like this:
{namespace="production"}
Filter by application and search for errors:
{namespace="production", app="api-service"} |= "error"
LogQL supports regex, label filtering, and rate calculations, which makes it straightforward to build dashboards and alerts once your logs are flowing.
Common Issues and Fixes
Promtail pods stuck in Pending: This usually means nodes do not have enough available memory. Promtail is lightweight but still needs headroom. Check node resource pressure with kubectl describe node and look at the Events section. If your nodes use taints, Promtail may also need matching tolerations added to its DaemonSet.
Loki connection failures in Grafana: The most common cause is an incorrect service URL in the data source configuration. The default internal URL is http://loki:3100 within the same namespace. Verify the exact service name with kubectl get svc -n monitoring.
Loki restarting repeatedly: Usually a storage issue. The default configuration uses local disk, which requires a persistent volume. If no suitable storage class exists, Loki will fail to start. Check kubectl describe pod on the Loki pod and look for volume mount errors.
Logs missing from some services: Most often a label mismatch. Promtail collects logs based on pod labels. If a service uses non-standard labels or the pod spec is missing common labels like app, Promtail may not pick them up. Review the Promtail configuration and adjust the scrape config to match your labeling conventions.
Best Practices for Production
Configure persistent storage for Loki from the start. Losing logs on a Loki pod restart makes the system unreliable for incident investigation.
Set a retention policy. Without one, your log storage grows indefinitely. Loki supports table-based retention that you can configure to keep logs for a specific number of days.
Apply resource limits to both Loki and Promtail. Without limits, a log spike from one noisy service can starve other workloads on the same node.
Use consistent labels across your services. The power of LogQL depends on being able to filter by app, namespace, environment, and similar labels. Establish a labeling convention and enforce it across deployments.
Enable Grafana authentication before exposing it beyond localhost. The default setup is unauthenticated and should not be reachable from outside the cluster without proper access controls in place.
Conclusion
The Grafana Loki stack gives you production-grade centralized logging without the operational cost of running Elasticsearch. Promtail handles log collection automatically as pods come and go, Loki keeps storage lean by indexing only labels, and Grafana provides the query interface your team needs to investigate incidents quickly.
The setup takes under an hour on an existing cluster. The time it saves during your first production incident pays for itself immediately.