Centralized logging is essential in production. Elasticsearch-based stacks (ELK) are powerful but expensive: they index every word in every log line, requiring massive storage and compute. Loki takes a different approach. It indexes only the labels (metadata about logs, like service name or environment), then stores the raw logs compressed. This makes it 10-100x cheaper while keeping you able to search and query logs in a Grafana dashboard.

If you’ve been avoiding centralized logging because of cost, Loki changes the equation. This guide covers the architecture, shows you how to configure Promtail to tail reverse-proxy JSON logs, writes LogQL queries you can copy, and explains the critical labels-vs-cardinality tradeoff that will save you from a surprise bill.

How Loki Works: Labels, Not Full-Text Search

Traditional logging (ELK) indexes every word:

"2026-06-16 10:15:23 ERROR [api] Request timeout for user_id=42 service=payment"

Elasticsearch stores this as a document with fields: timestamp, level, service, user_id, message. It indexes each field, so you can query “all logs where level=ERROR and user_id=42”. But storing and indexing a million log lines a day costs thousands of dollars monthly.

Loki indexes only labels:

{job="api", env="prod", level="error"} 2026-06-16T10:15:23Z "Request timeout for user_id=42 service=payment"

The labels (job, env, level) are indexed. The log line itself is not. To search for “user_id=42”, you run a LogQL query that:

  1. Selects logs with matching labels
  2. Filters the raw log content (slower, but limited to fewer logs)
{job="api", env="prod"}
| json
| line_format "{{.level}}: {{.message}}"
| user_id="42"

Result: 100x less storage, 100x lower cost, slightly slower full-text search (which you rarely need in practice).

Architecture: Loki (Storage) + Promtail (Agent)

Loki is the server. It stores logs and answers LogQL queries. It needs a backend (S3, MinIO, local filesystem for dev) to store actual log data.

Promtail is the agent. It runs on your Docker host (or in a container) and tails log files, scrapes container stdout/stderr, or reads syslog. It parses logs, adds labels, then ships them to Loki.

Container logs
      ↓
    [Promtail] (reads docker logs, parses, adds labels)
      ↓
    [Loki] (indexes labels, stores log lines)
      ↓
   [Grafana] (queries via LogQL)

docker-compose.yml: Loki + Promtail

Here’s a minimal production setup:

version: "3.9"

services:
  loki:
    image: grafana/loki:2.9
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/local-config.yml
      - loki_data:/loki
    command: -config.file=/etc/loki/local-config.yml
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3100/loki/api/v1/status/buildinfo"]
      interval: 30s

  promtail:
    image: grafana/promtail:2.9
    container_name: promtail
    volumes:
      - ./promtail-config.yml:/etc/promtail/config.yml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - loki

  api:
    image: myapp:latest
    container_name: api
    expose:
      - 3000
    # No special logging config needed
    # Promtail will tail container logs automatically

volumes:
  loki_data:
  grafana_data:

Loki Configuration: Storage and Retention

# loki-config.yml
auth_enabled: false

ingester:
  chunk_idle_period: 3m
  max_chunk_age: 1h
  max_streams_matcher_size: 10000
  chunk_encoding: snappy

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_streams_limit_bytes: 1000000  # 1MB per tenant

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/chunks
  index_cache_validity: 5m

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h  # Keep logs for 30 days

server:
  http_listen_port: 3100
  log_level: info

For production with large log volume, replace filesystem with S3:

storage_config:
  s3:
    s3: s3://region/bucket
    endpoint: https://s3.amazonaws.com
  aws:
    s3forcepathstyle: true

Promtail Configuration: Tailing Docker Logs

Promtail needs to know where logs are and what labels to add. Here’s a config that tails all container logs and parses JSON from a Traefik reverse proxy:

# promtail-config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/promtail-positions.yml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # Docker container logs
  - job_name: docker
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker
          __path__: /var/lib/docker/containers/*/*.log

    docker_sd_configs:
      - host: unix:///var/run/docker.sock

    relabel_configs:
      # Extract container name as label
      - source_labels: [__meta_docker_container_name]
        target_label: container

      # Extract container label "service"
      - source_labels: [__meta_docker_container_label_service]
        target_label: service

    pipeline_stages:
      # Parse Docker JSON log format (JSON wrapper around actual log)
      - json:
          expressions:
            message: log
            stream: stream

      # Extract timestamp
      - timestamp:
          source: time
          format: "2006-01-02T15:04:05.999999999Z07:00"

      # If the message looks like JSON (Traefik access logs), parse it
      - json:
          expressions:
            status_code: status_code
            request_host: request_host
            request_path: request_path
            duration_ms: request_duration_ms
          source: message

      # Create a level label based on status code
      - labels:
          status:
          request_host:
          service:

  # Traefik access logs specifically
  - job_name: traefik-access
    static_configs:
      - targets:
          - localhost
        labels:
          job: traefik
          __path__: /var/log/traefik/access.log

    pipeline_stages:
      # Traefik writes JSON one line per access
      - json:
          expressions:
            status_code: StatusCode
            request_host: RequestHost
            request_path: RequestPath
            duration: Duration
            method: RequestMethod
            remote_addr: RemoteAddr

      - labels:
          status_code:
          request_host:
          method:

LogQL: Querying Logs

LogQL is Grafana’s query language for Loki. Here are common patterns:

Basic label selection (indexed, fast)

# All logs from the api service
{service="api"}

# Logs from api where status is 5xx (parsed from JSON)
{service="api"} | json | status_code >= 500

# Multiple labels (AND)
{service="api", env="prod"}

# Label with regex (slower)
{service=~"api|web"}

Extracting fields and filtering

# Extract fields with json stage
{service="api"} | json | line_format "{{.status_code}} {{.request_path}}"

# Filter by extracted field
{service="api"} | json | status_code="500"

# Count errors per minute
{service="api"} | json | status_code >= 500 | rate(1m)

# Extract latency and show histogram
{service="api"} | json | duration > 1000 | rate(1m)

# Search for specific string in logs (grep)
{service="api"} | "timeout"

# Exclude pattern (negative grep)
{service="api"} != "health"

# Multiple conditions with pipe
{service="api"} | json | status_code >= 500 | "user_id" | duration > 5000

Metric queries (derived from logs)

# Request rate per second
rate({service="api"}[5m])

# Error rate
sum(rate({service="api"} | json | status_code >= 500 [5m]))
/
sum(rate({service="api"} [5m]))

# P99 latency (if you extract duration field)
histogram_quantile(0.99, sum(rate({service="api"} | json | duration [5m])) by (le))

# Errors per minute, broken down by status code
sum by (status_code) (rate({service="api"} | json | status_code >= 400 [5m]))

# Count of unique hosts
count by (request_host) (rate({service="api"} | json [5m]))

In Grafana, set up a panel with type “Logs” and paste the LogQL query. Use “Table” panel type if you want to display parsed fields in a table format.

Promtail Pipeline Stages Explained

Promtail’s pipeline stages transform logs as they’re ingested. Understanding them helps you extract the most value:

  • json: Parse JSON log lines and extract fields
  • regex: Extract fields using regex patterns
  • timestamp: Extract and normalize the log timestamp
  • labels: Create labels from extracted fields (use sparingly!)
  • line_format: Reformat the log line for display
  • match: Conditional stages (only process if pattern matches)
  • drop: Discard logs matching a pattern

Example: parsing a non-JSON log format:

pipeline_stages:
  - regex:
      expression: "(?P<timestamp>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}) (?P<level>\\w+) (?P<service>\\w+) (?P<message>.*)"

  - timestamp:
      source: timestamp
      format: "2006-01-02 15:04:05"

  - labels:
      level:
      service:

  - line_format: "[{{.level}}] {{.message}}"

  # Drop verbose debug logs to save storage
  - match:
      selector: '{level="DEBUG"}'
      action: drop

This reduces log volume by filtering out noise before it reaches Loki.

The Labels-vs-Cardinality Gotcha

This is critical: Loki indexes labels. If you add too many labels or use high-cardinality values (like user IDs or request IDs), you’ll blow up your index size and tank performance.

Bad:

{service="api", user_id="42", request_id="abc123", session_id="xyz"}

If you have 1 million unique user_ids, the index will be enormous.

Good:

{service="api", env="prod", level="error"}

Only add labels you’ll query by. Everything else goes in the log line.

If you need to search by user_id, query the logs post-index:

{service="api"} | json | user_id="42"

This is slower (full-text search on the log lines), but limited to the logs with service="api", so still fast.

Comparing to ELK

FeatureLokiElasticsearch (ELK)
Index strategyLabels only (low cardinality)Full-text (every field)
Storage overhead~5% of log volume~500% of log volume
Query speedSlower for full-text, fast for labelsFast for full-text
Cost for 1TB/day logs~$500/month~$10,000/month
ComplexitySimple (few config options)Complex (tuning, scaling)
Good forSearching by metadata, high-volume logsKeyword search, security analytics

Use Loki if your logs are operational (finding that error across all services). Use Elasticsearch if your logs are forensic (finding any mention of “user X” across all logs ever, word search in security logs).

Setting Up Grafana Datasource

  1. Open Grafana at http://localhost:3000 (admin / admin)
  2. Settings → Data Sources → Add
  3. Type: Loki
  4. URL: http://loki:3100
  5. Save & Test

Then create a panel:

  1. Dashboards → New → New Dashboard
  2. Add Panel → Type: Logs
  3. Query: {service="api"} or your LogQL
  4. Save

Key Takeaways

  1. Loki indexes labels, not content — use only essential labels to keep the index small
  2. Promtail is the agent — it tails logs, parses them, adds labels, ships to Loki
  3. LogQL is simple — label selection is fast, content filtering is slower but scoped
  4. JSON parsing extracts fields — parse Docker/Traefik logs to create searchable fields
  5. Retention policy prevents runaway storage — delete logs older than 30 days (or your SLA)
  6. High-cardinality labels blow up the index — user_id and request_id belong in log content, not labels
  7. Use rate() and histogram_quantile for metrics — derive performance metrics from logs
  8. Loki is 10-100x cheaper than ELK — for operational logging at scale, the savings are real
  9. Set up dashboards for common queries — error rate, latency, slow endpoints are metrics you’ll query daily
  10. Pair Loki with Prometheus and Grafana — together they’re a complete observability stack