loki_grafana_nginx_logs

@amitmund August 23, 2026

Loki + Grafana for Multiple Nginx Logs — Start to End

Stack: Loki (log storage/index) + Promtail (log shipper) + Grafana (visualization) + two nginx instances, fully containerized with Podman. Covers: architecture, nginx log formatting, Promtail scraping/parsing, LogQL queries, dashboards, alerting, label-cardinality pitfalls, and a full hands-on lab.

Why Loki (vs. shipping nginx logs to Elasticsearch or grepping files)
Architecture Overview
Core Concept: Labels vs. Parsed Fields (read this before writing any config)
Project Structure
Step 1 — Nginx Log Format (JSON, recommended)
Step 2 — Loki Configuration
Step 3 — Promtail Configuration (multiple nginx instances)
Step 4 — Podman Compose File (the whole stack)
Step 5 — Launch & Verify
Step 6 — Add Loki as a Grafana Data Source
Step 7 — Explore Your Nginx Logs
LogQL Reference for Nginx Logs
Step 8 — Build a Dashboard
Step 9 — Alerting on Error Rate
Multi-Instance Filtering with a Dashboard Variable
Common Mistakes & Troubleshooting
Production Notes (retention, scaling, security)
Full Hands-On Lab Checklist

1. Why Loki

Loki is built on one core idea: index only labels (metadata), not full log text. Compared to Elasticsearch-style full-text indexing, this makes Loki dramatically cheaper to run — at the cost of full-text search being slower (it greps chunks matching the label selector, rather than using an inverted text index). For nginx access/error logs, this trade-off is usually exactly right: you almost always filter by structural metadata first (which server, which log type, which time range) and only then look at content.

Loki pairs natively with Grafana (same company, designed together), so you get metrics (Prometheus/Mimir), logs (Loki), and traces (Tempo) in one UI — letting you pivot from "CPU spiked" straight to "here are the nginx errors from that exact minute" without switching tools.

2. Architecture Overview

┌───────────┐   writes    ┌──────────────┐
│ nginx-web1 │────────────▶│ access.log   │
│           │             │ error.log    │
└───────────┘             └──────┬───────┘
                                  │ tailed by
┌───────────┐   writes    ┌──────▼───────┐      pushes      ┌────────┐
│ nginx-web2 │────────────▶│  Promtail    │─────────────────▶│  Loki   │
│           │             │ (log shipper)│   over HTTP       │(storage/│
└───────────┘             └──────────────┘                   │ index)  │
                                                               └────┬────┘
                                                                    │ queried by
                                                              ┌─────▼─────┐
                                                              │  Grafana   │
                                                              │(dashboards,│
                                                              │ Explore,   │
                                                              │ alerts)    │
                                                              └───────────┘

Promtail tails log files and attaches labels (structural metadata), then pushes log lines + labels to Loki.
Loki indexes only the labels; log content is stored in compressed chunks.
Grafana queries Loki with LogQL — similar in spirit to PromQL, but for logs.

3. Core Concept: Labels vs. Parsed Fields

This is the single most important thing to get right with Loki, and the most common beginner mistake.

A label (e.g. job="nginx", instance="web1") creates a separate stream in Loki's index for every unique combination of label values. Few labels, each with few possible values → fast queries, small index.
If you make something like remote_addr or request_uri a label, every distinct IP address or URL creates a brand-new stream — this is called cardinality explosion and will degrade or crash a Loki deployment as traffic grows.
The fix: keep labels to genuinely structural, low-cardinality values (job, instance, log_type, environment). Everything else (status code, method, path, request time, user agent) should be extracted as a parsed field at query time using LogQL's | json or | logfmt or | pattern parsers — not baked into the label set.

This guide's Promtail config follows that rule: only job, instance, and log_type are labels. Status codes, methods, and timings are parsed on demand in LogQL queries (Section 12).

4. Project Structure

loki-nginx-lab/
├── podman-compose.yml
├── loki/
│   └── loki-config.yaml
├── promtail/
│   └── promtail-config.yaml
├── nginx/
│   ├── web1.conf
│   └── web2.conf
├── nginx-logs/
│   ├── web1/        (populated at runtime)
│   └── web2/        (populated at runtime)
└── grafana/
    └── provisioning/
        └── datasources/
            └── loki.yml

mkdir -p loki-nginx-lab/{loki,promtail,nginx,nginx-logs/web1,nginx-logs/web2,grafana/provisioning/datasources}
cd loki-nginx-lab

5. Step 1 — Nginx Log Format (JSON, recommended)

Nginx's default "combined" log format is plain text, which works with Loki too (via a regex pipeline stage), but a JSON log format is far simpler to parse reliably and is what this guide uses.

# nginx/web1.conf
log_format loki_json escape=json
  '{'
    '"time":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"status":"$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_user_agent":"$http_user_agent"'
  '}';

server {
    listen 80;
    server_name web1;

    access_log /var/log/nginx/access.log loki_json;
    error_log  /var/log/nginx/error.log warn;

    location / {
        root  /usr/share/nginx/html;
        index index.html;
    }

    # Test endpoints to generate different status codes later
    location /status/500 { return 500 "forced 500\n"; }
    location /status/404 { return 404 "forced 404\n"; }
    location /slow {
        # simulate latency for testing request_time-based queries
        echo_sleep 0.4;
        return 200 "slow response\n";
    }
}

# nginx/web2.conf — identical, just a different server_name for labeling
log_format loki_json escape=json
  '{'
    '"time":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"status":"$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_user_agent":"$http_user_agent"'
  '}';

server {
    listen 80;
    server_name web2;

    access_log /var/log/nginx/access.log loki_json;
    error_log  /var/log/nginx/error.log warn;

    location / {
        root  /usr/share/nginx/html;
        index index.html;
    }

    location /status/500 { return 500 "forced 500\n"; }
    location /status/404 { return 404 "forced 404\n"; }
}

The /slow endpoint uses the echo nginx module for a fake delay — if your nginx image doesn't include it, just skip that block; it's only there to generate varied request_time values for the latency query examples.

Already have real nginx servers with the default combined log format instead? Use this Promtail pipeline stage instead of json (Section 7):

pipeline_stages:
  - regex:
      expression: '^(?P<remote_addr>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<bytes>\d+)'

6. Step 2 — Loki Configuration

A minimal single-binary config, filesystem storage — fine for a personal lab or small deployment (see Section 17 for production scaling notes).

# loki/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

7. Step 3 — Promtail Configuration (multiple nginx instances)

# promtail/promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: nginx-web1-access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          instance: web1
          log_type: access
          __path__: /var/log/nginx/web1/access.log
    pipeline_stages:
      - json:
          expressions:
            time: time
      - timestamp:
          source: time
          format: RFC3339

  - job_name: nginx-web1-error
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          instance: web1
          log_type: error
          __path__: /var/log/nginx/web1/error.log

  - job_name: nginx-web2-access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          instance: web2
          log_type: access
          __path__: /var/log/nginx/web2/access.log
    pipeline_stages:
      - json:
          expressions:
            time: time
      - timestamp:
          source: time
          format: RFC3339

  - job_name: nginx-web2-error
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          instance: web2
          log_type: error
          __path__: /var/log/nginx/web2/error.log

Notice the access-log jobs only use the json + timestamp pipeline stages to set the real event time (from nginx's own timestamp, not Promtail's ingestion time) — they deliberately do not promote status/method/etc. to labels, per Section 3. Those fields still exist in the log line and get parsed at query time instead.

Adding a third nginx instance later? Copy one access + one error scrape_config block, change instance: web3 and the __path__. That's the entire scaling story for more nginx servers.

8. Step 4 — Podman Compose File (the whole stack)

# podman-compose.yml
version: "3.8"
services:
  loki:
    image: docker.io/grafana/loki:latest
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki/loki-config.yaml:/etc/loki/loki-config.yaml:Z
      - loki-data:/loki
    ports:
      - "3100:3100"

  promtail:
    image: docker.io/grafana/promtail:latest
    command: -config.file=/etc/promtail/promtail-config.yaml
    volumes:
      - ./promtail/promtail-config.yaml:/etc/promtail/promtail-config.yaml:Z
      - ./nginx-logs/web1:/var/log/nginx/web1:Z
      - ./nginx-logs/web2:/var/log/nginx/web2:Z
    depends_on:
      - loki

  nginx-web1:
    image: docker.io/library/nginx:latest
    volumes:
      - ./nginx/web1.conf:/etc/nginx/conf.d/default.conf:Z
      - ./nginx-logs/web1:/var/log/nginx:Z
    ports:
      - "8081:80"

  nginx-web2:
    image: docker.io/library/nginx:latest
    volumes:
      - ./nginx/web2.conf:/etc/nginx/conf.d/default.conf:Z
      - ./nginx-logs/web2:/var/log/nginx:Z
    ports:
      - "8082:80"

  grafana:
    image: docker.io/grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=lab123
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:Z
    depends_on:
      - loki

volumes:
  loki-data:

# grafana/provisioning/datasources/loki.yml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    isDefault: false

9. Step 5 — Launch & Verify

podman-compose up -d
podman-compose ps

Generate some traffic across both instances, including errors:

for i in $(seq 1 20); do curl -s http://localhost:8081/ > /dev/null; done
for i in $(seq 1 5);  do curl -s http://localhost:8081/status/500 > /dev/null; done
for i in $(seq 1 5);  do curl -s http://localhost:8081/status/404 > /dev/null; done

for i in $(seq 1 15); do curl -s http://localhost:8082/ > /dev/null; done
for i in $(seq 1 3);  do curl -s http://localhost:8082/status/500 > /dev/null; done

Confirm log files are actually being written:

tail -f nginx-logs/web1/access.log

Confirm Loki is receiving data (should return label names including job, instance, log_type):

curl -s http://localhost:3100/loki/api/v1/labels | python3 -m json.tool

10. Step 6 — Add Loki as a Grafana Data Source

Already provisioned automatically via Section 8's loki.yml — confirm it in the UI: log into Grafana (http://localhost:3000, admin/lab123) → Connections → Data sources → Loki should already be listed.

If you're adding it manually instead: Add data source → Loki → URL http://loki:3100 → Save & Test.

11. Step 7 — Explore Your Nginx Logs

Go to Explore (left sidebar) → select the Loki data source.

Start with the simplest possible query — every access log from web1:

{job="nginx", instance="web1", log_type="access"}

You should see raw JSON log lines streaming in, newest first. Try:

{job="nginx", log_type="access"}

to see both instances interleaved (no instance filter = all instances).

12. LogQL Reference for Nginx Logs

Basic label filtering

# All logs, both instances, access only
{job="nginx", log_type="access"}

# Just web2's errors
{job="nginx", instance="web2", log_type="error"}

Text filtering (before parsing — cheap, runs first)

{job="nginx", log_type="error"} |= "connect() failed"
{job="nginx", log_type="access"} != "healthcheck"

Parsing JSON fields at query time

{job="nginx", log_type="access"} | json

This makes every JSON key (status, request_method, request_uri, request_time, etc.) available as a field for further filtering/aggregation in the same query.

Filtering on a parsed field

# Only 5xx responses
{job="nginx", log_type="access"} | json | status >= 500

# Only a specific method
{job="nginx", log_type="access"} | json | request_method="POST"

Turning logs into metrics — request rate

# Requests/sec, per instance
sum by (instance) (rate({job="nginx", log_type="access"}[1m]))

Error rate as a percentage

sum(rate({job="nginx", log_type="access"} | json | status >= 500 [5m]))
/
sum(rate({job="nginx", log_type="access"} [5m]))

Status code breakdown

sum by (status) (
  count_over_time({job="nginx", log_type="access"} | json [5m])
)

Top requested paths

topk(10,
  sum by (request_uri) (
    count_over_time({job="nginx", log_type="access"} | json [5m])
  )
)

Latency percentile (needs `unwrap` to treat a field as a number)

quantile_over_time(0.95,
  {job="nginx", log_type="access"} | json | unwrap request_time [5m]
) by (instance)

Combined: slow AND erroring requests

{job="nginx", log_type="access"}
  | json
  | status >= 500
  | request_time > 0.3

13. Step 8 — Build a Dashboard

Create a new dashboard, folder "Nginx", and add these panels:

Panel 1 — Request Rate by Instance (Time series)

sum by (instance) (rate({job="nginx", log_type="access"}[1m]))

Panel 2 — Status Code Breakdown (Bar chart)

sum by (status) (count_over_time({job="nginx", log_type="access"} | json [5m]))

Panel 3 — 95th Percentile Response Time (Time series, unit: seconds)

quantile_over_time(0.95, {job="nginx", log_type="access"} | json | unwrap request_time [5m]) by (instance)

Panel 4 — Live Error Logs (Logs panel)

{job="nginx", log_type="error"}

Panel 5 — Top Requested Paths (Table)

topk(10, sum by (request_uri) (count_over_time({job="nginx", log_type="access"} | json [5m])))

Save the dashboard as "Nginx Overview."

14. Step 9 — Alerting on Error Rate

Alerting → Alert rules → New alert rule:

Query:

sum(rate({job="nginx", log_type="access"} | json | status >= 500 [5m]))
/
sum(rate({job="nginx", log_type="access"} [5m]))

Condition: IS ABOVE 0.05 (fires if more than 5% of requests are 5xx)
Evaluation: every 1m, for 5m (avoid flapping on brief spikes)
Labels: severity: critical

Test it:

for i in $(seq 1 50); do curl -s http://localhost:8081/status/500 > /dev/null; done

Wait ~5 minutes and check Alerting → Alert rules — it should transition to Firing.

15. Multi-Instance Filtering with a Dashboard Variable

Dashboard settings → Variables → New variable: - Name: instance - Type: Query, Data source: Loki - Query: label_values({job="nginx"}, instance) - Enable Multi-value + Include All

Update panel queries to use it:

sum by (instance) (rate({job="nginx", log_type="access", instance=~"$instance"}[1m]))

Now the dashboard has a dropdown to switch between web1, web2, or both — without duplicating a single panel.

16. Common Mistakes & Troubleshooting

Symptom	Likely Cause	Fix
No logs appear in Explore	Promtail can't reach Loki, or wrong `__path__`	Check `podman logs promtail`; confirm the mounted log path matches `__path__` exactly
Loki index growing huge / queries slow	Too many labels, or labels with high cardinality (path, IP, user agent as labels)	Remove those from `labels:` in Promtail config; parse them at query time instead (Section 3)
Timestamps look wrong / out of order	Using Promtail's ingestion time instead of nginx's own timestamp	Confirm the `timestamp` pipeline stage is present and pointed at the parsed `time` field
`\| json` query returns no extra fields	Log line isn't actually valid JSON (e.g. still using default combined format)	Confirm nginx is using the `loki_json` log format from Section 5, not the default
Podman volume permission errors	Rootless Podman + SELinux relabeling not applied	Confirm every bind mount has the `:Z` suffix (Section 8 already includes it)
Alert never fires despite real errors	Query syntax issue, or "for" duration longer than the test traffic burst	Test the raw query first in Explore before wiring it into an alert rule
Positions file resets, re-reads whole log on restart	`positions.yaml` not persisted across container restarts	Mount `/tmp` (or a dedicated path) as a named volume if you need this to survive restarts

17. Production Notes

Retention: this lab config keeps data forever by default. In production, configure a compactor with a retention_period (e.g. 30 days) so old chunks are automatically deleted — unbounded retention on filesystem storage will eventually fill the disk.
Scaling beyond one binary: the config here runs Loki as a single "monolithic" process — fine for a personal lab or small fleet. For serious production log volume, Loki supports a microservices mode (separate ingester/querier/distributor components) that scales horizontally.
Object storage: swap filesystem storage for S3/GCS/Azure Blob in any real deployment — filesystem storage doesn't survive a lost disk and doesn't scale past one node's local storage.
Security: this lab runs with auth_enabled: false — anyone who can reach port 3100 can push/query without authentication. Production Loki needs either its multi-tenancy auth enabled behind a proper identity layer, or to be firewalled so only trusted services (Promtail, Grafana) can reach it.
Log rotation: nginx's own logrotate config will rename/truncate log files periodically — Promtail handles nginx's default rotation correctly out of the box (it tracks file position, and correctly detects truncation/recreation), but confirm this behavior if you use a custom rotation setup.

18. Full Hands-On Lab Checklist

Create the project structure (Section 4)
Write both nginx configs with the JSON log format (Section 5)
Write the Loki config (Section 6)
Write the Promtail config for both instances (Section 7)
Write the Podman Compose file + Grafana data source provisioning (Section 8)
podman-compose up -d and confirm all 5 containers are running
Generate mixed traffic (success + 404 + 500) against both nginx instances
Confirm labels exist via curl localhost:3100/loki/api/v1/labels
Run the basic label-filter query in Explore
Run each LogQL query from Section 12 and confirm sensible output
Build all 5 dashboard panels from Section 13
Add the $instance variable and confirm the dropdown filters correctly
Create the error-rate alert rule and trigger it with a burst of 500s
Tear down: podman-compose down -v

loki_grafana_nginx_logs

Loki + Grafana for Multiple Nginx Logs — Start to End

Table of Contents

1. Why Loki

2. Architecture Overview

3. Core Concept: Labels vs. Parsed Fields

4. Project Structure

5. Step 1 — Nginx Log Format (JSON, recommended)

6. Step 2 — Loki Configuration

7. Step 3 — Promtail Configuration (multiple nginx instances)

8. Step 4 — Podman Compose File (the whole stack)

9. Step 5 — Launch & Verify

10. Step 6 — Add Loki as a Grafana Data Source

11. Step 7 — Explore Your Nginx Logs

12. LogQL Reference for Nginx Logs

Basic label filtering

Text filtering (before parsing — cheap, runs first)

Parsing JSON fields at query time

Filtering on a parsed field

Turning logs into metrics — request rate

Error rate as a percentage

Status code breakdown

Top requested paths

Latency percentile (needs `unwrap` to treat a field as a number)

Combined: slow AND erroring requests

13. Step 8 — Build a Dashboard

14. Step 9 — Alerting on Error Rate

15. Multi-Instance Filtering with a Dashboard Variable

16. Common Mistakes & Troubleshooting

17. Production Notes

18. Full Hands-On Lab Checklist

Filters

Loki + Grafana for Multiple Nginx Logs — Start to End

Table of Contents

1. Why Loki

2. Architecture Overview

3. Core Concept: Labels vs. Parsed Fields

4. Project Structure

5. Step 1 — Nginx Log Format (JSON, recommended)

6. Step 2 — Loki Configuration

7. Step 3 — Promtail Configuration (multiple nginx instances)

8. Step 4 — Podman Compose File (the whole stack)

9. Step 5 — Launch & Verify

10. Step 6 — Add Loki as a Grafana Data Source

11. Step 7 — Explore Your Nginx Logs

12. LogQL Reference for Nginx Logs

Basic label filtering

Text filtering (before parsing — cheap, runs first)

Parsing JSON fields at query time

Filtering on a parsed field

Turning logs into metrics — request rate

Error rate as a percentage

Status code breakdown

Top requested paths

Latency percentile (needs unwrap to treat a field as a number)

Combined: slow AND erroring requests

13. Step 8 — Build a Dashboard

14. Step 9 — Alerting on Error Rate

15. Multi-Instance Filtering with a Dashboard Variable

16. Common Mistakes & Troubleshooting

17. Production Notes

18. Full Hands-On Lab Checklist

Filters

Latency percentile (needs `unwrap` to treat a field as a number)