loki_grafana for a single nginx host and multiple sites

@amitmund August 23, 2026

Loki + Grafana for a Single Nginx Host, Multiple Sites

Environment: ONE host, ONE nginx process, MULTIPLE sites (server blocks), each with its own access.log and error.log. Nginx runs natively on the host; Loki, Promtail, and Grafana run as Podman containers alongside it.

Covers exactly what you asked for — total requests, requests over a selected time range, status code totals, top URIs, slowest-requests sorting, site up/down — plus a few extra metrics worth having.

Architecture for This Environment
Step 1 — Shared Nginx Log Format (one definition, many sites)
Step 2 — Per-Site Server Block Example
Step 3 — Promtail Config (one block per site, explicit and reliable)
Step 4 — Podman Compose (Loki + Promtail + Grafana only)
Step 5 — Launch & Verify
Total Number of Requests
Requests Over a Selected Time Range
Total Number of Each Status Code
Top URIs
Sorting by Response Time (slowest requests)
Site Up/Down Monitoring
Extra Metrics Worth Adding
Full Dashboard Layout Summary
Alerting Examples
Troubleshooting

1. Architecture for This Environment

┌─────────────────────────────────────────┐
│              Host machine                 │
│                                           │
│  nginx (native install)                  │
│   ├─ site1-access.log / site1-error.log   │
│   ├─ site2-access.log / site2-error.log   │
│   └─ site3-access.log / site3-error.log   │
│              │ (read-only bind mount)     │
│              ▼                            │
│  ┌─────────────────────────────────────┐ │
│  │  Podman containers                    │ │
│  │  ┌──────────┐  ┌──────┐  ┌─────────┐ │ │
│  │  │ Promtail  │─▶│ Loki  │◀─│ Grafana │ │ │
│  │  └──────────┘  └──────┘  └─────────┘ │ │
│  └─────────────────────────────────────┘ │
└─────────────────────────────────────────┘

Only Loki, Promtail, and Grafana are containerized. Nginx keeps running exactly as it does today — Promtail just reads its log files.

2. Step 1 — Shared Nginx Log Format

Define the JSON log format once, in the main http {} context (/etc/nginx/nginx.conf, or a file included from it) — every site's server block then just references it by name.

# inside the http {} block of /etc/nginx/nginx.conf
log_format loki_json escape=json
  '{'
    '"time":"$time_iso8601",'
    '"remote_addr":"$remote_addr",'
    '"request_method":"$request_method",'
    '"request_uri":"$request_uri",'
    '"status":"$status",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"http_user_agent":"$http_user_agent",'
    '"http_referer":"$http_referer"'
  '}';

3. Step 2 — Per-Site Server Block Example

# /etc/nginx/sites-available/site1.conf
server {
    listen 80;
    server_name site1.example.com;

    access_log /var/log/nginx/site1-access.log loki_json;
    error_log  /var/log/nginx/site1-error.log warn;

    root /var/www/site1;
}

# /etc/nginx/sites-available/site2.conf
server {
    listen 80;
    server_name site2.example.com;

    access_log /var/log/nginx/site2-access.log loki_json;
    error_log  /var/log/nginx/site2-error.log warn;

    root /var/www/site2;
}

Repeat for every site — the only things that change per site are the filename prefix and server_name/root. Reload nginx after adding configs:

sudo nginx -t && sudo systemctl reload nginx

Already have existing sites with the default combined format? You don't have to touch working sites — add access_log ... loki_json; as a second access_log line alongside your existing one to get JSON logs for Loki without removing your current logging:

access_log /var/log/nginx/site1-access.log        combined;   # existing
access_log /var/log/nginx/site1-access-json.log   loki_json;  # new, for Loki

Then point Promtail at the -json.log files instead.

4. Step 3 — Promtail Config

One explicit block per site keeps this reliable and easy to reason about — copy-paste and rename for each new site.

# promtail/promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: site1-access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          site: site1
          log_type: access
          __path__: /var/log/nginx/site1-access.log
    pipeline_stages:
      - json:
          expressions:
            time: time
      - timestamp:
          source: time
          format: RFC3339

  - job_name: site1-error
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          site: site1
          log_type: error
          __path__: /var/log/nginx/site1-error.log

  - job_name: site2-access
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          site: site2
          log_type: access
          __path__: /var/log/nginx/site2-access.log
    pipeline_stages:
      - json:
          expressions:
            time: time
      - timestamp:
          source: time
          format: RFC3339

  - job_name: site2-error
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          site: site2
          log_type: error
          __path__: /var/log/nginx/site2-error.log

site here is a label with as many values as you have sites — that's bounded and stable (doesn't grow with traffic), so it's exactly the kind of label Loki is fine with, unlike per-request values (Section 3 of the previous guide covers why that distinction matters).

Bonus, no extra config needed: Promtail automatically attaches a filename label to every line showing the exact file it came from — handy for ad-hoc debugging even before you've added a clean site label.

5. Step 4 — Podman Compose (Loki + Promtail + Grafana only)

# podman-compose.yml
version: "3.8"
services:
  loki:
    image: docker.io/grafana/loki:latest
    command: -config.file=/etc/loki/loki-config.yaml
    volumes:
      - ./loki/loki-config.yaml:/etc/loki/loki-config.yaml:Z
      - loki-data:/loki
    ports:
      - "3100:3100"

  promtail:
    image: docker.io/grafana/promtail:latest
    command: -config.file=/etc/promtail/promtail-config.yaml
    volumes:
      - ./promtail/promtail-config.yaml:/etc/promtail/promtail-config.yaml:Z
      - /var/log/nginx:/var/log/nginx:ro,Z
    depends_on:
      - loki

  grafana:
    image: docker.io/grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme123
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:Z
    depends_on:
      - loki

volumes:
  loki-data:

Use the same loki-config.yaml and Grafana datasource provisioning file from the previous guide — nothing about Loki or Grafana's own config changes here, only Promtail's target paths and the fact that nginx itself isn't containerized.

6. Step 5 — Launch & Verify

podman-compose up -d
curl -s http://localhost:3100/loki/api/v1/label/site/values | python3 -m json.tool

You should see ["site1", "site2", ...] — confirming Loki is receiving labeled logs from every site.

7. Total Number of Requests

Stat panel — total requests in whatever time range is currently selected on the dashboard ($__range is a Grafana built-in that always equals the selected range, so no manual config needed when you change the time picker):

sum(count_over_time({job="nginx", log_type="access"}[$__range]))

Per site instead of all sites combined:

sum by (site) (count_over_time({job="nginx", log_type="access"}[$__range]))

8. Requests Over a Selected Time Range

This is really the same capability as above, just as a time series panel instead of a single number — Grafana automatically re-queries whenever you change the time picker (top right) or zoom into part of a graph, no extra config needed:

sum by (site) (rate({job="nginx", log_type="access"}[$__interval]))

$__interval auto-adjusts the bucket size based on the zoom level (wider range = coarser buckets, so the graph stays readable whether you're looking at the last hour or the last month).

9. Total Number of Each Status Code

Bar chart or pie chart:

sum by (status) (count_over_time({job="nginx", log_type="access"} | json [$__range]))

Per-site breakdown (useful to see if one site is having more errors than others):

sum by (site, status) (count_over_time({job="nginx", log_type="access"} | json [$__range]))

10. Top URIs

Table panel, all sites combined:

topk(10,
  sum by (request_uri) (
    count_over_time({job="nginx", log_type="access"} | json [$__range])
  )
)

Scoped to one site via a dashboard variable (Section 14 shows how to set up $site):

topk(10,
  sum by (request_uri) (
    count_over_time({job="nginx", log_type="access", site="$site"} | json [$__range])
  )
)

11. Sorting by Response Time (slowest requests)

Two complementary views — one for "how's latency trending," one for "show me the actual slow requests."

A. Latency percentiles over time (Time series panel)

quantile_over_time(0.95,
  {job="nginx", log_type="access"} | json | unwrap request_time [$__interval]
) by (site)

Add a second query at 0.50 for median alongside p95 to see the full spread.

B. An actual sorted list of the slowest requests (Table panel)

LogQL streams logs in time order, not by value — so to get a true "slowest first" ranking, query for slow requests, then sort the resulting table in Grafana itself:

{job="nginx", log_type="access"} | json | request_time > 0.5

Set the panel visualization to Table.
Panel editor → Transform tab → add "Sort by" → field: request_time → Descending.
Optionally add "Organize fields" to hide noisy columns and keep just site, request_uri, status, request_time.

Adjust the 0.5 threshold to whatever "slow" means for your sites.

12. Site Up/Down Monitoring

Important distinction: access logs alone can't reliably tell you a site is down — zero log lines could mean "down" or could just mean "no visitors right now." You need an active check (something that deliberately requests the site on a schedule, regardless of real traffic).

Option A — Loki-native heartbeat (no new stack, recommended for your setup)

A tiny script that curls every site on a schedule and logs the result as JSON, ingested by Promtail just like nginx logs.

#!/bin/bash
# /usr/local/bin/site-heartbeat.sh
SITES=("site1:http://localhost/health" "site2:http://localhost:8082/health")
LOGFILE="/var/log/nginx/heartbeat.log"

for entry in "${SITES[@]}"; do
  site="${entry%%:*}"
  url="${entry#*:}"
  start=$(date +%s.%N)
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$url")
  end=$(date +%s.%N)
  elapsed=$(echo "$end - $start" | bc)
  healthy=false
  [ "$code" = "200" ] && healthy=true

  echo "{\"time\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"site\":\"$site\",\"http_code\":\"$code\",\"response_time\":$elapsed,\"healthy\":$healthy}" >> "$LOGFILE"
done

Run it every minute via cron or a systemd timer:

* * * * * /usr/local/bin/site-heartbeat.sh

Add a Promtail scrape block for it:

  - job_name: heartbeat
    static_configs:
      - targets: [localhost]
        labels:
          job: heartbeat
          __path__: /var/log/nginx/heartbeat.log
    pipeline_stages:
      - json:
          expressions:
            time: time
            site: site
            healthy: healthy
      - timestamp:
          source: time
          format: RFC3339
      - labels:
          site:

Stat panel — current status per site (green "UP" / red "DOWN" via value mappings):

sum by (site) (count_over_time({job="heartbeat"} | json | healthy="true" [5m]))

Set Field → Value mappings: 0 → "DOWN" (red), anything > 0 → "UP" (green).

Alert rule — fires when a site has had zero successful checks in 5 minutes:

sum by (site) (count_over_time({job="heartbeat"} | json | healthy="true" [5m]))

Condition: IS BELOW 1, evaluated every 1m, for 2m.

Option B — Prometheus Blackbox Exporter (the more "standard" uptime tool)

If you're open to adding Prometheus alongside Loki, blackbox_exporter is purpose-built for this: it probes HTTP(S)/TCP/ICMP targets and exposes probe_success (1/0) and probe_duration_seconds as real Prometheus metrics, with battle-tested Grafana dashboards already available in the community dashboard library. More moving parts than Option A, but a more conventional/robust choice if uptime monitoring becomes a bigger priority later (e.g. you also want SSL certificate expiry monitoring, which blackbox_exporter supports natively and Option A does not).

13. Extra Metrics Worth Adding

Bandwidth per site

sum by (site) (sum_over_time({job="nginx", log_type="access"} | json | unwrap body_bytes_sent [$__range]))

Approximate unique visitors (safe — doesn't label by IP)

count(count by (remote_addr) (count_over_time({job="nginx", log_type="access"} | json [$__range])))

This counts distinct IPs at query time without ever making remote_addr a label, so it doesn't create the cardinality problem discussed earlier.

Top error messages

Nginx's error log isn't JSON by default; parse it with a pattern stage:

{job="nginx", log_type="error"} | pattern `<_> [<level>] <_>: <msg>`

Then rank the most common error types:

topk(10, sum by (msg) (count_over_time({job="nginx", log_type="error"} | pattern `<_> [<level>] <_>: <msg>` [$__range])))

Bot / crawler traffic

sum(count_over_time({job="nginx", log_type="access"} | json | http_user_agent=~".*(?i)(bot|crawl|spider).*" [$__range]))

Top referrers

topk(10, sum by (http_referer) (count_over_time({job="nginx", log_type="access"} | json | http_referer != "-" [$__range])))

SSL certificate expiry (pointer, not a Loki query)

Logs can't tell you when a cert expires — this needs an active check too. blackbox_exporter's HTTPS probe module reports probe_ssl_earliest_cert_expiry natively (see Option B above), or a simpler standalone cron script using openssl s_client piped into a Loki-ingested log line, similar to the heartbeat pattern in Section 12.

14. Full Dashboard Layout Summary

Panel	Type	Query (see section)
Total Requests	Stat	Section 7
Requests Over Time (per site)	Time series	Section 8
Status Code Breakdown	Bar/Pie	Section 9
Top URIs	Table	Section 10
Latency Percentiles	Time series	Section 11-A
Slowest Requests	Table (sorted)	Section 11-B
Site Up/Down	Stat (per site)	Section 12
Bandwidth per Site	Time series	Section 13
Unique Visitors (approx)	Stat	Section 13
Top Error Messages	Table	Section 13
Live Error Logs	Logs panel	`{job="nginx", log_type="error"}`

Add a $site variable (Dashboard settings → Variables): - Query: label_values({job="nginx"}, site) - Enable Multi-value + Include All

Then swap site="$site" (or site=~"$site" for multi-value) into any panel above to make the whole dashboard filterable by site via one dropdown.

15. Alerting Examples

High error rate, per site:

sum by (site) (rate({job="nginx", log_type="access"} | json | status >= 500 [5m]))
/
sum by (site) (rate({job="nginx", log_type="access"} [5m]))

Condition: IS ABOVE 0.05, for 5m.

Site down — see Section 12's heartbeat alert.

Unusually slow site:

quantile_over_time(0.95, {job="nginx", log_type="access", site="site1"} | json | unwrap request_time [5m])

Condition: IS ABOVE 1 (seconds — adjust to your normal baseline).

16. Troubleshooting

Symptom	Likely Cause	Fix
No logs from a specific site	Wrong `__path__` in Promtail, or nginx config not reloaded	Confirm the file exists on the host at that exact path; `nginx -t && systemctl reload nginx`
Promtail can't read log files	Permissions — nginx logs are often root-owned, Promtail container runs as a different user	Bind-mount as shown with `:ro,Z`; if still denied, check the log file's mode/owner on the host
`$__range` query feels slow on a big time window	Scanning a lot of chunks for a large range	Normal for Loki on big ranges — narrow the time picker while iterating, widen only when needed
Heartbeat panel always shows "DOWN"	Health check URL wrong, or script not actually running on schedule	Run the script manually once and check `heartbeat.log` directly; verify the cron/timer is actually firing
`\| pattern` query for errors returns nothing	Nginx's actual error log format doesn't match the pattern string	Run `tail -5 /var/log/nginx/site1-error.log` and adjust the pattern to match the real format exactly