grafana demo with podman

@amitmund August 22, 2026

Covers: what Grafana is, installation, data sources, dashboards, panels, query languages, variables, transformations, alerting, provisioning-as-code, security, and a full hands-on lab building a real monitoring stack from scratch.

What Is Grafana & Why Use It
Architecture Overview
Installation (Docker, Linux, Windows, macOS, Cloud)
First Login & Navigation
Data Sources
Dashboards & Panels
Panel Types Explained
Query Languages — PromQL, LogQL, SQL
Variables & Templating
Transformations
Alerting (Unified Alerting)
Annotations
Provisioning as Code
Plugins
Users, Teams, Orgs & RBAC
Security Best Practices
Performance & Scaling Notes
The LGTM Stack (Loki, Grafana, Tempo, Mimir/Prometheus)
Common Mistakes & Best Practices
Hands-On Lab — Build a Full Monitoring Stack from Scratch
21. Quiz — Test Your Understanding

1. What Is Grafana & Why Use It

Grafana is an open-source visualization and observability platform. It doesn't store your data itself (in most setups) — it connects to data sources (Prometheus, InfluxDB, Loki, MySQL, Elasticsearch, CloudWatch, and 100+ others) and lets you query, visualize, alert on, and explore that data through dashboards.

Why it's the industry standard: - Data-source agnostic — one dashboard tool for metrics, logs, and traces from wildly different backends. - Rich visualization library — time series, heatmaps, gauges, geomaps, node graphs, and more, all configurable without code. - Templating — one dashboard can serve many hosts/services/environments via variables, instead of duplicating dashboards. - Alerting built in — define alert rules directly on your queries, route notifications to Slack, PagerDuty, email, webhooks, etc. - Provisioning as code — dashboards and data sources can be defined in YAML/JSON

and version-controlled, not just clicked together in a UI.

2. Architecture Overview

┌─────────────┐     queries      ┌──────────────────┐
│   Browser    │ ───────────────▶│  Grafana Server    │
│ (dashboards) │◀─────────────── │  (grafana-server)  │
└─────────────┘   rendered data  └─────────┬──────────┘
                                            │ queries data sources
                       ┌────────────────────┼────────────────────┐
                       ▼                    ▼                    ▼
                ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
                │ Prometheus   │     │  Loki (logs)  │     │  MySQL/Other │
                │ (metrics)    │     │               │     │              │
                └─────────────┘     └──────────────┘     └──────────────┘

Key facts: - Grafana itself stores its own metadata (users, dashboards if not provisioned, alert rules, data source configs) in an internal database — SQLite by default, or Postgres/MySQL for production/HA setups. - Grafana does not collect or store your metrics/logs — it queries whatever backend you point it at, live, at render/alert-evaluation time (with some caching). - The Grafana Agent (or Prometheus, Promtail, etc.) is a separate concern — those are the collectors that actually gather and ship data into your data sources. Grafana only visualizes/alerts on what's already stored there.

NOTE:

This is a classic Podman gotcha — unlike Docker, Podman doesn't assume Docker Hub by default for "short names" like grafana/grafana-oss.

Option A (recommended, no system config needed): Fully qualify every image name with the registry (docker.io/...). This also makes your compose file portable to any machine regardless of that machine's registry config.

Option B (system-wide fix): Add to /etc/containers/registries.conf:

unqualified-search-registries = ["docker.io"]

Then restart the Podman machine if on macOS/Windows (podman machine stop && podman machine start), or just retry on Linux.

Option A in the guide since it's more portable — updating every image reference in the file now.

Every image reference in the file is now fully qualified with docker.io/ (Grafana, Prometheus, Node Exporter, Loki, Promtail), and I added a callout right at the first podman run example explaining exactly why that error happens and why fully-qualifying is the more portable fix versus editing registries.conf. That error should be gone now — same file, updated in place.

3. Installation

Podman (fastest way to get started)

podman run -d \
  --name=grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  grafana/grafana-oss:latest

Visit http://localhost:3000 — default login is admin / admin (you'll be forced to change it on first login).

Rootless note: Podman runs containers rootless by default (a real security advantage over Docker's daemon model). Named volumes like grafana-storage above just work. Bind-mounting a host path (e.g. ./file.yml:/etc/file.yml) is different — on SELinux-enforcing systems (Fedora, RHEL, CentOS) the container won't be able to read/write it unless you add :Z (or :z for shared mounts) to the mount, e.g. -v ./prometheus.yml:/etc/prometheus.yml:Z. Ubuntu/Debian without SELinux can ignore this, but it's safe to include either way — the lab below adds it throughout.

Podman Compose (recommended — persists config, easy to extend)

Podman doesn't ship Compose support natively pre-v4; install the podman-compose tool (works with the same Compose YAML syntax Docker uses):

pip install podman-compose --break-system-packages
# or, on Fedora/RHEL: sudo dnf install podman-compose

Podman 4.x+ also supports podman compose directly if you have docker-compose or podman-compose installed as the backend — either invocation below works; this guide uses podman-compose explicitly since it has no extra dependency.

# podman-compose.yml  (same syntax as docker-compose.yml)
version: "3.8"
services:
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme123
    restart: unless-stopped

volumes:
  grafana-storage:

podman-compose up -d

Production tip: for a systemd-managed, daemonless setup (Podman's signature advantage), generate a systemd unit instead of relying on Compose long-term: podman generate systemd --new --files --name grafana, or write a native Quadlet .container file (Podman 4.4+) — search "Podman Quadlet" in the docs for the current syntax, since it's evolved across recent releases.

Linux (Debian/Ubuntu — APT)

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server

Linux (RHEL/CentOS/Fedora — YUM/DNF)

sudo tee /etc/yum.repos.d/grafana.repo << 'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

sudo dnf install -y grafana
sudo systemctl enable --now grafana-server

Windows

Download the .msi installer from https://grafana.com/grafana/download.
Run it — installs Grafana as a Windows service.
Access at http://localhost:3000.
macOS (Homebrew)

brew update
brew install grafana
brew services start grafana

Grafana Cloud (fully managed, no install)

Sign up at grafana.com — gives you hosted Grafana plus hosted Prometheus (Mimir), Loki, and Tempo on a free tier, useful if you don't want to run infrastructure yourself while learning.

After logging in (admin/admin by default, forced password change), the main areas you'll use constantly:

Home — landing page, recently viewed dashboards.
Dashboards (left sidebar) — browse, search, organize into folders.
Explore — an ad-hoc query interface, great for testing queries before building a panel around them.
Alerting — alert rules, contact points, notification policies, silences.
Connections → Data sources — where you configure Prometheus, Loki, etc.
Administration — users, teams, organizations, server-wide settings. Tip: Use Explore constantly while learning a new data source's query language — it's faster to iterate there than inside a panel editor.

5. Data Sources

A data source is a configured connection to a backend. You can add these via the UI (Connections → Data sources → Add data source) or via provisioning (see Section 13).

Example: Adding Prometheus via UI

Connections → Data sources → Add data source → Prometheus.
Set URL: http://prometheus:9090 (or http://localhost:9090 if running locally outside Docker).
Click Save & Test — Grafana pings the data source and confirms it can query it.
Example: Adding Prometheus via provisioning YAML

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

editable: false prevents UI changes from drifting away from your provisioned config — a common production practice (config lives in Git, not clicked together).

Common Data Sources You'll Encounter

Data Source	Use Case	Query Language
Prometheus	Metrics (the most common pairing with Grafana)	PromQL
Loki	Logs	LogQL
Tempo	Distributed traces	TraceQL
InfluxDB	Time-series metrics (alternative to Prometheus)	InfluxQL / Flux
MySQL/PostgreSQL	Relational business data, custom app metrics	SQL
Elasticsearch	Logs, full-text search data	Lucene / DSL
CloudWatch	AWS native metrics	CloudWatch Metrics Insights
Graphite	Legacy metrics systems	Graphite query syntax

6. Dashboards & Panels

A Dashboard is a collection of Panels arranged on a grid.
A Panel is one visualization tied to one or more queries.
Rows group panels visually and can be collapsed — useful for long dashboards.
Folders organize dashboards (e.g. "Production", "Team X", "Personal").
Creating Your First Dashboard

Dashboards → New → New Dashboard.
Add visualization → select your data source.
Write a query (e.g., in PromQL: up) → a panel renders immediately.
Set the panel title, choose a visualization type (right sidebar).
Save the dashboard (top right) — give it a name and a folder.
Panel JSON Model (good to understand early)

Every panel is ultimately just JSON under the hood — this is what lets dashboards be version-controlled and provisioned as code:

{
  "type": "timeseries",
  "title": "CPU Usage",
  "targets": [
    { "expr": "rate(node_cpu_seconds_total[5m])", "refId": "A" }
  ],
  "fieldConfig": {
    "defaults": { "unit": "percentunit" }
  }
}

You can view/edit this directly via Panel menu → Edit → JSON view, or export a whole dashboard's JSON via Dashboard settings → JSON Model.

7. Panel Types Explained

Panel Type	Best For	Example Use
Time series	Metrics over time (the default/most common)	CPU/memory usage graphed over the last 24h
Stat	A single current number, big and bold	"Current active users: 1,204"
Gauge	A value against a min/max range with thresholds	Disk usage % with red zone above 90%
Bar chart	Categorical comparisons	Requests per endpoint
Bar gauge	Multiple gauges in a compact list	Per-server CPU at a glance
Table	Raw tabular data, sortable/filterable	List of top slow SQL queries
Heatmap	Distribution over time (e.g. latency buckets)	Request latency histogram over time
Logs	Raw log lines from Loki/Elasticsearch	Live-tailing application logs
Node graph	Service dependency/topology visualization	Microservice call graph
Geomap	Geographic data	Requests by country
Pie chart	Simple proportional breakdown	Traffic share by browser
Text	Static markdown/HTML content	Dashboard instructions, links

Rule of thumb: Time series for trends, Stat/Gauge for "is this number OK right now," Table for anything you need to sort/search, Logs panel specifically for log data (don't try to force logs into a table).

8. Query Languages

PromQL (Prometheus) — the one you'll use most

# Current value of a metric
up

# Rate of increase over 5 minutes (for counters)
rate(http_requests_total[5m])

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Sum grouped by a label
sum by (status_code) (rate(http_requests_total[5m]))

# 95th percentile latency from a histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Filter by label
node_cpu_seconds_total{mode="idle", instance="web-01:9100"}

# Alert-style comparison
up == 0

Key PromQL concepts: - Counters only go up (e.g. total requests) — always wrap in rate() or increase() for meaningful graphs, never graph a raw counter directly. - Gauges can go up or down (e.g. memory usage) — graph directly, no rate(). - Histograms let you compute percentiles via histogram_quantile().

LogQL (Loki)

# All logs from a specific job
{job="varlogs"}

# Filter by text content
{job="varlogs"} |= "error"

# Filter out noise
{job="varlogs"} != "healthcheck"

# Parse and filter JSON logs
{job="app"} | json | level="error"

# Count log lines per minute (turns logs into a metric)
count_over_time({job="varlogs"} |= "error" [1m])

SQL (MySQL/PostgreSQL data sources)

SELECT
  $__timeGroup(created_at, '5m') as time,
  COUNT(*) as signups
FROM users
WHERE $__timeFilter(created_at)
GROUP BY 1
ORDER BY 1

$__timeFilter and $__timeGroup are Grafana macros that automatically bind to the dashboard's selected time range — essential for any SQL data source panel.

9. Variables & Templating

Variables let one dashboard serve many contexts (hosts, environments, namespaces) without duplication.

Creating a Variable

Dashboard settings → Variables → Add variable: - Name: instance - Type: Query - Data source: Prometheus - Query: label_values(up, instance) Now $instance is usable in any panel query:

up{instance="$instance"}

And a dropdown appears at the top of the dashboard letting viewers switch instances without editing anything.

Multi-value & "All" support

Enable Multi-value and Include All option on a variable to let users select several instances (or all) at once:

up{instance=~"$instance"}

Note =~ (regex match) instead of = — required for multi-value variables since Grafana joins selections with | internally (e.g. web-01|web-02).

Chained Variables

Variables can depend on each other — e.g. a datacenter variable filters which values appear in a subsequent instance variable:

Query for $instance: label_values(up{datacenter="$datacenter"}, instance)

Repeating Panels/Rows

Set a panel (or row) to Repeat by a variable — Grafana automatically clones that panel once per selected value, e.g. one CPU graph per selected server, without manually building each one.

10. Transformations

Transformations reshape query results after they come back from the data source, before rendering — useful when the raw query shape doesn't match what the panel needs.

Common transformations: - Merge — combine results from multiple queries into one table. - Rename by regex — clean up ugly auto-generated series names. - Filter by name/value — hide series matching a pattern. - Add field from calculation — compute a new column (e.g. used / total * 100). - Organize fields — reorder/hide/rename table columns. - Group by — aggregate rows, similar to SQL GROUP BY. Access via Panel editor → Transform tab. A common pattern: use a transformation to convert raw bytes into a percentage, instead of writing a more complex query when the data source doesn't support that math natively.

11. Alerting (Unified Alerting)

Modern Grafana (v9+) uses Unified Alerting — alert rules live in Grafana itself (or can be Prometheus-native rules that Grafana visualizes), evaluated on a schedule, routed through notification policies to contact points.

Creating an Alert Rule

Alerting → Alert rules → New alert rule.
Define the query (e.g. up == 0).
Set Condition: fire when the query result is above/below a threshold.
Set Evaluation: how often to check, and "for" duration (avoid flapping — e.g. only fire if up == 0 persists for 5 minutes straight).
Add labels (e.g. severity: critical) — used by notification policies to route.
Contact Points

Where notifications actually go: Slack, email, PagerDuty, Opsgenie, webhook, Microsoft Teams, etc. Configure under Alerting → Contact points.

Notification Policies

Routing rules — e.g. "anything labeled severity: critical goes to PagerDuty AND Slack; everything else just goes to a low-priority Slack channel." Policies form a tree, so you can nest more specific routes under broader defaults.

Example Alert Rule (as provisioned YAML)

apiVersion: 1
groups:
  - orgId: 1
    name: infra-alerts
    folder: Infrastructure
    interval: 1m
    rules:
      - uid: host-down
        title: Host Down
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: up
              instant: true
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.instance }} has been down for 5 minutes"

Silences

Temporarily mute alerts (e.g. during planned maintenance) without deleting the rule — Alerting → Silences → New silence, matched by label.

12. Annotations

Annotations mark events on a graph's timeline — deployments, incidents, config changes — so you can visually correlate "the error rate spiked right after this deploy."

Manual: Ctrl/Cmd+click on a graph to add one directly.
Query-based: configure an annotation query against a data source (e.g. a Loki query for {job="deploy-bot"}) so deploy events auto-appear on every dashboard.
API-driven: CI/CD pipelines can POST an annotation to Grafana's API right after a deployment completes — a very common real-world pattern.

curl -X POST http://admin:admin@localhost:3000/api/annotations \
  -H "Content-Type: application/json" \
  -d '{"text":"Deployed v2.3.1","tags":["deploy"]}'

13. Provisioning as Code

Instead of clicking dashboards together in the UI (which drifts and isn't version-controlled), provision everything from files.

Directory Structure

/etc/grafana/provisioning/
├── datasources/
│   └── prometheus.yml
├── dashboards/
│   └── dashboards.yml        # points Grafana at a folder of dashboard JSON
└── alerting/
    └── rules.yml

Dashboard Provisioning Config

# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: "default"
    orgId: 1
    folder: "Infrastructure"
    type: file
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Drop exported dashboard JSON files into /var/lib/grafana/dashboards — Grafana loads them automatically on startup and keeps them in sync.

Grafana as Code — Higher-Level Tools

Terraform (grafana/grafana provider) — manage dashboards, data sources, alert rules, and folders as Terraform resources, fitting into an existing IaC pipeline.
Grafonnet — a Jsonnet library for generating dashboard JSON programmatically, popular for large dashboard fleets with lots of repeated
structure across teams/services.

14. Plugins

Grafana's functionality extends via plugins: - Panel plugins — new visualization types (e.g. flowcharts, clocks, custom business visualizations). - Data source plugins — connect to backends not supported out of the box. - App plugins — bundle multiple panels/data sources into a cohesive mini-application inside Grafana.

grafana-cli plugins install grafana-piechart-panel
sudo systemctl restart grafana-server

Browse the official plugin catalog at grafana.com/grafana/plugins.

15. Users, Teams, Orgs & RBAC

Organization — a fully isolated Grafana workspace (separate dashboards, data sources, users) — most self-hosted setups use just one.
Team — a group of users within an org, used to assign permissions in bulk (e.g. "the SRE team can edit all Infrastructure dashboards").
Roles: Viewer, Editor, Admin (org-level) — plus fine-grained RBAC in newer/Enterprise Grafana for permission scopes narrower than these three broad roles.
Folder permissions — restrict who can view/edit dashboards within a specific folder, independent of their org-wide role.
Example: Restricting a Folder to One Team

Folder settings → Permissions → Add permission → select team → set role (View/Edit/Admin) — then remove the broader "Everyone" default permission so only that team (plus org Admins) can access it.

16. Security Best Practices

Change the default admin password immediately — never leave admin/admin.
Disable anonymous access unless explicitly needed (GF_AUTH_ANONYMOUS_ENABLED=false, the default).
Use SSO (OAuth, SAML, LDAP) for real teams instead of individually managed local accounts.
Restrict data source permissions — not every viewer needs query access to every data source; use data source permissions to scope this.
Never expose Grafana directly to the internet without a reverse proxy + TLS — terminate HTTPS in front of it (nginx, Caddy, or a cloud load balancer), don't rely on Grafana's own HTTP server directly facing the internet.
Rotate API keys / service account tokens regularly, and scope them to the minimum needed role.
Use editable: false on provisioned resources so UI changes can't
silently diverge from your version-controlled config.

17. Performance & Scaling Notes

Grafana itself is lightweight; the bottleneck is almost always the data source (a huge, unoptimized PromQL query against millions of series will be slow no matter how good Grafana's rendering is).
For high availability, run multiple Grafana instances behind a load balancer, backed by a shared Postgres/MySQL database (not the default SQLite, which doesn't support multiple concurrent instances safely) and a shared alerting configuration.
Query caching (Grafana Enterprise, or a caching proxy in front of your data source) reduces repeated load for popular dashboards.
Keep dashboard panel counts reasonable — a dashboard with 60 panels all
querying on load will feel sluggish; consider splitting into linked dashboards.

18. The LGTM Stack

Loki (logs) + Grafana (visualization) + Tempo (traces) + Mimir or Prometheus (metrics) — a commonly deployed, fully open-source observability stack where Grafana is the single pane of glass across all three signal types. This matters because you can pivot directly from a metric spike, to the logs from that exact time window, to a trace of the specific slow request — all without leaving Grafana.

19. Common Mistakes & Best Practices

Graphing raw counters without rate() — a raw counter graph is just a line going up forever and tells you nothing useful; always rate/increase it.
Too many panels per dashboard — hurts both load time and readability; favor several focused dashboards linked together over one giant one.
Not using variables — duplicating a dashboard per server/environment instead of templating it is the single most common early mistake.
Alerting on raw instantaneous values without a "for" duration — causes alert flapping on brief, harmless blips; almost always pair a threshold with a sustained-duration requirement.
Building dashboards only in the UI — no version control, no review process, no easy rollback; provision important dashboards as code once they're stabilized.
Ignoring units — a panel showing "3600" is meaningless without knowing it's seconds; always set the correct unit in field config (Grafana will then
auto-format as "1h").

20. Hands-On Lab — Build a Full Monitoring Stack From Scratch

This lab builds: Prometheus (metrics storage) + Node Exporter (system metrics collector) + Grafana (visualization), fully containerized, then walks through building a real dashboard with variables and an alert.

Step 1 — Project Setup

mkdir grafana-lab && cd grafana-lab
mkdir -p prometheus grafana/provisioning/datasources grafana/provisioning/dashboards grafana/dashboards

Step 2 — Prometheus Config

# prometheus/prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["node_exporter:9100"]

Step 3 — Grafana Provisioning

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

# grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: "lab-dashboards"
    orgId: 1
    folder: "Lab"
    type: file
    options:
      path: /var/lib/grafana/dashboards

Step 4 — Podman Compose File

The :Z suffix on every bind mount below tells Podman to relabel the host path for container access under SELinux (Fedora/RHEL) — harmless to include on Ubuntu/Debian too, so it's safe to leave in either way.

# podman-compose.yml
version: "3.8"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:Z
    ports:
      - "9090:9090"

  node_exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"

  grafana:
    image: grafana/grafana-oss:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=lab123
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:Z
      - ./grafana/dashboards:/var/lib/grafana/dashboards:Z
    depends_on:
      - prometheus

node_exporter on Podman, host metrics: the Node Exporter image is normally run with host mounts (/proc, /sys) for real host metrics on bare metal. This lab keeps it simple (container-only metrics) to focus on the Grafana workflow — see the image's docs if you want it reporting your actual host's CPU/memory instead of the container's.

Step 5 — Launch Everything

podman-compose up -d
podman-compose ps   # confirm all three containers are running

If a container fails to start, check rootless port binding — ports below 1024 need extra privilege under rootless Podman, but everything in this lab (3000, 9090, 9100) is safely above that threshold, so no special config is needed. - Prometheus UI: http://localhost:9090 — check Status → Targets, both prometheus and node_exporter should show as UP. - Grafana UI: http://localhost:3000 — log in as admin / lab123.

Step 6 — Confirm the Data Source Auto-Loaded

Connections → Data sources — Prometheus should already be there (provisioned, not manually added) and marked as default.

Step 7 — Build Your First Panel

Dashboards → New → New Dashboard → Add visualization → Prometheus.
Query: rate(node_cpu_seconds_total{mode="idle"}[1m])
Title it "CPU Idle Rate", set visualization to Time series.
Under Standard options → Unit, set to Percent (0.0-1.0).
Step 8 — Add a Memory Usage Panel

New panel, query:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Title: "Memory Usage %". Visualization: Gauge. Set thresholds: green below 70, yellow 70-90, red above 90 (Field → Thresholds).

Step 9 — Add a Variable for Multi-Instance Support

Dashboard settings → Variables → New variable: - Name: instance - Type: Query, Data source: Prometheus - Query: label_values(up, instance) - Enable Multi-value and Include All Update your panel queries to use it:

rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[1m])

(With only one node_exporter running, you'll see one value in the dropdown — the pattern is what matters; this scales to any number of hosts unchanged.)

Step 10 — Save & Export the Dashboard as Code

Save the dashboard (name it "Lab Overview", folder "Lab"). Then: Dashboard settings → JSON Model → copy the JSON → save it as grafana/dashboards/lab-overview.json. Restart the stack (podman-compose restart grafana) and confirm the dashboard reloads automatically from that file — you've now provisioned it as code.

Step 11 — Create an Alert Rule

Alerting → Alert rules → New alert rule.
Query: up{job="node_exporter"}.
Condition: IS BELOW 1 (fires when the exporter goes down).
Evaluation: every 1m, for 2m.
Add label severity: warning.
Save — under Contact points, the default is usually a placeholder; for this lab, add an email or webhook contact point pointed at a test endpoint (e.g. https://webhook.site for a quick, no-signup test URL) to see a real notification fire.
Step 12 — Trigger and Observe the Alert

podman-compose stop node_exporter

Wait ~2-3 minutes, then check Alerting → Alert rules — your rule should transition to Firing, and (if configured) your contact point should receive a notification. Bring it back:

podman-compose start node_exporter

Confirm the alert resolves back to Normal.

Step 13 — Explore Logs (Optional Extension)

Add Loki + Promtail to the compose file to extend this lab into full LGTM-style observability:

  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log:Z
      - ./promtail-config.yml:/etc/promtail/config.yml:Z
    command: -config.file=/etc/promtail/config.yml

Add Loki as a second provisioned data source the same way as Prometheus, then build a Logs panel querying {job="varlogs"}.

Lab Cleanup

podman-compose down -v   # -v also removes the Prometheus/Grafana data volumes

21. Quiz — Test Your Understanding

Why should you always wrap a Prometheus counter metric in rate() before graphing it?
What's the difference between a Grafana variable with single-value vs multi-value enabled, in terms of the operator you use in your query (= vs =~)?
What does setting editable: false on a provisioned data source accomplish?
Why does an alert rule typically need a "for" duration in addition to a threshold condition?
What's the difference between what Grafana stores itself versus what a data source like Prometheus stores?
In the LGTM stack, which component is responsible for logs, and which for traces?
Why is SQLite (Grafana's default internal database) unsuitable for a multi-instance HA Grafana deployment?
Answer Key
Counters only ever increase — a raw graph just shows an ever-climbing line that reveals nothing about the actual rate of events; rate() converts it into "how fast is this increasing," which is the actually useful signal.
Single-value uses = (exact match); multi-value uses =~ (regex match), since Grafana joins multiple selections with | internally.
Prevents anyone from modifying that data source through the UI, so it can't drift from the version-controlled provisioning file — the file is always the single source of truth.
Without a sustained-duration requirement, brief, harmless blips above/below a threshold would trigger alerts (flapping) instead of only firing for genuinely persistent problems.
Grafana stores its own metadata (users, dashboard definitions if not provisioned, alert rules, org/team config) in its internal database; Prometheus stores the actual time-series metric data itself — Grafana queries it live rather than storing a copy.
Loki handles logs; Tempo handles traces.
SQLite doesn't safely support multiple processes writing concurrently, which multiple Grafana instances behind a load balancer would need to do — a shared Postgres/MySQL database is required for that setup.

Table of Contents

21. Quiz — Test Your Understanding

1. What Is Grafana & Why Use It

and version-controlled, not just clicked together in a UI.

2. Architecture Overview

NOTE:

3. Installation

Podman (fastest way to get started)

Podman Compose (recommended — persists config, easy to extend)

Linux (Debian/Ubuntu — APT)

Linux (RHEL/CentOS/Fedora — YUM/DNF)

Windows

macOS (Homebrew)

Grafana Cloud (fully managed, no install)

4. First Login & Navigation

5. Data Sources

Example: Adding Prometheus via UI

Example: Adding Prometheus via provisioning YAML

Common Data Sources You'll Encounter

6. Dashboards & Panels

Creating Your First Dashboard

Panel JSON Model (good to understand early)

7. Panel Types Explained

8. Query Languages

PromQL (Prometheus) — the one you'll use most

LogQL (Loki)

SQL (MySQL/PostgreSQL data sources)

9. Variables & Templating

Creating a Variable

Multi-value & "All" support

Chained Variables

Repeating Panels/Rows

10. Transformations

11. Alerting (Unified Alerting)

Creating an Alert Rule

Contact Points

Notification Policies

Example Alert Rule (as provisioned YAML)

Silences

12. Annotations

13. Provisioning as Code

Directory Structure

Dashboard Provisioning Config

Grafana as Code — Higher-Level Tools

structure across teams/services.

14. Plugins

15. Users, Teams, Orgs & RBAC

Example: Restricting a Folder to One Team

16. Security Best Practices

silently diverge from your version-controlled config.

17. Performance & Scaling Notes

querying on load will feel sluggish; consider splitting into linked dashboards.

18. The LGTM Stack

19. Common Mistakes & Best Practices

auto-format as "1h").

20. Hands-On Lab — Build a Full Monitoring Stack From Scratch

Step 1 — Project Setup

Step 2 — Prometheus Config

Step 3 — Grafana Provisioning

Step 4 — Podman Compose File

Step 5 — Launch Everything

Step 6 — Confirm the Data Source Auto-Loaded

Step 7 — Build Your First Panel

Step 8 — Add a Memory Usage Panel

Step 9 — Add a Variable for Multi-Instance Support

Step 10 — Save & Export the Dashboard as Code

Step 11 — Create an Alert Rule

Step 12 — Trigger and Observe the Alert

Step 13 — Explore Logs (Optional Extension)

Lab Cleanup

21. Quiz — Test Your Understanding

Filters