Exploring AI

Running Gemma 4 Locally with Ollama on Ubuntu 24.04

Running Gemma 4 Locally with Ollama on Ubuntu 24.04 Server (8 GB RAM, No GPU)

1. Overview

Ollama is a lightweight runtime that packages, downloads, and serves LLMs (using quantized GGUF weights) with a simple CLI and local REST API.
Gemma 4 is Google DeepMind's latest open model family. It ships in four sizes: e2b, e4b, 26b, 31b (the "E" stands for "effective" parameters — these are efficiency-tuned for edge/low-resource devices).
On an 8 GB RAM box with no GPU, you want gemma4:e2b (~7.2 GB download). e4b (~9.6 GB) will likely swap heavily or fail to load comfortably; the 26B/31B variants are out of reach without much more RAM.

2. Prerequisites

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl

Check free RAM and disk space (you need ~10-15 GB free disk for the model + overhead):

free -h
df -h /

3. Add Swap (strongly recommended on 8 GB RAM)

Even the e2b model can spike memory usage during context processing. A swap file prevents an OOM kill.

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
free -h

4. Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

Verify:

ollama --version

Ollama installs as a systemd service. Check it's running:

sudo systemctl status ollama

If it's not running:

sudo systemctl enable --now ollama

5. (Optional) Bind Ollama to Your Network

By default Ollama only listens on localhost:11434. If you want to reach it from other machines on your LAN:

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Save, then reload:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Open the port if you have ufw enabled:

sudo ufw allow 11434/tcp
sudo ufw allow 3000/tcp

ollama --help
Large language model runner

Usage:
  ollama [flags]
  ollama [command]

Available Commands:
  serve        Start Ollama
  create       Create a model
  show         Show information for a model
  run          Run a model
  stop         Stop a running model
  pull         Pull a model from a registry
  push         Push a model to a registry
  signin       Sign in to ollama.com
  signout      Sign out from ollama.com
  list         List models
  ps           List running models
  cp           Copy a model
  rm           Remove a model
  launch       Launch the Ollama menu or an integration
  help         Help about any command

Flags:
  -h, --help         help for ollama
      --nowordwrap   Don't wrap words to the next line automatically
      --verbose      Show timings for response
  -v, --version      Show version information

Use "ollama [command] --help" for more information about a command.

Ollama supported llms

https://ollama.com/library

6. Pull Gemma 4 (E2B — best fit for 8 GB RAM, no GPU)

ollama pull gemma4:e2b

## I am trying (gemma4:e4b) # lets see.

ollama pull gemma4:e4b

This downloads roughly 7.2 GB. It will take a while on a modest connection — let it finish before running.

7. Run It

ollama run gemma4:e2b

or
ollama run gemma4:e4b

This drops you into an interactive chat prompt. Type a message and press Enter. Exit with /bye or Ctrl+D.

Expect it to be slow — on CPU-only hardware with 8 GB RAM, figure roughly 1-3 tokens/second. It's usable for short Q&A and scripting, not snappy chat.

8. Test via the REST API

Once the model has been pulled, you can call it without the interactive shell:

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e2b",
  "messages": [{"role": "user", "content": "Explain swap memory in one sentence."}],
  "stream": false
}'

9. Useful Management Commands

ollama list          # see downloaded models
ollama ps            # see what's currently loaded in memory
ollama rm gemma4:e2b # remove a model to free disk space
ollama stop gemma4:e2b  # unload from memory without deleting

10. Tuning for Low RAM

Reduce context window if responses are slow to start — pass a smaller num_ctx in your API calls (default is 128K, which is overkill and memory-hungry for casual use):

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e2b",
  "messages": [{"role": "user", "content": "Hi"}],
  "options": {"num_ctx": 4096},
  "stream": false
}'

curl http://localhost:11434/api/chat -d '{
  "model": "gemma4:e4b",
  "messages": [{"role": "user", "content": "Hi"}],
  "options": {"num_ctx": 4096},
  "stream": false
}'

{"model":"gemma4:e4b","created_at":"2026-07-28T13:59:59.141842642Z","message":{"role":"assistant","content":"Hello! How can I help you today? 😊","thinking":"Thinking Process:\n\n1.  **Analyze the input:** The input is \"Hi\". It is a simple, friendly greeting.\n2.  **Determine the intent:** The user wants to initiate a conversation or simply acknowledge my presence in a friendly manner.\n3.  **Formulate the response strategy:** The response should be equally friendly, welcoming, and prompt further interaction (i.e., ask how I can help).\n4.  **Draft potential responses:**\n    *   *Option A (Simple):* Hello!\n    *   *Option B (Engaging):* Hi there! How can I help you today?\n    *   *Option C (Very enthusiastic):* Hey! What's up? 😊\n5.  **Select the best option:** Option B is usually the most effective as it mirrors the warmth of the greeting while immediately establishing my function and availability to assist.\n\n6.  **Final response generation:** \"Hello! How can I help you today?\" (Adding a slight variation in tone/emojis helps, but keeping it professional yet warm is key.)"},"done":true,"done_reason":"stop","total_duration":41568115216,"load_duration":431568367,"prompt_eval_count":17,"prompt_eval_duration":733207000,"eval_count":241,"eval_duration":40370967000}

Close other services while running inference — on an 8 GB box, every GB matters.
Don't run e4b or larger unless you upgrade RAM; they will thrash swap and feel unusable on spinning or even SATA SSD storage.
If you need something noticeably faster on this hardware, consider gemma3:1b or gemma3:4b as a lighter fallback — smaller, text-only, and much quicker on CPU, at the cost of capability.

11. Quick Reference Table

Tag	Approx. Size	Fits 8 GB RAM (no GPU)?	Notes
`gemma4:e2b`	7.2 GB	Yes (with swap)	Best default pick for this box
`gemma4:e4b`	9.6 GB	Marginal / not recommended	Will swap heavily
`gemma4:26b`	~18 GB	No	Needs much more RAM
`gemma4:31b`	~20 GB	No	Needs much more RAM

12. Uninstall (if ever needed)

sudo systemctl stop ollama
sudo systemctl disable ollama
sudo rm -rf /usr/share/ollama /usr/local/bin/ollama

from Gemini:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"
Environment="OLLAMA_NUM_PARALLEL=1"

Install and Run Open WebUI (Docker)

sudo docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui \
  --restart always \
  ghcr.io/open-webui/open-webui:main

sudo docker ps

Access the Web Interface

Open your web browser and navigate to:
http://<SERVER-IP>:3000

Create your admin account on the initial setup screen (this is stored locally on your server).

Select gemma4:4b from the top model dropdown menu and start chatting.

Tuning Context & Performance for CPU

To avoid high CPU usage and memory consumption when handling long chat histories:

In Open WebUI, go to Settings → Admin Settings → Models.

Click the edit icon next to your 4B model.

Under Advanced Parameters, set:

Context Length (num_ctx): 4096

Threads (num_thread): 4

Management Commands

ollama list           # List downloaded models
ollama ps             # View currently loaded model in RAM
ollama stop <model>   # Unload model from RAM
ollama rm <model>     # Delete a model to free disk space

Open WebUI Container

sudo docker logs -f open-webui   # View web UI logs
sudo docker restart open-webui   # Restart Web UI
sudo docker stop open-webui      # Stop Web UI

In Gemma 4, thinking mode is controlled directly via a specialized system prompt token: <|think|>. When this token is present at the start of the system prompt, the model generates an internal chain-of-thought enclosed within <|channel>thought ... <channel|> before outputting its final response.</p>

Method 1: Globally via Open WebUI Model Settings

If you want gemma4:e2b or gemma4:e4b to always use thinking mode by default in Open WebUI:

Go to Workspace → Models (or Admin Settings → Models) and click the Edit (pencil) icon next to your Gemma 4 model.

Set System Prompt with Thinking Token

In the System Prompt input box, add the <|think|> token at the beginning:

<|think|> You are a helpful AI assistant.

3.Set Thinking Budget (Optional):

On lower-spec hardware (like 4 CPU cores), uncapped reasoning can slow down response times. You can cap the thinking budget by adding explicit instructions directly in the system prompt alongside the token:

<|think|> Keep your internal reasoning concise and under 300 words.

4.Save Model Changes:Click Save at the bottom. Future chats using this model will automatically trigger Gemma 4's chain-of-thought engine.

If you only want thinking mode enabled for specific complex prompts (like math or code debugging):

In any active chat window, click the Controls icon (sliders icon near top right).

Expand the System Prompt field.

Prepend <|think|> to your system prompt for that conversation.