All the app used to run LLM locally
Tools used for running llm models
| Tool | Type | Runs On | Best For | Open Source |
|---|---|---|---|---|
| llama.cpp | Inference Engine | CPU/GPU/Apple | Maximum compatibility | ✅ |
| LM Studio | GUI App | Desktop | Easy local ChatGPT | ❌ |
| Jan | GUI App | Desktop | Fully offline chat | ✅ |
| Draw Things | Image Gen | macOS/iOS | Apple-native image gen | ❌ |
| DiffusionBee | Image Gen | macOS | Simple SD on Mac | ✅ |
| JoyFusion | Image Gen | Apple devices | Styled image templates | Freemium |
| vLLM | Serving Engine | Linux GPU | Production API serving | ✅ |
| Ollama | CLI Tool | Desktop | Simple model running | ✅ |
| MLX LM | Framework | Apple Silicon | Native Mac performance | ✅ |
| Docker Model Runner | Container Tool | Docker | Containerized AI | ✅ |
| Lemonade | Local Server | AMD hardware | AMD NPU inference | ✅ |
| SGLang | Serving Engine | Linux GPU | Structured generation | ✅ |
| Unsloth Studio | Fine-tuning | Cloud/Local | Fast model fine-tuning | ✅ |
| Pi | AI Assistant | Web/Mobile | Conversational AI | ❌ |
| Hermes Agent | AI Agent | Self-hosted | Learning agent | ✅ |
| Atomic Chat | GUI + API | All platforms | Cross-platform local AI | ✅ |
| OpenClaw | Agent Framework | Self-hosted | Custom agents | ✅ |
Local AI & LLM Tools Reference
Local LLM Inference Engines
llama.cpp
A C/C++ implementation of LLaMA model inference focused on running large language models locally with minimal resources. It supports quantization (GGUF format), runs on CPU, GPU (CUDA, Metal, Vulkan, ROCm), and even mobile devices. It's the foundational engine behind many local LLM apps.
- Best for: Maximum performance and flexibility on consumer hardware
- Formats: GGUF
- Platforms: macOS, Windows, Linux, iOS, Android
vLLM
A high-throughput, memory-efficient inference and serving engine for LLMs. Built by UC Berkeley's Sky Computing Lab, it uses PagedAttention to dramatically improve GPU utilization for serving multiple users or handling long contexts.
- Best for: Production API serving, multi-user deployments
- Key feature: PagedAttention for efficient KV-cache management
- Platforms: Linux (GPU-focused)
MLX LM
Apple's MLX framework for running language models natively on Apple Silicon (M1/M2/M3/M4). Optimized for the Neural Engine and unified memory architecture, it delivers excellent performance per watt on Macs.
- Best for: Apple Silicon Mac users wanting native performance
- Key feature: Unified memory, Metal GPU acceleration
- Platforms: macOS (Apple Silicon)
SGLang
A structured generation language for efficient LLM serving, developed by LMSYS/UC Berkeley. It enables faster inference through co-designed runtime and compiler, with support for complex prompts, regex constraints, and multi-turn conversations.
- Best for: Structured output generation, API serving with constraints
- Key feature: RadixAttention for caching, structured generation
- Platforms: Linux (GPU)
Docker Model Runner
Docker's official tool for running AI models in containers. It provides a standardized way to package, distribute, and run models using Docker, with an OpenAI-compatible API endpoint.
- Best for: DevOps teams, containerized deployments
- Key feature: Docker-native model packaging
- Platforms: Anywhere Docker runs
Local Image Generation Apps
Draw Things
A native AI image generation app for macOS and iOS that runs Stable Diffusion locally using Core ML. It offers a polished UI with support for ControlNet, inpainting, and various samplers.
- Best for: Apple users wanting a polished local image gen experience
- Key feature: Core ML optimized, runs entirely on-device
- Platforms: macOS, iOS
DiffusionBee
A free, open-source Stable Diffusion GUI app for macOS. One of the earliest and simplest ways to run image generation locally on a Mac without command-line knowledge.
- Best for: Beginners wanting one-click local Stable Diffusion
- Key feature: One-click installer, no dependencies
- Platforms: macOS (Intel & Apple Silicon)
JoyFusion
A native AI drawing app for macOS, iPad, and iPhone that runs Stable Diffusion locally using Core ML. Features 60+ one-click style templates, text-to-image, image-to-image, ControlNet, and high-resolution rendering.
- Best for: Apple users who want styled image generation with templates
- Key feature: 60+ style templates, fully local processing
- Platforms: macOS, iPadOS, iOS
Local LLM GUI Apps
LM Studio
A desktop GUI application for discovering, downloading, and running local LLMs. Features a polished chat interface, model search from Hugging Face, and an OpenAI-compatible local server.
- Best for: Casual users wanting a ChatGPT-like experience with local models
- Key feature: Built-in model browser, one-click download
- Platforms: macOS, Windows, Linux
Jan
An open-source ChatGPT alternative that runs 100% offline on your computer. Supports multiple inference engines (llama.cpp, TensorRT-LLM), has a clean UI, and includes features like thread organization and model management.
- Best for: Privacy-focused users wanting a fully offline ChatGPT replacement
- Key feature: 100% offline, extensible engine support
- Platforms: macOS, Windows, Linux
Atomic Chat
A free, open-source local AI chat app for Mac, Windows, Linux, iOS, and Android. Runs 1,000+ models locally (Llama, Qwen, DeepSeek, Mistral, Gemma, Kimi, etc.) with no account, no cloud, and no subscription. Includes an OpenAI-compatible API server, agent support, and TurboQuant for faster inference.
- Best for: Cross-platform local AI with agent capabilities
- Key feature: OpenAI-compatible API at
localhost:1337, one-click agent launch - Platforms: macOS, Windows, Linux, iOS, Android
Ollama
The most popular local LLM runner with a simple CLI. Download and run models with a single command (ollama run llama3). Includes a REST API, model library, and easy modelfile customization.
- Best for: Developers wanting the simplest CLI experience
- Key feature:
ollama runone-liner, extensive model library - Platforms: macOS, Windows, Linux
Model Serving & Fine-Tuning
Lemonade
A local AI server optimized for AMD hardware (Ryzen AI NPU, Radeon GPUs). Provides an OpenAI-compatible API for running text, image, and speech models locally with support for hybrid NPU+iGPU execution.
- Best for: AMD Ryzen AI PC users wanting NPU-accelerated inference
- Key feature: NPU + iGPU hybrid mode, ROCm/Vulkan backends
- Platforms: Windows, Linux
Unsloth Studio
A fine-tuning platform by Unsloth, known for making LLM fine-tuning 2x faster while using 70% less memory. Provides a no-code/low-code interface for customizing open-weight models.
- Best for: Users wanting to fine-tune models without deep ML expertise
- Key feature: 2x faster fine-tuning, 70% less VRAM
- Platforms: Cloud + local
AI Agents
OpenClaw
An open-source AI agent framework created by Peter Steinberger (Nov 2025). It enables persistent, autonomous AI agents with memory and skills. Kimi Claw is Moonshot's cloud-hosted wrapper around OpenClaw.
- Best for: Building self-hosted autonomous agents
- Key feature: 248k+ GitHub stars, extensive plugin ecosystem
- Platforms: Self-hosted (cloud or local)
Hermes Agent
An open-source, self-improving AI agent built by Nous Research (launched Feb 2026). Features persistent memory, self-evolving skills, contained sub-agents, and 20+ messaging platform integrations. It learns from every task and gets better over time.
- Best for: Users wanting an agent that learns and improves continuously
- Key feature: Self-evolving skills, 140k+ GitHub stars, MIT licensed
- Platforms: Linux, macOS, WSL2 (self-hosted)
AI Assistants
Pi
A personal AI assistant by Inflection AI, designed to be supportive, kind, and conversational. Known for its warm personality and voice conversation capabilities. Available via web, iOS, Android, and WhatsApp.
- Best for: Casual, emotional, supportive AI conversations
- Key feature: Exceptional voice mode, empathetic responses
- Platforms: Web, iOS, Android, WhatsApp
Quick Comparison Matrix
| Tool | Type | Runs On | Best For | Open Source |
|---|---|---|---|---|
| llama.cpp | Inference Engine | CPU/GPU/Apple | Maximum compatibility | Yes |
| LM Studio | GUI App | Desktop | Easy local ChatGPT | No |
| Jan | GUI App | Desktop | Fully offline chat | Yes |
| Draw Things | Image Gen | macOS/iOS | Apple-native image gen | No |
| DiffusionBee | Image Gen | macOS | Simple SD on Mac | Yes |
| JoyFusion | Image Gen | Apple devices | Styled image templates | Freemium |
| vLLM | Serving Engine | Linux GPU | Production API serving | Yes |
| Ollama | CLI Tool | Desktop | Simple model running | Yes |
| MLX LM | Framework | Apple Silicon | Native Mac performance | Yes |
| Docker Model Runner | Container Tool | Docker | Containerized AI | Yes |
| Lemonade | Local Server | AMD hardware | AMD NPU inference | Yes |
| SGLang | Serving Engine | Linux GPU | Structured generation | Yes |
| Unsloth Studio | Fine-tuning | Cloud/Local | Fast model fine-tuning | Yes |
| Pi | AI Assistant | Web/Mobile | Conversational AI | No |
| Hermes Agent | AI Agent | Self-hosted | Learning agent | Yes |
| Atomic Chat | GUI + API | All platforms | Cross-platform local AI | Yes |
| OpenClaw | Agent Framework | Self-hosted | Custom agents | Yes |
Sources
- Atomic Chat: https://atomic.chat/
- Hermes Agent: https://hermes-agent.org/
- JoyFusion: https://8ai.ac/tools/joyfusion/
- Lemonade: https://lemonade-server.ai/
- LM Studio: https://lmstudio.ai/
- Ollama: https://ollama.com/
- Unsloth: https://unsloth.ai/
- vLLM: https://vllm.ai/
- SGLang: https://sglang.ai/
- OpenClaw / Kimi Claw: https://www.kimi.com/help/kimi-claw/overview
Few of best free open LLM
| Model | Parameters | License | Best Use Case | Min RAM (Q4) | Ollama Command |
|---|---|---|---|---|---|
| Phi-4-mini | 3.8B | MIT | Low-resource devices, CPU-only, basic reasoning | ~2.3 GB | ollama run phi4-mini |
| Gemma 3 (4B) | 4B | Gemma Terms | Multimodal input on laptops, efficient chat | ~2.5 GB | ollama run gemma3:4b |
| Qwen3 (8B) | 8B | Apache 2.0 | Balanced coding, multilingual support, general assist | ~4.6 GB | ollama run qwen3:8b |
| Devstral Small | 24B | Apache 2.0 | Agentic coding workflows, multi-file editing | ~16 GB | ollama run devstral |
| Gemma 3 (27B) | 27B | Gemma Terms | Single-GPU multimodal tasks, long context (128K) | ~18 GB | ollama run gemma3:27b |
| Qwen3 (30B) | 30B | Apache 2.0 | Best overall dense model, coding, reasoning | ~18 GB | ollama run qwen3:30b |
| Llama 4 Scout | 17B (MoE) | Community | Massive context (10M tokens), document analysis | ~55 GB | ollama run llama4-scout |
| gpt-oss (20B) | 20B | Apache 2.0 | Private reasoning, adjustable "thinking" mode | ~16 GB | ollama run gpt-oss:20b |
| Laguna XS 2.1 | 33B (MoE) | Apache 2.0 | Advanced agentic coding, long-horizon planning | ~20 GB | ollama run laguna-xs-2.1 |
| DeepSeek-V4 | 236B (MoE) | MIT | Enterprise-grade coding, complex math (Multi-GPU) | 48+ GB | ollama run deepseek-v4 |
| System Resource Tier | Model | Parameters | License | Primary Coding Use Case | Min RAM/VRAM (Q4) | Ollama Command |
|---|---|---|---|---|---|---|
| Ultra-Low (CPU/Laptop) | Qwen2.5-Coder | 1.5B | Apache 2.0 | Basic autocompletion, learning, scripting | ~1.2 GB | ollama run qwen2.5-coder:1.5b |
| Low (8GB RAM) | Phi-4-mini | 3.8B | MIT | Lightweight logic, simple refactoring | ~2.5 GB | ollama run phi4-mini |
| Entry GPU (8-12GB VRAM) | Qwen2.5-Coder | 7B | Apache 2.0 | Strong single-file generation, Python/JS focus | ~4.5 GB | ollama run qwen2.5-coder:7b |
| Mid-Range (16GB VRAM) | Devstral Small | 24B | Apache 2.0 | Agentic workflows, multi-file editing, terminal use | ~14 GB | ollama run devstral |
| Mid-Range (16GB VRAM) | gpt-oss | 20B | Apache 2.0 | Reasoning-heavy tasks, OpenAI-compatible logic | ~14 GB | ollama run gpt-oss:20b |
| High-End (24GB VRAM) | Qwen3-Coder | 30B (MoE) | Apache 2.0 | Best overall local coder, 256K context, fast inference | ~19 GB | ollama run qwen3-coder:30b |
| High-End (24GB VRAM) | Laguna XS 2.1 | 33B (MoE) | Apache 2.0 | Specialized code execution, complex planning | ~20 GB | ollama run laguna-xs-2.1 |
| Enthusiast (48GB+ VRAM) | Llama 4 Scout | 109B (MoE) | Community | Massive repo analysis (10M context), multimodal | ~55 GB | ollama run llama4-scout |
| Enterprise (Multi-GPU) | DeepSeek-V4 | 236B (MoE) | MIT | SOTA complex reasoning, math, full-stack engineering | 140+ GB | ollama run deepseek-v4 |
Most downloaded open llm models
| Model Name | Parameter Count | Disk Size | RAM / VRAM Footprint | Primary Use Case / Application | Ollama Pull Command / Availability |
|---|---|---|---|---|---|
sentence-transformers/all-MiniLM-L6-v2 |
~22.7M | ~90 MB | ~0.25 GB | Fast, lightweight dense vector embeddings for semantic search, clustering, and duplicate detection. | ollama pull all-minilm |
cross-encoder/ms-marco-MiniLM-L-6-v2 |
~22.7M | ~90 MB | ~0.25 GB | High-speed re-ranking of search results/retrieved chunks in RAG pipelines. | ollama pull hf.co/cross-encoder/ms-marco-MiniLM-L-6-v2 |
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 |
~118M | ~470 MB | ~0.5 - 1.0 GB | Multilingual semantic search and text similarity across 50+ languages. | ollama pull paraphrase-multilingual |
sentence-transformers/all-mpnet-base-v2 |
~109M | ~440 MB | ~0.5 - 1.0 GB | High-accuracy English text embeddings for production vector databases. | ollama pull hf.co/sentence-transformers/all-mpnet-base-v2 |
intfloat/e5-large-v2 |
~335M | ~1.34 GB | ~1.5 - 2.0 GB | High-precision dense document retrieval and semantic matching. | ollama pull hf.co/intfloat/e5-large-v2 |
amazon/chronos-2 |
~710M | ~2.80 GB | ~3.0 - 4.0 GB | Universal zero-shot time-series forecasting (univariate, multivariate, and covariates). | Not supported in Ollama (Time-Series / PyTorch) |
timm/mobilenetv3_small_100.lamb_in1k |
~2.5M | ~10 MB | < 0.1 GB | Lightweight real-time image classification on mobile, IoT, and edge devices. | Not supported in Ollama (Vision / PyTorch) |
intfloat/multilingual-e5-small |
~118M | ~470 MB | ~0.5 - 1.0 GB | Fast cross-lingual vector search and document retrieval in memory-constrained environments. | ollama pull hf.co/intfloat/multilingual-e5-small |
openai-community/gpt2 |
~124M | ~500 MB | ~0.5 - 1.0 GB | Small autoregressive text generation for testing pipelines and language modeling experiments. | ollama pull hf.co/openai-community/gpt2 |
Qwen/Qwen2.5-1.5B-Instruct |
~1.54B | ~3.10 GB | ~1.0 - 3.5 GB | On-device conversational AI, lightweight code generation, and structured reasoning. | ollama pull qwen2.5:1.5b |
autogluon/chronos-2 |
~710M | ~2.80 GB | ~3.0 - 4.0 GB | AutoGluon integration for Chronos-2 time-series forecasting pipelines. | Not supported in Ollama (Time-Series) |
cross-encoder/ms-marco-MiniLM-L-12-v2 |
~33M | ~130 MB | ~0.3 - 0.5 GB | Higher-accuracy search re-ranking stage for RAG applications. | ollama pull hf.co/cross-encoder/ms-marco-MiniLM-L-12-v2 |
sentence-transformers/paraphrase-multilingual-mpnet-base-v2 |
~278M | ~1.10 GB | ~1.2 - 1.5 GB | Top-tier cross-lingual sentence embeddings and semantic matching. | ollama pull hf.co/sentence-transformers/paraphrase-multilingual-mpnet-base-v2 |
meta-llama/Llama-3.2-1B-Instruct |
~1.23B | ~2.50 GB | ~0.8 - 2.5 GB | Fast, local instruction-following, summarization, and agentic workflows on low-resource hardware. | ollama pull llama3.2:1b |
openai/clip-vit-large-patch14 |
~428M | ~1.71 GB | ~2.0 - 2.5 GB | Multimodal image-to-text / text-to-image search and zero-shot image classification. | Not supported natively in Ollama (Vision-Language) |
google-bert/bert-base-uncased |
~110M | ~440 MB | ~0.5 - 1.0 GB | Fine-tuning for text classification, named entity recognition (NER), and sentiment analysis. | ollama pull hf.co/google-bert/bert-base-uncased |
BAAI/bge-small-en-v1.5 |
~33M | ~130 MB | ~0.3 - 0.5 GB | Ultra-fast, highly resource-efficient English vector embeddings for RAG systems. | ollama pull bge-small |
Qwen/Qwen2-0.5B |
~490M | ~1.00 GB | ~0.5 - 1.2 GB | Ultra-small base LLM for embedded devices, auto-completion, and quick text processing. | ollama pull qwen2:0.5b |
google-t5/t5-small |
~60M | ~240 MB | ~0.3 - 0.5 GB | Sequence-to-sequence tasks such as text summarization, translation, and question answering. | ollama pull hf.co/google-t5/t5-small |
FacebookAI/xlm-roberta-base |
~278M | ~1.10 GB | ~1.2 - 1.5 GB | Multilingual classification, NER, and token-level tasks across 100 languages. | ollama pull hf.co/FacebookAI/xlm-roberta-base |
BAAI/bge-reranker-v2-m3 |
~568M | ~2.30 GB | ~2.5 - 3.0 GB | Multilingual candidate re-ranking for enterprise search engines. | ollama pull bge-m3 |
txi-internal-testing/tiny-Qwen2ForCausalLM-2.5 |
~2.4M | < 10 MB | < 0.1 GB | Unit testing and CI/CD validation stub model for Qwen2 architecture. | Internal testing stub (Not available on Ollama) |
nomic-ai/nomic-embed-text-v1.5 |
~137M | ~550 MB | ~0.6 - 1.0 GB | Long-context (up to 8,192 tokens) document embeddings for large-text retrieval. | ollama pull nomic-embed-text |
autogluon/chronos-bolt-small |
~46M | ~180 MB | ~0.3 - 0.5 GB | High-speed, CPU-friendly zero-shot time-series forecasting (energy, financial, demand planning). | Not supported in Ollama (Time-Series) |
BAAI/bge-large-en-v1.5 |
~335M | ~1.34 GB | ~1.5 - 2.0 GB | Enterprise-grade English text embeddings for demanding semantic search benchmarks. | ollama pull bge-large |
FacebookAI/roberta-large |
~355M | ~1.42 GB | ~1.5 - 2.0 GB | Heavyweight text classification, intent recognition, and natural language inference (NLI). | ollama pull hf.co/FacebookAI/roberta-large |
FacebookAI/roberta-base |
~125M | ~500 MB | ~0.5 - 1.0 GB | Industry standard encoder for classification, sentiment analysis, and feature extraction. | ollama pull hf.co/FacebookAI/roberta-base |
Qwen/Qwen2-Embedding-0.5B |
~490M | ~1.00 GB | ~1.2 - 1.5 GB | Generative LLM-based vector embedding extraction for semantic retrieval. | ollama pull hf.co/Qwen/Qwen2-Embedding-0.5B |
distilbert/distilbert-base-uncased |
~66M | ~260 MB | ~0.3 - 0.5 GB | Fast, lightweight text classification microservices with low latency requirements. | ollama pull hf.co/distilbert/distilbert-base-uncased |
BAAI/bge-base-en-v1.5 |
~109M | ~440 MB | ~0.5 - 1.0 GB | Balanced English embedding model offering an optimal trade-off between latency and accuracy. | ollama pull bge-base |