Tooling
llama.cpp: Run Open-Weight Models Locally Without APIs
llama.cpp is an open-source inference engine enabling users to run large language models entirely on local hardware—laptops, desktops, or clusters—without API calls, telemetry, or external dependencies. The project supports diverse hardware from Apple Silicon to NVIDIA and AMD GPUs, with optimized kernels for each platform.
llama.cpp is a lightweight, open-source inference runtime that executes large language models on user hardware without cloud dependencies. The project has accumulated over 123,500 stars on GitHub and provides a unified binary that runs across heterogeneous platforms: Apple Silicon (M1–M Ultra), NVIDIA GPUs (RTX 3090, RTX 4090, RTX 5090, A100, H100), AMD Radeon and MI-series cards, Intel Arc, Google TPU variants, NVIDIA Jetson edge devices, and standard CPUs.
Key features include zero-cost local execution (no API keys, no telemetry), retention of models and conversation data on the user's machine, and cross-platform compatibility with hand-tuned GPU and CPU kernels. Installation is available via shell script, package managers (Homebrew, Winget), or from source.
The tool integrates with the Pi coding agent framework: users can run `llama serve` to host a model, install the pi-llama plugin, and launch Pi for automatic local model discovery without configuration overhead. Supported models include Qwen 3.6 (Alibaba's multimodal reasoning models), Gemma 4 and Gemma 3 (Google's open models with 140+ language support), and GPT-OSS (OpenAI's first open-weight models since GPT-2, optimized for reasoning and function calling).
The project targets both individual developers and enterprise clusters seeking private, autonomous AI inference without vendor lock-in.
Readers’ Forum
No contributions yet — open the debate.
Open the discussion
No account or password needed — just enter your e-mail and we’ll send you a one-time sign-in link. First time here? You’re set up automatically.
Your rating will be applied automatically after you sign in.
Check your inbox
We’ve sent a sign-in link to …. Open it on this device — this tab will sign you in automatically.
Nothing arrived? Check your spam folder — and mark the mail as "Not spam" so it lands in your inbox next time.