Ollama vs llama.cpp vs vLLM: which to run locally

Three tools dominate self-hosted inference, and picking the wrong one is the most common reason people conclude local models are slow.

They optimise for different things. One is built for a person at a laptop, one for a server handling many users, and one is the engine underneath much of the rest.

The short version

	Ollama	llama.cpp	vLLM
Built for	One or a few users	Maximum control	Many concurrent users
Setup	Two commands	Compile and configure	Python, GPU, config
Runs on CPU	Yes	Yes, well	Not the point
Concurrency	Weak	Weak	Its whole reason to exist
Model format	Handles it for you	GGUF, your choice	Standard weights

Serving one person and serving two hundred are different engineering problems. The tool that wins at one loses badly at the other.

Ollama: start here

Ollama wraps model download, quantisation format and a local HTTP server behind two commands. You install it, name a model, and you have a working endpoint.

The endpoint is OpenAI-compatible, which matters more than it sounds. Your application code doesn’t need to know whether it’s talking to a local model or a hosted one, so switching later is a base URL rather than a rewrite.

What you give up is control. Sampling parameters, quantisation choices and memory layout are decided for you, and the defaults are sensible rather than optimal.

It also handles a handful of simultaneous requests poorly. That’s not a defect; it’s the design target, and it’s the right target for most people starting out.

llama.cpp: the layer underneath

llama.cpp does the actual inference in C++ across CPU and GPU, and it’s what several friendlier tools are built on.

Use it directly when you need something the wrapper won’t give you: a specific quantisation level, control over how many layers sit on the GPU, or deployment somewhere Python is inconvenient.

Quantisation is the lever that makes any of this fit. Dettmers and colleagues showed in LLM.int8() that 8-bit inference held up “without any performance degradation” on models up to 175B parameters, and four-bit went considerably further.

It’s also the best option for CPU-only machines, which matters more than the GPU-centric coverage suggests. Plenty of useful workloads run acceptably on a laptop with no discrete graphics.

The quantisation control is the real draw. Frantar and colleagues’ GPTQ work reported 3 and 4-bit quantisation “with negligible accuracy degradation relative to the uncompressed baseline”, and choosing that level yourself is how you fit a model into the memory you actually have.

vLLM: when several people share the machine

The moment you’re serving a team rather than yourself, the constraint changes from does it fit to how many requests can share it.

vLLM was built for exactly that. Kwon and colleagues’ PagedAttention paper reported improving throughput “by 2-4x” over prior systems at the same latency, by managing attention memory the way an operating system manages pages.

The gain comes from not wasting memory reserved for conversations that turned out short, which lets many more requests run in parallel on the same card.

The cost is operational weight. It expects a GPU, a Python environment and configuration, and it’s overkill for a single user by a wide margin.

The number that decides it

Concurrency is the variable, and most people guess it wrong in both directions.

Fifty employees with access to an internal assistant is not fifty concurrent requests. It’s usually two or three at peak, because people think between messages and most of them aren’t using it at all.

An agent workload is the opposite. One user can generate ten simultaneous calls if the agent fans work out to parallel sub-agents, which is how modern coding agents are built.

So count requests in flight rather than seats. That single number tells you which of these three tools you need, and it’s cheap to measure once something is running.

Memory per conversation is the other half. Each active request holds its own context, so ten parallel agents on one card need roughly ten times the working memory of one chat, and that is what saturates a machine long before the weights do.

Choosing, concretely

One user, exploring: Ollama. It’ll take twenty minutes and you’ll learn what local models can actually do before committing to anything.

A small team, occasional use: still Ollama, on a shared machine. The concurrency limits only bite when requests genuinely overlap.

Production traffic, many users: vLLM. The throughput difference is large enough that it changes how much hardware you need to buy.

Unusual hardware or tight memory: llama.cpp, because the control is the point.

What none of them fix

The engine changes throughput, not capability. A 7B model served brilliantly is still a 7B model, and our guide to what fits on your machine covers the memory arithmetic that actually decides quality.

None of them make your documents searchable either. That’s a retrieval problem, and our guide to vector storage covers where it belongs.

And none of them close the gap to a frontier hosted model on hard reasoning. Open weights have narrowed it, with two of the six labs at the top of the 2026 AI Index ratings publishing weights, but narrowed is not eliminated.

A sensible path

Start with Ollama and a 7B or 8B model. Measure it against ten real tasks from your own work rather than against a benchmark table.

Move to vLLM when you can name the specific concurrency limit you hit, and to llama.cpp when you can name the specific parameter you need. Migrating between them is mostly a configuration change, since they all speak the same API shape.

Doing it in that order costs you almost nothing and saves the common failure, which is standing up production serving infrastructure for a workload that turned out to be four requests an hour.

Keep the OpenAI-compatible interface at your call site whichever you choose. It’s the thing that makes all of these decisions reversible, and reversibility is worth more than picking correctly the first time.