How to run an LLM locally: what fits on your hardware

Can you run a decent language model on your own machine? Yes, and the main thing standing between you and a working setup is picking a model that fits your memory.

Get that arithmetic right and the rest takes about twenty minutes. Get it wrong and you’ll conclude local models are useless, which is the usual outcome.

The memory arithmetic

A model’s weights have to sit in memory to run at reasonable speed. The number of parameters times the bytes per parameter gives you the floor.

At full 16-bit precision that’s two bytes each, so a 7 billion parameter model wants roughly 14GB before you’ve done anything. Quantisation is what makes this tractable.

Quantisation stores each weight in fewer bits. Four-bit weights cut the memory requirement by roughly four times, and the quality loss is smaller than the number suggests.

Dettmers and colleagues showed the principle holds at scale in LLM.int8(), reporting inference at 8-bit “without any performance degradation” for models up to 175B parameters.

Four-bit methods went further. Frantar and colleagues’ GPTQ reported quantising 175B-parameter models to 3 or 4 bits per weight “with negligible accuracy degradation relative to the uncompressed baseline”, which is the technique behind most files you’ll download.

Whether you can run an LLM locally on your machine

Memory available	Realistic model	What it’s good for
8GB	3B to 7B at 4-bit	Summarising, drafting, simple extraction
16GB	7B to 13B at 4-bit	General assistant work, light coding
32GB	~30B at 4-bit	Noticeably better reasoning
64GB+	70B at 4-bit	Approaches hosted mid-tier quality

Leave headroom above the model file. Context uses memory too, and it grows with conversation length, so a model that exactly fills your card will fail partway through a long chat.

Splitting a model between GPU and system RAM works but collapses in speed. If most layers don’t fit on the card, pick a smaller model rather than accepting a few tokens per second.

Getting started: run an LLM locally

The path of least resistance is Ollama, which wraps model download, quantisation format and a local server behind two commands. Install it, run a model by name, and you have a working setup plus an HTTP endpoint your code can call.

Underneath most of these tools sits llama.cpp, which does the actual inference in C++ across CPU and GPU. You can use it directly for more control over quantisation and sampling, at the cost of more setup.

Start with a 7B or 8B model regardless of your hardware. It runs fast enough to iterate on, and you’ll learn what local models can and cannot do before committing to a 40GB download.

Small models got surprisingly good

The reason this is worth doing in 2026 rather than 2023 is that the small end of the range improved faster than the large end.

Mistral reported that its 7B model outperformed Llama 2 13B across all benchmarks tested, which was an early signal that parameter count alone stopped predicting quality.

Open weights have kept arriving since. Meta released a 30B model under Apache 2.0 this year, and a 30B at 4-bit is a realistic download for a well-specified laptop.

What you gain by running an LLM locally

Data never leaves the machine, which resolves a whole category of questions about client documents, medical notes and anything covered by a confidentiality obligation.

Cost becomes fixed. You’ve already bought the hardware, so high-volume repetitive work costs nothing extra, which is a different shape from per-token pricing.

And nothing changes underneath you. A hosted model can be updated or retired; a file on your disk behaves identically next year, which matters if you’ve tuned prompts against its specific quirks.

What you give up

Capability, mostly. A 13B model is meaningfully worse than a frontier hosted model at complex reasoning, long documents and code, and no amount of prompting closes that gap.

Speed varies enormously by hardware. A well-specified GPU gives you a responsive experience; CPU-only inference on a large model can be slower than typing the answer yourself.

Context handling is also weaker at the small end. Stanford’s Lost in the Middle found accuracy “significantly degrades when models must access relevant information in the middle of long contexts”, and smaller models degrade sooner.

Quantisation has a floor

Given how well four-bit works, the obvious question is why not go lower. The answer is that quality falls off a cliff rather than sloping down gently.

Two and three-bit versions exist and they behave oddly. Output stays grammatical, which makes the degradation hard to spot, but instruction-following and arithmetic get noticeably worse.

The practical rule is that a smaller model at four bits beats a larger model squeezed to two. If a 13B at 4-bit doesn’t fit, run a 7B at 4-bit rather than a 13B at 2-bit.

The hybrid setup most people land on

Treating this as a choice between local and hosted is the wrong framing, and almost nobody who does this seriously runs one exclusively.

Route high-volume, low-complexity and privacy-sensitive work to the local model. Send the hard reasoning and long documents to a hosted one. Most workloads split roughly along those lines without much effort.

Because Ollama exposes an HTTP API, swapping between the two is usually a change of base URL rather than a rewrite, so it’s cheap to test which tasks each handles better.

Making it more useful

Adding retrieval helps more than upgrading the model. A local model plus an index of your own documents outperforms a larger model guessing from memory, and our guide to vector search covers the storage side.

Local fine-tuning is also within reach now. Dettmers and colleagues’ QLoRA reported tuning a 65B model “on a single 48GB GPU”, which puts adapters for smaller models comfortably inside consumer hardware.

Start small, measure what it actually gets right on your own work, and scale up only where you can name the thing the smaller model failed at.

Once more than a couple of people rely on it, the surrounding pieces start to matter more than the model, which our guide to the self-hosted stack works through layer by layer.

Adapters are worth understanding before you reach for one, since fine-tuning teaches behaviour rather than facts and the distinction decides whether the effort pays.

One habit worth adopting early: keep a file of ten real tasks you’d want this for, and run each new model against all ten. It takes five minutes and it settles arguments that benchmark tables never will.