# vLLM vs Ollama: which inference engine should you actually run?

The honest answer to "vLLM vs Ollama" depends less on benchmarks than on the hardware sitting on your desk. Here's how to choose — and how to test the claims on your own machine.

Published 2026-11-07 · https://inpherio.co.uk/blog/vllm-vs-ollama-which-inference-engine-should-you-actually-run

Key points:
- Hardware decides the answer more than benchmarks do: without a supported accelerator, vLLM is off the table.
- Ollama is llama.cpp with model management and an API on top — single-user speeds are close; the real gap is concurrency.
- vLLM's edge is aggregate throughput under many simultaneous requests, thanks to continuous batching and PagedAttention.
- Most published comparison numbers don't transfer to your machine — test with a fixed protocol before believing any of them.
- For one user chatting locally, quant choice and context settings matter more than engine choice.

## The short answer

It depends on your hardware — and the hardware decides it more than any benchmark does.

- **Ollama** if you're getting started or it's just you using the model. One command to install, one to pull a model, done.
- **llama.cpp** if you're on a Mac or a CPU-only box, or you want full control over every knob.
- **vLLM** if you have a proper accelerator and you're serving concurrent requests — an app, other people, anything where requests pile up.

One fact settles half the debate before it starts: **Ollama is built on llama.cpp**. It uses llama.cpp as its default backend, so for a single user chatting with one model, speeds on the same quant are close. The genuine difference is vLLM's PagedAttention and continuous batching, which keep a GPU efficient when many requests arrive at once.

On hardware: vLLM is built around datacentre-class accelerators. CUDA is the well-trodden path, with a compute-capability floor of roughly 7.0 — Volta and newer, so no Pascal cards like a GTX 1080 Ti. ROCm support exists for AMD GPUs but is rougher round the edges, and CPU support is experimental rather than something you'd choose. On a Mac it's a non-starter. If you don't have a supported accelerator, the comparison is over before it begins.

| | vLLM | Ollama | llama.cpp |
|---|---|---|---|
| NVIDIA GPU | Yes (CC 7.0+) | Yes | Yes |
| AMD GPU | Yes, via ROCm (rougher) | Partial | Yes (Vulkan/ROCm) |
| Apple Silicon | No | Yes (Metal) | Yes (Metal) |
| CPU only | Experimental only | Yes | Yes |
| GGUF quants | Experimental, not recommended | Yes | Yes |
| Throughput under load | Best | Modest | Modest |
| Ease of setup | Steepest | Easiest | Middle |
| OpenAI-compatible API | Yes | Yes | Yes (`llama-server`) |

## What each engine actually is

**llama.cpp** is a C/C++ inference engine. It runs nearly everywhere — CPU, CUDA, Apple Metal, Vulkan — and it's the home of the GGUF format, with quants ranging from tiny 2-bit files to near-lossless Q8. It's CLI-first: powerful, fast, and covered in flags.

**Ollama** is llama.cpp with model management and an API on top: `ollama pull llama3.1:8b`, Modelfiles for custom configurations, automatic GPU detection, and a REST endpoint that speaks the OpenAI chat format. It's the convenience layer, and it's genuinely convenient.

**vLLM** is a Python server built for throughput. Its headline features — PagedAttention, continuous batching, tensor parallelism across multiple GPUs — are all aimed at one thing: serving many requests at once efficiently. It's OpenAI-compatible out of the box, strong at high concurrency and long contexts, and it carries a heavier install: CUDA, PyTorch, and the version-matching headaches that come with them.

The overlap is worth admitting: for one user chatting with one model, engine choice matters less than the quant you pick and the hardware you run it on. A Q4 model on a 1080 Ti behaves like a Q4 model on a 1080 Ti, whichever server is in front of it.

## Throughput: when vLLM pulls ahead (and when it doesn't)

This is the "is vLLM faster?" part, and the honest answer has two halves.

**Under concurrent load, vLLM wins, and it's not subtle.** Continuous batching means new requests join running ones without waiting for a batch to finish, and PagedAttention manages the KV cache in pages rather than contiguous blocks, so GPU memory isn't wasted on padding. Aggregate tokens per second scale far better as requests pile up. If you're serving an app with a handful of simultaneous users, this is the whole reason vLLM exists.

**Single-stream, one user, one request at a time: the gap is modest.** llama.cpp's quantised CUDA kernels are excellent, and since Ollama sits on llama.cpp, the same model at the same quant on the same GPU produces similar single-request speeds across all three. What moves the needle for a solo user is quantisation choice and prompt length, not the server binary.

There's a counter-trade-off, too: vLLM's own startup and VRAM overhead are higher. On an 11GB card, you may fit a model comfortably in llama.cpp that vLLM can't serve at all, because vLLM reserves memory for its scheduler and pre-allocated KV cache. More throughput machinery costs memory before it earns anything back.

## Hardware: the real decider

**Apple Silicon (M-series).** llama.cpp's Metal backend is the sensible choice, and unified memory lets you load quants that would never fit in an equivalently priced GPU's VRAM. Ollama on top makes it one command. vLLM is a non-starter here — that's not a trade-off, it's a wall.

**Older NVIDIA (Pascal, e.g. GTX 10-series).** vLLM is out: its compute-capability floor rules out pre-Volta cards. llama.cpp's CUDA kernels still work well on a 1080 Ti, and Ollama inherits that. An 11GB Pascal card typically runs 7B–8B models at Q4 respectably — a Q4_K_M 7B file is around 4–5GB.

**AMD Radeon.** ROCm makes vLLM possible on supported cards, but it's the rougher path — fewer tested combinations, more forum threads. llama.cpp's Vulkan backend is often the lower-friction route on consumer AMD hardware.

**Modern NVIDIA, 16GB+.** Now vLLM is viable, and worthwhile if you're serving others. If it's just you, Ollama or llama.cpp will feel identical day to day with far less setup.

**CPU-only box.** llama.cpp is the only real option of the three, and small models — 3B and under at Q4 — are genuinely usable for light tasks. Don't expect miracles from an 8B model on CPU, but don't write the box off either.

If you own hardware that sits idle most of the day, serving models on it is one way to put it to work — more on that at the end.

## Benchmarks you can trust: how to test on your own kit

Most "X vs Y" numbers online don't transfer to your machine, because they vary wildly with quant, context length and batch size. A Q8 comparison tells you nothing about your Q4 setup; a 32k-context test tells you nothing about a 2k chat.

So instead of quoting numbers, here's a protocol you can run in an afternoon:

1. **Same model, same quant, everywhere it's possible.** Pick one model — say [granite4.2:3b](/models/granite4.2:3b) or [deepseek-r1:8b](/models/deepseek-r1:8b) — and pin the exact GGUF file for llama.cpp and Ollama. For vLLM, use the same base model in safetensors and note that the format differs.
2. **Fixed prompt, fixed generation length.** Same prompt text, same `max_tokens`, same sampling settings. Randomness is the enemy of comparison.
3. **Warm run.** Discard the first generation (model load, cache warm-up) and measure the second.
4. **Record tok/s and peak memory.** Generation speed and peak RAM/VRAM — both matter, and VRAM is often the real constraint.
5. **Test concurrency.** Run 1, then 4, then 8 simultaneous requests. This is where the engines genuinely diverge, and where vLLM either justifies itself or doesn't.

The tooling makes this a 20-minute job per engine. llama.cpp ships `llama-bench`, which sweeps prompts and generations across configurations from a single command:

```bash
./llama-bench -m granite4.2-3b-q4_k_m.gguf -p 512 -n 256 -ngl 99
```

vLLM ships its own benchmark scripts (`benchmarks/benchmark_serving.py` in the repo) for load testing against the OpenAI endpoint. For Ollama, any HTTP load generator pointed at its API works fine — it's the same llama.cpp underneath.

(Live on Inpherio: https://inpherio.co.uk/models/glm-ocr:latest)

## Which to pick: three common setups

**A single-GPU desktop, just experimenting.** Start with Ollama — `brew install ollama` or the Windows installer, `ollama pull`, and you're generating within minutes. Move to raw llama.cpp when you want to tune thread counts, context handling, or kernel choices. Trade-off: you'll eventually hit Ollama's defaults and want the control back.

**Mac owner.** Ollama for convenience; llama.cpp directly — or MLX, Apple's own framework, if you want to squeeze more out of Apple Silicon. vLLM is not an option, so the decision is already narrowed for you. Trade-off: MLX's model catalogue is smaller than the GGUF ecosystem.

**Serving an app or multiple users.** vLLM on a proper NVIDIA GPU, accepting the steeper setup and the GPU requirement as the cost of admission. Ollama handles a few parallel requests (see its `OLLAMA_NUM_PARALLEL` setting, plus speculative decoding support), but its throughput under load is far below vLLM's — the gap shows quickly once requests genuinely pile up.

One more consideration: if your reason for running an engine is to serve other people — not just yourself — the Inpherio connector supports all three, so the "which engine" question becomes a hosting question rather than a research project. See [every supported engine](/engines), and if you own a machine that could earn while idle, [the providers page](/providers) explains how hosting works.

## Gotchas we'd warn a friend about

- **Ollama's default context window is small.** Long prompts get silently truncated unless you raise it (`num_ctx` in a Modelfile, or the relevant API parameter). This is the single most common "my model got dumb" report, and it's a settings issue, not a model issue.
- **vLLM install pain is real.** Matching CUDA toolkit, driver and PyTorch versions is where enthusiasm goes to die. Docker (`vllm/vllm-openai` image) is the sane route — it pins the versions for you.
- **Model formats differ.** GGUF everywhere in llama.cpp/Ollama; Hugging Face safetensors in vLLM. The same model can behave slightly differently per format, and quantisation schemes differ too — a "Q4" GGUF and a 4-bit AWQ are not the same thing. vLLM does have an experimental GGUF loader, but it's not the intended path; treat safetensors as the default there.
- **The KV cache eats VRAM fast at long contexts, on all three.** No engine escapes this. Double your context and you can double your memory footprint; on a small card, context length is often the setting that decides whether a model fits at all.
- **llama.cpp's flag sprawl.** The flexibility is the point, but `-ngl`, `-c`, `-t`, `-fa` and friends take a while to internalise. Start with sensible defaults and change one thing at a time.

## What we'd pick

For most people reading this: **Ollama today, llama.cpp when you outgrow it, vLLM only when concurrency is the actual problem.** The engine that's running when you're chatting with one model matters far less than the quant, the context settings and the card underneath it — and no benchmark on someone else's machine changes what happens on yours.

If you'd rather not run any of this yourself, the same models are callable through one API — [deepseek-r1:8b](/models/deepseek-r1:8b) and the rest are live on the [marketplace](/marketplace) now, and the [developer quick start](/developers) shows how to switch over in a few lines.

## FAQ

**Is vLLM faster than Ollama?** Only under concurrent load. With one user and one request at a time, speeds on the same GPU and quant are close; vLLM pulls ahead when many requests arrive simultaneously thanks to continuous batching.
**Can vLLM run on Apple Silicon or a CPU-only box?** Not sensibly. vLLM is built around datacentre-class accelerators: CUDA is the well-trodden path, ROCm exists but is rougher, and CPU support is experimental. On a Mac or CPU-only box, use llama.cpp or Ollama.
**Does Ollama use llama.cpp?** Yes — llama.cpp is Ollama's default backend, so single-request performance on the same quant is very similar. Ollama adds model management, a REST API and automatic hardware detection on top.
**Which engine should I use with a single gaming GPU?** Ollama to start, raw llama.cpp when you want control over threads, context and kernels. vLLM only earns its setup cost if you're serving multiple users at once.
**Do all three expose an OpenAI-compatible API?** Yes — vLLM and Ollama natively, and llama.cpp via its built-in server (`llama-server`), which also speaks the OpenAI chat completions format.
**Can I serve GGUF quants with vLLM?** Not as the intended path. vLLM is built around Hugging Face safetensors and its own quantisation formats (AWQ, GPTQ, FP8); GGUF loading exists experimentally, but GGUF is llama.cpp and Ollama territory.
