# How to install vLLM: pip, Docker, or WSL2 — and a `vllm serve` that works

vLLM is the engine you pick when you need throughput, but the install is where most people give up. Here's the whole path: pip vs Docker, the Windows reality, a working `vllm serve` command, and a five-minute benchmark to prove it.

Published 2026-10-17 · https://inpherio.co.uk/blog/how-to-install-vllm-pip-docker-or-wsl2-and-a-vllm-serve-that-works

Key points:
- vLLM's practical floor is a CUDA GPU of compute capability 7.0+ (Volta and newer); ROCm exists but is rougher, CPU is experimental, and there's no supported native Windows install.
- Three viable paths: pip in a dedicated venv (fastest), Docker with the NVIDIA Container Toolkit (most reproducible), or WSL2 on Windows (driver on Windows, everything else inside).
- One command serves an OpenAI-compatible API: `vllm serve <model>` on port 8000 — the flags worth knowing are --gpu-memory-utilization, --max-model-len and --quantization.
- Verify with a single-request tokens/s measurement, then a concurrent load test — concurrency is where vLLM earns its keep over llama.cpp-based engines.
- If your hardware or use case doesn't fit, Ollama or LM Studio is the honest answer — and hosted open models behind an OpenAI-compatible API cover the rest.

Search for "how to install vLLM" and you'll mostly find stale documentation pages, scattered GitHub issues and tutorials that stop at `pip install vllm` without ever getting you to a working server. This is the walkthrough that closes the gap: the three realistic paths, the Windows answer nobody wants to hear, a `vllm serve` command that actually works, and a quick benchmark to prove the install did something.

If you're still deciding *whether* vLLM is the right engine at all, our [vLLM vs Ollama comparison](/blog/vllm-vs-ollama-which-inference-engine-should-you-actually-run) covers that. This post is the 'how'.

## What you need before you start

Don't spend an afternoon on this before checking the floor:

- **GPU:** vLLM's practical floor is an NVIDIA CUDA GPU of roughly compute capability 7.0 — Volta and newer. Pascal cards like the GTX 10-series are out.
- **AMD:** ROCm support exists but is rougher round the edges. It works; expect more friction.
- **CPU:** experimental, and not what vLLM is for. Use llama.cpp-based engines for that.
- **OS:** Linux is the first-class platform. On Windows, plan for WSL2 or Docker — there is no native Windows path worth recommending.
- **VRAM:** budget model weights in the dtype you load, plus KV-cache headroom. A 7–8B model in FP16 wants roughly 16GB. On 12GB, use an AWQ or GPTQ quantised build, or cut the context length.

If your hardware doesn't clear that bar, stop here. [Ollama or LM Studio](/blog/lm-studio-vs-ollama-which-one-should-you-actually-install) will serve you better and cost you ten minutes instead of an afternoon.

## Three ways to install: pip, Docker, or WSL2 — which one?

| Path | Setup time | Reproducibility | Upgrade pain | Best for |
|---|---|---|---|---|
| pip (Linux) | Quickest — a venv and one install | Low–medium | High — vLLM pins heavy CUDA-adjacent deps | Trying vLLM once on a Linux box |
| Docker | Moderate — Docker plus the container toolkit | High | Low — pull a new image | Running multiple engines, long-term server |
| WSL2 + pip (Windows) | Slowest — WSL2 setup first | Low–medium | High, plus WSL quirks | Windows users with an NVIDIA GPU |

A note on WSL2: it's not really a third install method — it's the *Windows prerequisite*. Once WSL2 is running Ubuntu, you install vLLM inside it with pip or Docker, exactly as on Linux.

**Our call:** if you'll run more than one engine or model stack, Docker wins — the official image ships with matching CUDA libraries so there's no driver/toolkit juggling. If you just want to try vLLM once, pip in a fresh venv is quicker. More detail on the engine itself is on [our vLLM page](/engines/vllm).

## Install with pip (Linux or WSL2)

Prerequisites: an up-to-date NVIDIA driver, and Python. You do **not** need a full CUDA toolkit install — the pip wheel bundles its own CUDA runtime, so only the driver matters.

```bash
python -m venv ~/venvs/vllm
source ~/venvs/vllm/bin/activate
pip install vllm
vllm --version
```

Two things to expect:

- **The download is big.** The wheel is multi-GB because it bundles CUDA libraries. This is normal.
- **Compile errors on unusual setups.** If pip fights you over flash-attention or kernel compilation, don't debug it — switch to the Docker image, which is the officially tested path.

One rule worth following strictly: **keep vLLM in its own environment.** vLLM pins specific versions of PyTorch and other CUDA-adjacent dependencies, and installing it into a shared environment is the classic pip failure mode with packages like this — a reliable way to break everything else that lives in it. If you later upgrade vLLM in an env where torch versions have drifted, expect breakage.

## Install with Docker

Docker is the most reproducible path because the official image ships with CUDA libraries matched to the vLLM build inside it.

**1. Install Docker and the NVIDIA Container Toolkit**, then verify the GPU is visible:

```bash
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi
```

If that prints your GPU details, you're set.

**2. Run vLLM:**

```bash
docker run --gpus all \
  -p 8000:8000 \
  --shm-size 8g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-7B-Instruct-v0.3
```

Two flags worth being explicit about:

- `--shm-size 8g` — vLLM uses shared memory for inter-process tensor transfer; the Docker default (64MB) is too small and causes cryptic crashes.
- The volume mount — without it, every container restart re-downloads tens of GB of model weights. Mount the Hugging Face cache and it downloads once.

**When Docker is the wrong answer:** quick experiments on a machine you already use for other Python work. Spinning up a container to test one model is more ceremony than pip in a venv.

## Windows: the WSL2 reality

Here's the honest answer to "how to install vLLM on Windows": **there is no supported native Windows install.** The working route is WSL2, and it's genuinely fine once set up.

1. **Enable WSL2 and install Ubuntu** — from an admin PowerShell: `wsl --install` (this installs Ubuntu by default), then reboot.
2. **Install the NVIDIA driver on Windows — not inside WSL.** This trips people up. The Windows driver is shared with WSL2 automatically; installing a driver inside the distro breaks things. Just the standard Game Ready or Studio driver from NVIDIA's site.
3. **Inside Ubuntu**, verify with `nvidia-smi` — it should work out of the box — then follow the pip section above.

Known frictions:

- **VRAM is shared with the Windows desktop.** Your browser and compositor take a couple of GB before vLLM starts. Budget for it: on a 12GB card, treat it as roughly 10GB usable.
- **File I/O across `/mnt/c` is slow.** Keep models and caches inside the WSL2 filesystem (`~/`), not on the Windows mount, or downloads and loads crawl.

If WSL2 feels like too much scaffolding for what you want, that's a legitimate reason to run [Ollama or LM Studio on Windows instead](/blog/lm-studio-vs-ollama-which-one-should-you-actually-install) — same models, far less ceremony, at the cost of vLLM's concurrency edge.

## Serving your first model: `vllm serve` that actually works

Once installed, one command gets you a server:

```bash
vllm serve mistralai/Mistral-7B-Instruct-v0.3 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192
```

What the flags do:

- `--gpu-memory-utilization` — the fraction of VRAM vLLM may use for weights plus KV cache. The default is 0.9. On a dedicated GPU box that's right; on a machine that also drives a display, drop to 0.8 or lower.
- `--max-model-len` — maximum context length. vLLM sizes the KV cache for the model's full advertised context (often 32K–128K), which can blow small-VRAM cards out of the water. Setting it to 8192 or 4096 is the single most effective fix on 12–16GB cards.
- `--dtype` — `auto` uses the checkpoint's dtype (usually BF16/FP16). `--dtype half` if you're on older hardware without BF16.
- `--quantization awq` (or `gptq`) — for pre-quantised builds, which is how you fit a 7–8B model comfortably on 12GB.

You now have an **OpenAI-compatible API on port 8000**: `/v1/chat/completions`, `/v1/completions` and `/v1/models`.

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [{"role": "user", "content": "Explain PagedAttention in two sentences."}]
  }'
```

Any OpenAI SDK works against it by changing `base_url` to `http://localhost:8000/v1` — the same swap you'd make to call open models through [a hosted OpenAI-compatible API](/developers).

**Don't panic at the first start.** It's slow: weight download (unless cached), memory profiling, CUDA graph capture. Several minutes of silence is normal. Subsequent starts with cached weights are much faster.

## Verify it's actually fast: a 5-minute benchmark

An install that runs is not an install that's *fast*. Five minutes of checking:

**1. Sanity checks.** `curl http://localhost:8000/v1/models` should list your model. Fire one short completion and confirm it returns. Watch `nvidia-smi` — VRAM use should roughly match your budget (weights + cache), and if it's spilling into system memory something's misconfigured.

**2. Single-request tokens/s.** Send one streaming request, count the tokens, divide by time. This is your baseline number — and the one to compare against Ollama on the *same model and quant*. As we covered in the [vLLM vs Ollama piece](/blog/vllm-vs-ollama-which-inference-engine-should-you-actually-run), single-request speeds between the two are close, because Ollama's default backend is llama.cpp and both engines are doing similar work per token.

**3. Concurrency — the actual point of vLLM.** Fire 8–16 simultaneous requests (a simple `asyncio` loop, or a load tool) and watch aggregate throughput. Continuous batching means concurrent requests share GPU batches instead of queueing one at a time, so total tokens/s should hold up far better than a llama.cpp-based server under the same load. This is where vLLM earns its keep.

**If it's slow:** drop `--max-model-len`, raise `--gpu-memory-utilization` (on a headless box), and check you're not spilling to system RAM.

One habit worth keeping: run the same fixed protocol on both engines before believing anyone's comparison numbers — including ours. Published benchmarks don't transfer to your machine.

## When you shouldn't bother (and what to do instead)

Skip vLLM entirely if:

- **You have no CUDA or ROCm GPU** — llama.cpp-based engines run on CPU, Apple Silicon and everything else.
- **You're a single user chatting** — vLLM's concurrency advantage buys you nothing; Ollama or LM Studio is less ceremony for the same experience. If you'd rather not run anything at all, [mistral:7b](/models/mistral:7b) and [qwen3:8b](/models/qwen3:8b) are both callable hosted for pennies per million tokens.
- **You're on a Mac** — there's no Metal path worth using; MLX and llama.cpp are the right tools there.

And skip the self-hosting ceremony itself if you just need to *call* open models from code occasionally: Inpherio's [marketplace](/marketplace) runs 95 open models behind one OpenAI-compatible API — the same `base_url` swap you made for vLLM, no GPU babysitting.



If you do have the hardware and it sits idle between your own experiments, the same server you just built can earn from other people's requests — hosts keep everything except a 15% platform fee, and new hosts get 12 months fee-free.



Either way, you should now own two things: a working vLLM server on port 8000, and a measured tokens/s number you can trust. That number — not anyone's benchmark chart — is what tells you whether vLLM was worth the install.

## FAQ

**Can I install vLLM on Windows without WSL2?** No. There's no supported native Windows install; the working route is WSL2 with the NVIDIA driver installed on the Windows side, then pip or Docker inside the distro. | Does vLLM work on AMD GPUs? | Yes, via ROCm, but it's rougher round the edges than the CUDA path — expect more setup friction and fewer prebuilt options. | How much VRAM do I need to run vLLM? | Budget model weights in the dtype you load plus KV-cache headroom: a 7–8B model in FP16 wants roughly 16GB; on 12GB use an AWQ/GPTQ quantised build or cut --max-model-len. | Why is `vllm serve` so slow to start the first time? | First start downloads weights, profiles memory and captures CUDA graphs. Subsequent starts with cached weights are much quicker. | pip vs Docker for vLLM — which should I use? | pip in a fresh venv if you just want to try it once; Docker if you'll run multiple engines or want reproducible upgrades. | How do I check vLLM is actually running faster than Ollama? | Run the same model and quant through both with a fixed protocol: one single-request tokens/s measurement, then 8–16 concurrent requests and compare aggregate throughput. | Can I run vLLM on a Mac? | Not really — there's no Metal path worth using. On Apple Silicon, llama.cpp-based engines (Ollama, LM Studio) or MLX are the right tools.
