Run an LLM on an old laptop with 512MB–4GB of RAM: the honest guide

The smolmaxxing repo and its Hacker News discussion (475 points) blew up last week because "will it run on a 2009 ThinkPad?" never stops being asked. Here's the practical answer: models, quants, flags and realistic speeds.

Listen to this post · 14 min

Accessibility: the whole post is read aloud — Space plays or pauses, ← → skip 10 seconds, and you can change the speed. It keeps playing with your screen locked.

Share this post

In short

  • 512MB–1GB: smollm2:360m or qwen2.5:0.5b at Q4, context capped at 1–2K — anything more won't fit.

The question never dies: "will it run on a 2009 ThinkPad?" Last week it got its most systematic answer yet — the smolmaxxing repo, whose Hacker News discussion pulled 475 points and 180 comments on 12 November. It's a community effort to make LLMs work on small and old devices, and it confirms what anyone who's tried this knows: yes, you can run a useful model on very little RAM, but the margins are unforgiving and most guides skip the part that actually kills you.

This is the guide we wish existed when we first tried it: which models fit at each RAM tier, which quant survives, which llama.cpp flags matter, and what speed to honestly expect.

The short answer: what runs on 512MB–4GB

The rule of thumb before anything else: take the model file size, add 20–30% for the KV cache and runtime overhead, and leave the OS at least 500MB. A "1GB model" is not a 1GB machine; it's a 1.3GB machine that also needs to run a desktop or a Pi's system processes.

With that in mind, here's what actually fits:

ModelQuantFile sizeMin realistic RAMGood forHonest weakness
smollm2:360mQ4_K_M~270MB512MB–1GBShort drafting, classificationShallow; invents facts readily
qwen2.5:0.5bQ4_K_M~400MB1GBDrafting, routing, structured short tasksWeakest reasoning of the Qwen family
Llama 3.2 1BQ4_K_M~800MB1–2GBFirst model that feels like an assistantLimited world knowledge
falcon3:1bQ4_K_M~800MB1–2GBSimilar tier to Llama 3.2 1BLess community tooling
Qwen2.5-1.5BQ4_K_M~1GB2GBBetter instruction-following at 2GBSlower than the 1Bs, obviously
Llama 3.2 3BQ4_K_M~2GB3–4GBThe tier where it stops feeling like a toyNeeds the full 4GB tier
gemma-3-1bQ4_K_M~800MB1–2GBSolid small chat modelContext handling tighter than the size suggests
granite3.3:2bQ4_K_M~1.6GB3GBEnterprise-ish tone, tool useHeavier than the 1Bs for the quality gained
Qwen3-1.7BQ4_K_M~1.1GB2–3GBBest quality-per-MB in this rangeThinking tokens eat your tiny context

The tiers in one breath:

  • 512MB–1GB: smollm2:360m and qwen2.5:0.5b are the realistic picks. Anything larger won't fit once you account for the OS and the KV cache.
  • 1–2GB: Llama 3.2 1B and falcon3:1b at Q4; 2GB gets you into Qwen2.5-1.5B territory.
  • 2–4GB: Llama 3.2 3B, gemma-3-1b, granite3.3:2b, Qwen3-1.7B — the tier where output starts feeling like a real assistant rather than a toy.

If you're at the 2–4GB tier and want models designed for edge hardware with long context, Liquid AI's LFM3 family (all 2.4GB or under) is worth a look — we covered what each LFM3 model does and needs when it landed.

For anything below these floors — the true 2009 ThinkPad tier — the smolmaxxing repo is the current community reference, with device-by-device notes and configurations people have actually run.

Quantisation: which GGUF level survives tiny RAM

Q4_K_M is the standard quality/size compromise, and on anything with 2GB or more it should be your default. On 1–2GB machines, though, quantisation stops being a quality dial and becomes the difference between running and not running: Q3_K_M or even Q2_K is how you shave a model down to fit.

The trade-off is real and worth stating plainly: below Q4, small models degrade noticeably. A 0.5B at Q2 is often worse than nothing — it has so little headroom that aggressive quantisation pushes it into incoherence. The better move is almost always a smaller model at Q4 or Q5 rather than a bigger one crushed to Q2. A 0.5B at Q4 beats a 1.5B at Q2 on both quality and speed on this hardware.

Q8 buys quality but roughly doubles file size versus Q4 — only worth it if you have genuine headroom, which on this hardware you usually don't.

How to check before committing: the file size on the model's Hugging Face GGUF page is your RAM budget (minus the 20–30% overhead). Don't reason from parameter counts — reason from bytes.

One general habit worth keeping whatever version you run: smoke-test unusual quants (Q2, Q3, the exotic ones) with a short prompt before you rely on them. Small models at aggressive quants are exactly where format edge cases show up first — and as we wrote in our llama.cpp v1.0 piece, the version bump's quant-format continuity is a promise worth verifying, not assuming.

Context length: the hidden RAM cost

Here's the part most guides skip: the KV cache, not the model file, is what usually OOMs a tiny machine.

The KV cache stores the attention state for every token in your context, and it scales with context length × layers × heads. On a 512MB machine, even 2K tokens of context can be too much once the model is loaded. Double the context, roughly double the cache.

Practical caps:

  • Sub-1GB machines: 1–2K context. That's it.
  • 2GB: 4K.
  • 4GB: 8K.

And llama.cpp's default context — 4096, per the project's documentation — will happily OOM a Pi or an ancient laptop if you don't override it. Set it deliberately with -c.

Two tricks that help when supported:

  1. Flash attention (-fa) reduces memory pressure during computation on supported builds.
  2. KV cache quantisation — -ctk q8_0 -ctv q8_0 — roughly halves cache RAM for a small quality cost. On a 512MB machine this is the difference between 1K and 2K of usable context.

Set expectations honestly: these machines are for short prompts and short answers — summarising a paragraph, drafting a sentence, classifying text. RAG over a filing cabinet is the wrong job for this hardware; the prompt processing alone will break you (more below).

The llama.cpp flags that actually matter

Get llama.cpp from the releases or build from source. On a Raspberry Pi, be honest with yourself: an ARM build from source takes a while on the device itself. The release binaries are fine for most people. Our full llama.cpp engine guide covers the rest of the flag reference.

The command for a 1GB-class machine:

llama-server -m qwen2.5-0.5b-q4_k_m.gguf -c 2048 -t 2 --mlock -ctk q8_0 -ctv q8_0

What each flag does:

  • -m — the model file. Non-negotiable.
  • -c 2048 — context size. The single most important flag on low RAM; the default will OOM you.
  • -t 2 — threads. Match physical cores, not logical ones. Hyperthreading hurts on old CPUs; two physical cores beat four logical.
  • --mlock — pins the model in RAM so the OS can't page it out. Costs you the ability to do anything else with that RAM, but stops mid-generation thrash.
  • -ctk q8_0 -ctv q8_0 — KV cache quantisation, halving cache RAM as above.

On --mmap versus --no-mmap: mmap (the default) lets the OS load model pages lazily, keeping RAM free — but on a swap-backed machine it can thrash badly once generation starts. If you have just enough RAM, --mlock is the safer choice; if you're genuinely tight, mmap at least lets the model start.

Swap and zram: zram (compressed swap in RAM) is the Pi owner's friend — it's fast enough to absorb pressure spikes. Disk swap, on the other hand, makes generation crawl: once the model pages out, you're measuring tokens per minute, not per second. If you see constant disk activity during generation, you're over budget.

The Ollama route, if you'd rather not compile: Ollama is llama.cpp underneath (we covered why in vLLM vs Ollama), so the same maths applies — see the Ollama engine guide for the full picture. Two things to know:

OLLAMA_CONTEXT_LENGTH=2048 ollama serve

Or set num_ctx in a Modelfile:

FROM qwen2.5:0.5b
PARAMETER num_ctx 2048

Ollama's default context length has historically been generous for tiny machines — check what your version defaults to before assuming it fits. And to pre-empt the engine question: vLLM is off the table here. Per vLLM's documentation, its practical hardware floor is a CUDA GPU of roughly compute capability 7.0, with CPU support still experimental — old laptops are llama.cpp or Ollama territory, full stop.

Honest tokens/sec: what to expect

These are expectations by hardware class, drawn from community reports and the smolmaxxing repo's device notes — not measurements of ours, and your machine will vary:

  • 2009-era Core 2 laptop: a 0.5B at Q4 runs in the low single digits tok/s. Usable for batch jobs; painful interactively.
  • Raspberry Pi 5: roughly 5–10 tok/s on a 0.5B at Q4. Genuinely usable.
  • 2015-era ThinkPad (Broadwell/Skylake): a 1B model at Q4 is comfortably readable — this is the tier where local stops being a stunt.

For calibration, a common rule of thumb: a few tokens per second is roughly reading pace, so anything above that feels responsive in a chat. Below it, treat the machine as a batch worker, not a conversation partner.

Prompt processing is the real pain. Old CPUs process prompts far slower than they generate tokens — a long prompt can take minutes before the first output token appears. This is normal physics, not a bug. Keep prompts short.

What these machines are genuinely good for:

  • Text classification and routing
  • Short drafting and rewriting
  • Guard models — llama-guard3:1b checking another model's output is a real, sensible use
  • Home automation glue — "did the user mean on or off"

What they're not good for: coding agents, long-context RAG, anything with vision. Don't fight it.

When the old laptop isn't enough: the cheap way out

If 2 tok/s makes you give up, the same tiny models cost almost nothing hosted: qwen2.5:0.5b and smollm2:360m are listed from £0.009 per 1M output tokens — a full novel's worth of output for pennies. There's a free tier for trying without commitment.

The counterpoint is worth stating: the point of smolmaxxing is often privacy or fun, not cost. If the data must stay on the machine, the API isn't the answer — and no price makes it one.

Troubleshooting: the four ways this fails

1. OOM on load. Context too large or quant too big. Fix in this order: drop -c, drop the quant level, drop the model size. Don't skip steps — the first one is usually enough.

2. Runs but crawls. Swap thrash. Open htop and watch swap during generation — if it's growing, you're over budget. Add zram, or accept a smaller model. Disk swap is not a RAM substitute for inference.

3. Gibberish or repeated text. Usually a Q2 quant on a tiny model — go back to Q4 with a smaller model. But check the chat template first: a wrong template produces identical symptoms and is the more common cause. Fix the template before blaming the model.

4. First token takes forever. Prompt processing on old CPUs. Shorten prompts. This is normal, not a bug.

For genuine edge cases — the 2009 ThinkPad tier — the smolmaxxing issue tracker is the community spot for device-specific configurations.

What we'd pick

  • 512MB–1GB: qwen2.5:0.5b at Q4_K_M, -c 1024, zram enabled. smollm2:360m if even that's tight.
  • 2GB: Llama 3.2 1B at Q4_K_M, -c 4096. The first configuration we'd actually recommend to someone.
  • 4GB: Qwen3-1.7B or Llama 3.2 3B at Q4_K_M, -c 8192. This is a real assistant for short tasks.

The tiny-model tier is the unglamorous end of the local AI world — no trending models, no benchmarks to argue about, just bytes and RAM. That's exactly why it's worth getting right.

Questions people ask

Can I run an LLM on a laptop with 2GB of RAM?

Yes. Llama 3.2 1B or falcon3:1b at Q4 fits with a 2K–4K context, leaving room for the OS. Expect readable but not fast output on a 2012-era machine.

Models in this post — live on Inpherio

Have a GPU, a Mac or a spare PC?

Let it earn while you sleep.

Run open models for people who need them. You set the price and keep everything but the 15% fee.

Become a host

Building with AI?

Every open model, one API.

OpenAI-compatible. Priced by the people who run the hardware.

Start free See the marketplace

Read “Run an LLM on an old laptop with 512MB–4GB of RAM: the honest guide” at https://inpherio.co.uk/blog/run-an-llm-on-an-old-laptop-with-512mb-4gb-of-ram-the-honest-guide