GraphicCardPrices

How much VRAM does a local LLM need?

Illustration of a desk at night with a chat app on the monitor beside a glass-sided PC and its lit graphics card
Illustration

By GraphicCardPrices. Published .

There is no single VRAM number that guarantees a local language model will fit. The exact model file, quantization, context length, runtime and other GPU use all matter.

Start with the weights, then leave room for the KV cache and runtime. Use the figures below as estimates, and test the exact configuration you plan to run.

Start with model weights

A useful lower-bound estimate is parameter count × bits per weight ÷ 8. At ideal 4-bit weights, that gives:

Weight-only arithmetic estimates. GiB uses 1,073,741,824 bytes.
Model sizeIdeal 4-bit weightsWhat this excludes
8B3.73 GiBQuantization metadata, context cache, runtime and applications
14B6.52 GiBQuantization metadata, context cache, runtime and applications
32B14.90 GiBQuantization metadata, context cache, runtime and applications
70B32.60 GiBQuantization metadata, context cache, runtime and applications

These figures are arithmetic lower bounds, not measured VRAM requirements. A real quantized file includes extra data; some formats also keep selected weights at higher precision. Check the exact file size and quantization, such as the Qwen3 tags and package sizes on Ollama.

Leave room for context and the runtime

Model weights are only one part of GPU memory use. The runtime also needs workspace, and the application needs room for its other allocations. The key/value (KV) cache stores attention data for tokens in the active context; its size depends on the model architecture, cache format and how many tokens are in use. Longer contexts and concurrent requests can use substantially more memory.

Ollama's context-length documentation says: “Setting a larger context length will increase the amount of memory required to run a model.” (source). Quantization changes the weight representation, while KV-cache options are a separate consideration; see the Hugging Face quantization guide and KV-cache documentation.

A card marketed as 32 GB holds about 29.80 GiB using binary units. That is already below the ideal 32.60 GiB weight estimate for a 70B model at 4-bit, before any overhead. Smaller models can still fail to fit if the actual file, context, runtime and display use together exceed available VRAM.

Practical VRAM starting points

Use these as ranges to test, not official minimums or fit guarantees. The example cards are comparisons, not a promise that every model configuration will fit.

Planning ranges only; verify the exact model file, context length and runtime.
Starting pointGPU memoryWhat to tryExample cards
8 GB class8 GBTry a small quantized model at a short context; an 8B model can still be tight once the actual file and runtime are included. sourceRTX 4060, RTX 5060, RTX 3060 (12GB version)
12–16 GB class12–16 GBMore room for smaller models and some larger quantized files; verify the exact package and context rather than relying on parameter count. sourceRTX 5060 Ti (16GB version), RTX 4060 Ti (16GB version), RTX 5070 Ti
24 GB+ class24 GB+More headroom for mid-size quantized models or longer contexts, but this still is not a fit guarantee for every configuration. sourceRTX 3090, RTX 4090, RTX 5090

Lowest recorded prices by tier

The cheapest New and Used listings we have recorded for each threshold, among NVIDIA GeForce RTX cards with at least that much VRAM. Used includes renewed and refurbished cards. These are recorded Amazon US prices, not live checkout quotes; the model page shows the listings and their conditions.

Computed from recorded Amazon US listings for this page response.
TierLowest NewLowest Used
8 GB class (8GB+)$419.99 RTX 5050 8GB
Recorded 2026-10-24 11:09:56 UTC
$249.97 RTX 2060 SUPER 8GB
Recorded 2026-09-30 05:03:30 UTC
12–16 GB class (12GB+)$459.99 RTX 3060 12GB
Recorded 2026-11-05 19:57:25 UTC
$419.97 RTX 3060 12GB
Recorded 2026-11-21 07:09:48 UTC
24 GB+ class (24GB+)$2,499.99 RTX 3090 24GB
Recorded 2026-11-21 07:11:12 UTC
$1,777.22 RTX 3090 24GB
Recorded 2026-10-24 04:46:46 UTC

Check whether a model will fit

  1. Choose the exact model and quantized file; use its download or package size as a better starting point than parameter count alone.
  2. Set the context length you actually plan to use. A short prompt and a long document can have different cache needs.
  3. Close GPU-heavy applications, load the model, and check allocated VRAM in your runtime or system monitor.
  4. Test representative prompts and, if relevant, simultaneous requests. Keep a margin rather than planning to use every last MiB.

If it does not fit, try a smaller model or quantization, reduce context length, or offload layers to system RAM. Offloading can reduce speed; multiple GPUs do not always behave like one seamless memory pool.

Compare GPUs and estimate memory

Use the LLM VRAM estimator for a quick weight estimate, the local LLM GPU guide for hardware context, and GPU prices by VRAM for recorded listings. Compare cards: /compare/rtx-3060-vs-rtx-4060/, /compare/rtx-5060-ti-vs-rtx-4060-ti/.

Local LLM VRAM questions

How much VRAM does an 8B model need?

At ideal four-bit weights, 8B parameters are about 3.73 GiB before format overhead, KV cache, runtime workspace and other GPU use. The exact file and context determine whether it fits.

Can a 32 GB GPU run a 70B model at four bits?

The ideal weight-only estimate is about 32.60 GiB, already above the roughly 29.80 GiB represented by 32 decimal GB. Real files and runtime overhead add more, so a single 32 GB card is not enough for that idealized estimate.

Does a longer context use more VRAM?

Yes. The KV cache stores attention data for tokens in the active context, so a longer context can require more memory. The exact amount depends on the model, cache format and runtime.

Sources

Memory needs vary by exact model file, runtime and workload. Check current documentation before buying hardware.