Qwen3-32B VRAM Calculator: FP16 vs Q8_0 vs Q4_K_M Memory Math and GPU Benchmarks

Qwen3-32B is the largest dense model in the Qwen3 family and one of the most capable models you can run on a single workstation GPU. Whether it fits depends almost entirely on two choices: quantization (how many bits each weight uses) and context length (how many tokens the KV cache must hold). This guide works through the memory math from first principles, so you can size hardware before you buy it, and then shows the exact commands to serve the model with Ollama and vLLM.

If you just want a number, use the calculator at the end of this article. If you want to know why the number is what it is, read on.

The short answer

All figures include about 1 GB of runtime overhead (CUDA or Metal context plus activation buffers) and assume an unquantized FP16 KV cache. Gigabytes here are binary (GiB), which is how GPU memory is reported.

The practical takeaways:

• Q4_K_M at 8k fits a 24 GB RTX 4090 with a little headroom.
• Q4_K_M at 32k needs a 32 GB RTX 5090.
• Q8_0 doesn't fit on any single consumer GPU. You need two cards, a 48 GB+ workstation card or Apple unified memory.
• FP16 is a multi-GPU or Mac Studio workload.

Step 1: Weight memory

Weight memory is the easy part: parameter count multiplied by bytes per parameter.

Qwen3-32B has 32.8 billion parameters. In FP16 (or BF16) each parameter takes 2 bytes:

32.8e9 params × 2 bytes = 65.6e9 bytes ≈ 61.1 GiB

Quantized GGUF formats store weights in blocks with a small amount of scaling metadata, so their effective bits per weight are slightly higher than the name suggests:

• Q8_0: 8-bit weights plus one FP16 scale per 32-weight block, about 8.5 bits per weight.
• Q4_K_M: a mix of 4-bit and 6-bit "k-quant" blocks; the M (medium) variant keeps more sensitive tensors such as attention value projections at higher precision. It averages roughly 4.85 bits per weight.

Q8_0:   32.8e9 × 8.5 / 8  ≈ 34.9e9 bytes ≈ 32.5 GiB
Q4_K_M: 32.8e9 × 4.85 / 8 ≈ 19.9e9 bytes ≈ 18.5 GiB

These line up with the published GGUF file sizes for the model, which is a good sanity check: the file you download is essentially the weight memory you will need.

Step 2: KV cache memory

The KV cache stores the attention keys and values for every token in the context, for every layer. It grows linearly with context length, and at long contexts it can rival the weights themselves.

The formula is:

KV bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_value

From the Qwen3-32B model config:

• 64 transformer layers
• 64 query heads, but only 8 key/value heads (grouped-query attention)
• head dimension of 128

With an FP16 cache (2 bytes per value):

2 × 64 × 8 × 128 × 2 = 262,144 bytes = 256 KiB per token

Multiply by the context length:

Grouped-query attention is what makes long contexts feasible here. With 64 KV heads instead of 8, the same 32k cache would need 64 GB.

Qwen3-32B supports 32,768 tokens natively and up to 131,072 with YaRN rope scaling. Only enable YaRN when you actually need long inputs, because it can slightly reduce quality on short prompts.

Quantizing the KV cache

Both Ollama and vLLM can store the KV cache at 8-bit precision, halving its footprint. That is often the difference between fitting and not fitting at 32k:

• Ollama: set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0.
• vLLM: pass --kv-cache-dtype fp8.

An 8-bit KV cache has a small quality cost that is usually hard to measure in practice. A 4-bit cache (q4_0 in Ollama) saves more but degrades long-context recall more noticeably.

Step 3: Hardware estimates

Single-user decode speed on a local GPU is almost always limited by memory bandwidth: every generated token requires reading all the weights (plus the filled KV cache) once. That gives a useful first-order estimate:

tokens/sec ≈ memory bandwidth × efficiency ÷ bytes read per token

The table below applies that formula with a 70% efficiency factor, which is typical of llama.cpp-based runtimes for dense models. Treat these as estimates for planning, not measured results. Real numbers vary by runtime version, driver, prompt length and batch size.

A few notes on the hardware:

• Why no M4 Ultra? Apple has not shipped an M4 Ultra; the current Mac Studio tops out with the M3 Ultra (and the M4 Max in lower configurations). The M3 Ultra's base configuration has 96 GB of unified memory, of which macOS lets the GPU use roughly 75% by default. You can raise that limit with sudo sysctl iogpu.wired_limit_mb=<value>, at the cost of memory for everything else.
• The RTX 5090's advantage is bandwidth as much as capacity. At 1,792 GB/s it decodes Q4_K_M almost twice as fast as the 4090, and its extra 8 GB is exactly what a 32k context needs.
• "Doesn't fit" means doesn't fit on the GPU. llama.cpp and Ollama can offload some layers to system RAM, but decode speed then falls to single digits because PCIe and DDR5 bandwidth are an order of magnitude lower than VRAM.

Prompt processing is a different story

The estimates above cover generation. Prompt processing (prefill) is compute-bound rather than bandwidth-bound, and NVIDIA GPUs are dramatically faster than Apple silicon at it. If your workload involves feeding long documents (RAG, contract review, log analysis), the gap between an RTX 5090 and a Mac Studio is much larger than the decode numbers suggest: expect time-to-first-token on a 30k-token prompt to differ by several times.

Which quantization should you use?

Q4_K_M is the default for good reason. On 30B-class models the quality loss compared with FP16 is small on most benchmarks, and it is the only option that fits on a single consumer GPU. Start here.

Q8_0 is close to lossless and worth it when exact numbers or code matter (financial extraction, structured output, agentic tool use) and you have the memory. On consumer hardware that means two GPUs or a Mac.

FP16 is mainly useful as a reference baseline, or when you serve with vLLM and want maximum throughput across many concurrent users on data-center GPUs.

Running Qwen3-32B with Ollama

Ollama's qwen3:32b tag is the Q4_K_M build. Pull and run it:

ollama pull qwen3:32b
ollama run qwen3:32b

For the 8-bit build:

ollama pull qwen3:32b-q8_0

Ollama defaults to a short context window. Raise it server-wide, and enable the 8-bit KV cache so 32k fits comfortably on a 32 GB card:

OLLAMA_CONTEXT_LENGTH=32768 \
OLLAMA_FLASH_ATTENTION=1 \
OLLAMA_KV_CACHE_TYPE=q8_0 \
ollama serve

Check how much memory the loaded model is actually using, and whether any of it spilled to the CPU:

ollama ps

The PROCESSOR column should read 100% GPU. Anything else means layers were offloaded and you should lower the context or the quantization.

Running Qwen3-32B with vLLM

vLLM is the better choice when several users or services hit the model at once, thanks to continuous batching and PagedAttention. It works with Hugging Face checkpoints rather than GGUF.

On a single 32 GB RTX 5090, use the official AWQ 4-bit checkpoint with an FP8 KV cache:

vllm serve Qwen/Qwen3-32B-AWQ \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser qwen3

For full BF16 precision, split the model across two GPUs with tensor parallelism:

vllm serve Qwen/Qwen3-32B \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --reasoning-parser qwen3

vLLM pre-allocates a fraction of GPU memory (--gpu-memory-utilization) for weights and KV cache at startup. If it fails with an out-of-memory error, reduce --max-model-len before anything else. Both servers expose an OpenAI-compatible API, so existing client code works with a changed base URL:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-32B-AWQ", "messages": [{"role": "user", "content": "Summarize grouped-query attention in two sentences."}]}'

Measure your own numbers

Estimates get you to the right hardware tier; measurements confirm it. While the model generates, watch actual memory use:

nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv -l 1

And for a quick throughput figure, Ollama prints timing statistics with --verbose:

ollama run qwen3:32b --verbose "Explain the difference between Q4_K_M and Q8_0 quantization."

Look at the eval rate line for decode speed and prompt eval rate for prefill. Run each test several times and discard the first, which includes model load time.

Size other configurations

Use the calculator below to try other context lengths and quantizations, or compare Qwen3-32B against gpt-oss and DeepSeek-R1 distills. It uses the same formulas as this article.