GPU VRAM Calculator for LLMs

What it does

How to use GPU VRAM Calculator for LLMs

1. 1
   Pick a preset or your model

   Click one of the 8 presets ("Run Llama 3.3 70B locally", "QLoRA fine-tune Llama 3.3 70B", "Run DeepSeek R1 671B locally", etc.) for a sensible starting point. Or pick a model from the searchable dropdown — Llama 3.x, Mistral / Mixtral, Qwen 3, DeepSeek V3 / R1, Gemma 3, Phi 4, and more.

2. 2
   Choose your use case

   Inference (running the model), LoRA (parameter-efficient fine-tune), QLoRA (4-bit base + adapters — the standard for fine-tuning large models on a single GPU), Full fine-tuning (every parameter trainable, ~7× the VRAM of inference), or Pre-training. Inputs change automatically.

3. 3
   Pick a precision / quantization

   For inference: fp16 (full quality), AWQ / GPTQ 4-bit (production), GGUF Q4_K_M (most popular for local), or anything from Q8_0 down to Q2_K. For training: bf16 mixed precision is the modern default. Each option shows bytes/param and qualitative quality loss.

4. 4
   Set context, batch, and framework

   Drag the context-length log slider (512 → 1M tokens) or click a quick pill. Set batch size (1 to 256). For inference, pick your framework (vLLM, llama.cpp, TensorRT-LLM, ExLlamaV2, Ollama, MLC-LLM). For training, pick your optimizer (AdamW / 8-bit AdamW / Lion / AdaFactor) and ZeRO / FSDP stage.

5. 5
   Read the VRAM card

   The hero number at the top is your total VRAM in GB, with a 5% safety margin. Below it: a green list of GPUs that fit (with how many you need) and a red list that won't. Borderline fits (over 92% utilisation) are flagged amber.

6. 6
   Open "Show the math"

   Every estimate expands to show the exact formulas — model weights = params × bytes/param, KV cache = 2 × n_layers × n_kv_heads × head_dim × seq × batch × bytes, plus optimizer state, gradients, activations, and overhead. Auditable, reproducible, copy-friendly.

7. 7
   Use the "What If?" panel

   Live table ranking 5–12 alternative configurations by VRAM delta — switch precision, halve the context, enable gradient checkpointing, switch to QLoRA, etc. The single highest-leverage feature for "how do I make this fit?" questions.

8. 8
   Project cloud cost or share the link

   The Cloud Cost Projector picks any fitting GPU configuration and shows hourly / monthly / yearly cost across 8 providers (RunPod, Lambda Labs, Vast.ai, CoreWeave, AWS, GCP, Azure). Copy the shareable link to send the exact configuration to a teammate or paste in a Reddit / X thread.

When to use this

Common errors & fixes

Calculator shows you fit on a single 4090 but you OOM at runtime

Two likely culprits. (1) Activation factor — real activation memory varies by impl; bump it from 15% to 25% in Advanced and re-check. (2) KV cache grows with actual prompt length, not the slider — if you're serving real users hitting 8K-context prompts but the slider says 4K, the calculator under-estimates. Set the slider to your worst-case context.

QLoRA fine-tune appears to fit on 24GB but bitsandbytes errors

NF4 dequantization workspace and the optimizer state can spike during the first backward pass. Try (a) gradient checkpointing on, (b) batch size 1, (c) 8-bit paged AdamW (bitsandbytes paged_adamw_8bit), (d) reduce LoRA rank from 64 to 16. The calculator surfaces these as "What If?" rows; pick the one that drops total below ~22 GB to leave headroom.

70B inference recommends an H100 but Llama.cpp on a 24GB 4090 actually works fine

You're probably running a CPU/GPU split with --n-gpu-layers. The calculator assumes the entire model lives in VRAM. For partial offload, divide n_layers proportionally. Llama.cpp's output gives you the exact split that fits — start there, not from the calculator.

KV cache is "only 2GB" in the calculator but 30GB in vLLM at runtime

vLLM pre-allocates a KV cache pool sized for many concurrent sequences (controlled by --gpu-memory-utilization). The calculator estimates the cache for one sequence at the configured batch size. To match vLLM's reservation, set batch size to your max concurrent sequences (typically 16–256 in production).

Multi-GPU recommendation says "8× H100" but DeepSpeed ZeRO-3 on 4× H100 works

The matrix recommends without ZeRO/FSDP partitioning. To model partitioned training, switch the use case to "Full FT", set ZeRO stage to ZeRO-3 or FSDP, and the per-GPU VRAM drops accordingly. The matrix then shows the smaller fit.

Estimate seems too high for AWQ — "Llama 70B AWQ should be 35GB, not 42GB"

You're comparing weights-only vs total VRAM. Weights at AWQ 4-bit = 70.6B × 0.5 ≈ 35.3 GB. Add KV cache (~2.5 GB at 8K context), activations (~3 GB with Flash Attention), and overhead (~1.5 GB) and you land near 42 GB. Open "Show the math" — every number is broken out.

Technical details

Frequently Asked Questions

How much VRAM do I need to run Llama 3.1 70B?

Depends on precision. fp16 ≈ 142 GB (won't fit any single GPU), Q8_0 ≈ 75 GB (fits 1× H100 80GB), AWQ 4-bit ≈ 40 GB (fits 1× RTX 6000 Ada 48GB or 1× A100 80GB), Q4_K_M ≈ 42 GB (same), Q3_K_M ≈ 35 GB (fits 1× RTX 6000 Pro Blackwell 96GB easily), Q2_K ≈ 26 GB (fits 1× RTX 5090 32GB). Add 2–5 GB for KV cache at 8K context. The calculator above gives the exact number for your configuration.

Can I run a 70B model on a single RTX 4090?

Yes, but only at Q2_K (~26 GB needed for weights, doesn't leave room for KV cache + overhead) or Q3_K_S with CPU offload of some layers via llama.cpp's --n-gpu-layers. Realistically, 4090 (24 GB) handles 70B at Q2_K with short context only. For practical 70B serving on consumer hardware, you want a 4090 + 3090 (48 GB combined) or a single RTX 6000 Pro Blackwell (96 GB) or RTX 5090 (32 GB at Q3_K_S).

What's the difference between GGUF Q4_K_M and AWQ?

GGUF Q4_K_M is llama.cpp's 4-bit format optimized for CPU + GPU mixed inference, with K-block quantization and selective higher-precision blocks for important weights ("M" = medium). AWQ is "Activation-aware Weight Quantization", a different 4-bit scheme optimized for GPU inference with vLLM, TensorRT-LLM, and ExLlamaV2. Quality is roughly equivalent (both within 1 perplexity point of fp16); AWQ tends to be faster on GPU because the kernels are GPU-optimized; Q4_K_M is the default for local llama.cpp / Ollama / LM Studio because of cross-platform support.

How much VRAM do I need to fine-tune Llama 3.1 8B?

Full fine-tune ≈ 85 GB (needs 4× A100 80GB with ZeRO-2). LoRA ≈ 28 GB (1× RTX 6000 Ada 48GB or 1× A100 40GB). QLoRA ≈ 12 GB (fits comfortably on a single 24 GB consumer GPU). Numbers are at batch size 1, seq length 2048, AdamW optimizer, gradient checkpointing on. Larger batches and longer sequences scale activations roughly linearly.

Why does long context use so much VRAM?

Because the KV cache scales linearly with sequence length and batch size. Llama 3.1 8B at 8K context with batch 1 has a KV cache of ~1 GB. At 128K context, that grows to ~16 GB — equal to the model weights themselves. Add concurrent users (batch 16) and you're at 256 GB of KV cache alone. The formula is `2 × n_layers × n_kv_heads × head_dim × seq × batch × bytes_per_kv` — see the calculator's "Show the math" for your specific numbers.

Is QLoRA worth the quality tradeoff vs full LoRA or full FT?

For most fine-tunes, yes. QLoRA hits 95–99% of full-FT quality on standard fine-tuning benchmarks (Alpaca, OpenOrca, etc.) at 5–10% of the VRAM. The compromises are real but small: marginally noisier gradients (4-bit base introduces quantization error), slightly worse loss curves, requires QLoRA-friendly optimizers (paged 8-bit AdamW). For domain adaptation, instruction tuning, or chat fine-tuning — QLoRA is the right default. For fine-tuning that pushes the base model into genuinely new capabilities (multilingual, novel reasoning), full LoRA on bf16 base sometimes pulls ahead.

Can I run DeepSeek V3 671B locally?

Yes if you have ~400 GB of VRAM total — typically 2× MI300X (192 GB each) or 4× H200 (141 GB each, with NVLink) or 8× H100 (80 GB each). At Q4_K_M ≈ 380 GB. The MLA architecture compresses KV cache 16× vs standard attention, so context length doesn't blow up the budget like it would for a non-MLA 671B model would. The "Run DeepSeek R1 671B locally" preset on the calculator above shows the math.

What's the cheapest GPU for running LLMs?

For under 13B models: a used RTX 3090 ($600–900) is unbeatable on cost-per-VRAM. For 30B–70B at Q4_K_M: RTX 5090 (32 GB, ~$2,000) or used dual 3090 (48 GB combined). For renting: Vast.ai marketplace spot instances start at $0.20/hr for 3090, $0.40/hr for 4090, $0.85/hr for A100 80GB, $1.49/hr for H100. For frontier models (405B, 671B), MI300X on TensorWave at ~$3.99/hr per GPU is often cheaper than H100 because of the larger 192 GB capacity.

Does CPU offloading actually work?

Yes, but with throughput hits. Llama.cpp's `--n-gpu-layers` keeps some layers on GPU and the rest in CPU RAM, giving you 3–10 tokens/sec for a 70B at Q4_K_M on a 24GB consumer card vs 30–60 tokens/sec when fully on GPU. For training, DeepSpeed ZeRO-Infinity offloads optimizer states to CPU and even NVMe; the calculator models this with the "CPU offload optimizer state" toggle, which can roughly halve training VRAM at the cost of significantly slower step time.

How much VRAM for batch inference at scale?

KV cache scales linearly with batch size. Llama 3.1 70B at fp16, 4K context, batch 1 ≈ 142 GB weights + 1.3 GB KV. At batch 32 ≈ 142 GB weights + 41 GB KV — total ~190 GB, fits 2× H100 80GB or 1× H200 141GB tightly. At batch 256 ≈ 470 GB total — needs 4× H200 or 3× B200. vLLM with paged attention squeezes more sequences into the same KV pool by avoiding fragmentation. Set batch size to your real concurrent serving load on the calculator above.

RTX 4090 vs RTX 5090 for LLMs — which to buy?

For LLM serving / fine-tuning, the 5090 is the clear winner — 32 GB vs 24 GB VRAM (33% more), 1.79 TB/s vs 1.0 TB/s memory bandwidth (1.79× — bandwidth is what bounds inference token throughput on quantized models), GDDR7 vs GDDR6X, and supports fp8 natively. The 5090 fits Llama 3.3 70B at Q3_K_S where the 4090 cannot. The price premium ($1,999 vs $1,599 MSRP) is justified for any LLM workload. For pure gaming or small-model dev, the 4090 is still excellent.

Can I split a model across multiple consumer GPUs?

Yes. Tensor parallelism (vLLM's `--tensor-parallel-size 2`, llama.cpp's automatic split, exllamav2's `gpu_split`) splits each layer across GPUs. Two RTX 3090s in tensor parallel = 48 GB usable VRAM and ~80% the throughput of a single A6000. NVLink helps but isn't required (3090 supports NVLink, 4090/5090 do not — communication falls back to PCIe and adds 5–15% overhead per extra GPU). The calculator factors a 12% per-extra-GPU overhead into multi-GPU recommendations to be honest about this.

What's the difference between fp8 and int8?

Both are 1 byte per parameter, but they're fundamentally different. fp8 (E4M3 or E5M2) is a floating-point format with exponent — it preserves a wide dynamic range, so it's near-lossless on most LLMs without recalibration. int8 is integer with a per-tensor or per-channel scale factor — it requires calibration data and typically loses 0.5–1.5 perplexity points without QAT. For inference, fp8 is the modern preference on H100/H200/B200 (which have native fp8 tensor cores); int8 is older and primarily lives on consumer cards (4090, 5090) via bitsandbytes.

How accurate is this calculator?

Within 10% of measured nvidia-smi values for the configurations we've tested. Sources of variance: (1) Activation memory varies 5–25% depending on attention implementation and CUDA version. (2) Framework overhead varies 0.5–4 GB by framework. (3) Quantization libraries (bitsandbytes, AutoGPTQ, exllamav2) have different workspace allocations. (4) Speculative decoding, prefix caching, and other vLLM features add memory not modelled here. (5) MoE expert prefetching can spike VRAM beyond steady-state. Always test with actual workloads before committing budget to hardware. The 5% safety margin in the headline number gives some cushion.

Why does my actual VRAM usage differ from the estimate?

Most common reasons: (1) You're running a different precision than configured (e.g. fp16 base with int4 quantized loras = mixed). (2) Your sequence length in production is longer than what you set in the slider — KV cache scales linearly with seq length. (3) Your batch size in production is bigger — same effect. (4) The framework reserves a KV cache pool larger than one sequence (vLLM's default). (5) You're using flash-attn-3 (saves more activation memory than flash-attn-2). (6) Real-world workloads include speculative decoding scratch, embedding lookups, RoPE caches that the calculator doesn't enumerate but typically fit within the 1.5 GB overhead. If your actual differs by more than 25%, please email devzonetools@gmail.com with your config — we'll add the case to the calibration set.

Does this calculator save my data anywhere?

No. All math runs in your browser. Your inputs are encoded into the URL query string so you can share configurations, but nothing is sent to a server. There's no signup, no telemetry on inputs, no analytics on dropdown selections. Same privacy model as the LLM Cost Calculator and RAG Cost Calculator.

Are closed-source models (Claude, GPT, Gemini) supported?

No — they're served via API, not self-hosted, so VRAM isn't a useful concept for them. To estimate API costs for closed models, use the LLM Cost Calculator instead (links from related tools). For open-weight equivalents (Llama 3.1 405B vs GPT-4-class quality, DeepSeek R1 vs o1-class reasoning, Qwen 3 235B vs frontier) — those are all in the dropdown.

Can I add a model not in the dropdown?

Yes — pick "Custom model" in the model selector, then enter total params, layers, hidden dim, attention heads, KV heads (for GQA), vocab size, and (if MoE) experts. The math works for any transformer-style architecture. For non-transformer models (Mamba, RWKV, hybrid) the formulas are different and not currently supported.

How do I share a specific configuration with someone?

Click "Copy shareable link" at the bottom of the outputs panel. The URL encodes every input as compact query parameters (e.g. `?m=llama-3.3-70b&q=gguf-q4_k_m&u=inference&s=8192`). Paste into Slack, Reddit, X — clicking the link recreates your exact configuration. Shareable links don't expire and don't track who clicked them.

Related Tools