GPU Memory Calculator
An estimate of the GPU memory (VRAM) a language model needs to run or to fine-tune: its weights at a precision, the KV cache for a context and batch, Adam optimizer states or LoRA adapters, and a margin you choose.
Result…
How it's calculated
Weights = parameters × bytes per parameter (fp32 4, fp16/bf16 2, int8 1, int4 0.5) KV cache = 2 × layers × (hidden size, or KV heads × head size) × context × batch × bytes Full fine-tune (mixed-precision Adam) = parameters × 16 bytes LoRA = base weights + trainable parameters × 16 bytes Total = sum × (1 + margin ÷ 100)
A model's weights take its parameter count times the bytes each one is stored in. To generate text it also keeps a key and a value vector for every token in every layer, the KV cache, which grows with the context and the batch; models with grouped-query attention store fewer key/value heads, so enter those rather than the hidden size. Full fine-tuning with Adam in mixed precision needs about 16 bytes a parameter (fp16 weights and gradients plus fp32 master weights, momentum and variance), as set out in the ZeRO paper; some setups keep gradients in fp32 and need more. LoRA freezes the base model and trains small adapters, so only they carry gradients and optimizer states. Activations, which can dominate training memory, are not modelled: the margin is where you allow for them and for framework overhead. Enter the layers, hidden size and heads from the model's own config file.
Your numbers, step by step
About the GPU Memory calculator
This GPU memory calculator estimates how much VRAM a language model needs before you rent a server or buy a card. To run a model, it adds the weights (parameters times the bytes each one is stored in, from fp32 down to int4) to the KV cache, which grows with the layers, the key/value size, the context length and the batch. To fine-tune, it counts about 16 bytes a parameter for full training with Adam in mixed precision, or the frozen base weights plus small LoRA adapters. You enter the layers, hidden size and heads from the model's own config file, so it works for any model rather than a fixed list. A margin of your choosing covers activations and framework overhead, and an optional per-GPU size shows how many cards the total fills. Every step is shown with your numbers, and the result is labelled for what it is: an estimate.
How to use it
- Choose the “Use”: “Run (inference)”, “Full fine-tune (Adam)” or “LoRA fine-tune”.
- Enter the “Model size” in billions of parameters and pick the “Weight precision”.
- For inference, enter the “Layers” and either the “Hidden size” or the “Key/value heads” and “Head size”, then the “Context length” and “Batch size”.
- For LoRA, enter the “Trainable share (adapters)” your training script reports.
- Set an “Extra margin” for activations and overhead, and optionally the “Memory per GPU (optional)” to see how many GPUs you need.
Worked example
An 8-billion-parameter model in fp16 takes 8 × 10⁹ × 2 bytes = 16 GB of weights. With 32 layers, 8 key/value heads and a head size of 128, each token's KV cache is 2 × 32 × 8 × 128 × 2 = 131,072 bytes, so an 8,192-token context for one sequence adds about 1.07 GB. A 10% margin brings the total to about 18.78 GB (17.49 GiB), which fits on one 24 GB GPU. Full fine-tuning the same model with Adam needs 8 × 10⁹ × 16 bytes = 128 GB before activations, while LoRA training 0.5% of the parameters needs about 16.64 GB.
Updated
Questions people ask
How do I calculate the VRAM needed to run an LLM?
Multiply the parameter count by the bytes per parameter for the weights, then add the KV cache: 2 × layers × key/value size × context × batch × bytes. Add a margin for the framework and activations.
Why does full fine-tuning need so much more memory than inference?
Training with Adam in mixed precision keeps fp16 weights and gradients plus fp32 master weights, momentum and variance, about 16 bytes a parameter, as set out in the ZeRO paper. Inference only needs the weights and the KV cache.
What does grouped-query attention change in the KV cache?
Models with grouped-query attention share each key/value head across several query heads, so they store far fewer keys and values. Enter the key/value heads and head size from the config instead of the hidden size, or the cache is overstated.
How does quantizing to int4 or int8 reduce memory?
The weights shrink to a half or a quarter of their fp16 size, since int8 uses 1 byte and int4 half a byte a parameter. Quantized formats also store small scale factors, and the KV cache keeps its own precision, so the saving is a little less than the weights alone suggest.
How accurate is this GPU memory estimate?
It counts the main parts but not activations, attention kernel workspace, the CUDA context or fragmentation, which vary by framework and settings. Treat it as a planning figure and measure the real peak on your own setup. The LLM token calculator helps size the context, and the LLM cost calculator compares hosted pricing.