GPU Training Cost Calculator
Estimate the compute to train a language model (about 6 × parameters × tokens FLOPs) or run it on tokens (about 2 × parameters per token), the GPU-hours at your GPU's peak speed and utilisation, the wall-clock time on N GPUs and the cost at your hourly price.
Result…
How it's calculated
Training compute C ≈ 6 × N × D FLOPs (N parameters, D tokens) Inference compute ≈ 2 × N × D FLOPs GPU-hours = C ÷ (peak FLOPS × utilisation) ÷ 3600 Wall-clock = GPU-hours ÷ GPUs Cost = GPU-hours × price per GPU-hour 1 PF-day = 8.64 × 10^19 FLOPs
Training a dense transformer takes about 6 floating point operations per parameter per token: about 2 for the forward pass and 4 for the backward pass. That estimate comes from Kaplan et al., "Scaling Laws for Neural Language Models" (2020), which also defines the PF-day used here. Dividing the total by what one GPU actually achieves, its peak speed times the utilisation (model FLOPs utilisation, MFU), gives GPU-hours, which set the cost at your hourly price. You enter the GPU's peak speed from its datasheet, so the estimate works for any GPU or accelerator. For whether the model fits in memory, use the GPU memory calculator.
Your numbers, step by step
About the GPU Training Cost calculator
This GPU training cost calculator gives a first estimate of the compute, time and money needed to train a language model. It uses the rule that training a dense transformer takes about 6 floating point operations per parameter per token, so the total is about 6 × parameters × tokens FLOPs, shown also in PF-days. You enter your GPU's peak speed from its datasheet and the utilisation you expect, and it works out the GPU-hours, the wall-clock time on your number of GPUs and the cost at the hourly price you pay, plus the cost per million tokens. An inference mode uses about 2 FLOPs per parameter per token for processing text. It suits researchers, startups and students planning a training run or comparing cloud offers. No GPU specifications or prices are built in, and every step is shown.
How to use it
- Choose what to “Estimate”: “Training” or “Inference (processing tokens)”.
- Enter the “Model parameters” and “Tokens” with their units.
- Enter the “GPU peak speed” from the datasheet and the “Utilisation (MFU)” you expect.
- Enter the “Number of GPUs” and the “Price per GPU-hour”, then read the cost, GPU-hours and wall-clock time.
Worked example
Training a 7 billion parameter model on 1 trillion tokens takes about 6 × 7 × 10^9 × 10^12 = 4.2 × 10^22 FLOPs, or 486.1 PF-days. On GPUs with a peak of 500 TFLOPS at 40% utilisation, each achieves 2 × 10^14 FLOPs per second, so the run needs 58,333 GPU-hours. On 64 GPUs that is about 37 days 23 hours, and at 2.50 per GPU-hour it costs about 145,833 in your currency.
Updated
Questions people ask
How do I estimate the FLOPs needed to train a model?
Multiply 6 by the number of parameters and the number of training tokens. The 6 comes from about 2 FLOPs per parameter per token for the forward pass and about 4 for the backward pass, as set out in Kaplan et al., Scaling Laws for Neural Language Models.
What utilisation (MFU) should I enter?
Model FLOPs utilisation is the share of the GPU's peak speed your training actually achieves. It depends on the model, the software and how many GPUs talk to each other, so measure it on a short run of your own setup if you can.
Which peak FLOPS figure should I use from a GPU datasheet?
Use the dense figure for the number format you train in, such as BF16. Datasheets often also list a higher figure with sparsity, which ordinary dense training does not reach.
Why might the real cost be higher than this estimate?
The estimate covers one clean run. Experiments, restarts, evaluation, data preparation and idle GPUs all add to the bill, and spreading work over many GPUs adds communication overhead.
Does this calculator check whether the model fits in GPU memory?
No, it estimates compute only. Use the GPU memory calculator to check memory, and the LLM cost calculator to compare with paying per token for a hosted model.