LLMScalelive Hugging Face data

Fine-tuning VRAM — LoRA, QLoRA and full fine-tune memory

What does fine-tuning cost in GPU memory? Training drops the KV cache but adds gradients, optimizer state and the activations kept for the backward pass — which is why a model you can serve on one card often cannot be trained on it.

1

Choose a model

built-in DBgatedllama8.03B params·32L · 4096d · 32/8 headsGQA
2

Training setup

Type exact numbers instead of picking from the lists. Remembered in this browser.
ⓘThe LoRA adapter's rank: the width of its low-rank update, which sets how many trainable parameters it adds.13.6M
ⓘSequences processed in each optimizer step on each GPU.
ⓘTokens in each training example.131,072
3

Memory needed

16.63 GiB
per device
Base weights — 14.96 GiBGradients — 26 MiBOptimizer — 156 MiBActivations — 512 MiBOverhead — 1.00 GiB

Freezing the base removes almost all gradient and optimizer memory, but the backward pass still runs through every layer — so activations cost the same as a full fine-tune. Checkpointing, batch size and sequence length are the levers left.

4

Which GPUs fit

budgets 95% of each card, as vLLM does
fits ✓
16.63 GiB / 24 GB (74% of usable)

Mixed-precision Adam at 16 bytes/parameter (bf16 weights + bf16 gradients + fp32 master, m and v). Activations follow Korthikanti et al. 2022 at ~34 bytes per token per layer per hidden unit, flash-attention assumed, or 2 bytes with full recomputation. Real runs vary with framework and kernel choices; treat this as a planning figure.

How fine-tuning memory is worked out

Fine-tuning holds more than the weights. Every trainable parameter also needs a gradient and optimizer state, and the forward pass keeps activations for the backward pass. LoRA and QLoRA freeze the base model and train small adapters, so gradients and optimizer state shrink to the adapters' size; QLoRA also stores the frozen base in 4-bit NF4.

  • base weights = parameters × 2 bytes (BF16), or × 0.516 bytes (QLoRA NF4)
  • gradients = trainable parameters × 2 bytes
  • optimizer (Adam) = trainable parameters × 12 bytes: FP32 master weights + two moments
  • activations = layers × sequence length × batch × hidden size × 34 bytes, or × 2 with gradient checkpointing
  • total = base + gradients + optimizer + activations + 1 GiB overhead

A full fine-tune trains every parameter; for LoRA the count follows from the rank and the target modules — the attention projections, or every linear layer. Activations assume flash attention. The engine is checked against two published results: the ZeRO paper's 16 bytes per parameter for mixed-precision Adam, and QLoRA fine-tuning a 65B model on one 48 GB card, which it puts at 43.3 GiB.

Questions

How much VRAM does a full fine-tune need?
About 16 bytes per parameter before activations: 2 for the weights, 2 for gradients and 12 for Adam. That is roughly 104 GiB for a 7B model, which is why a full fine-tune usually means several GPUs, or sharding with ZeRO-3 or FSDP.
LoRA or QLoRA — what is the memory difference?
Both train the same adapters; the difference is the frozen base. LoRA keeps it at BF16 (2 bytes per parameter), QLoRA at NF4 (0.516 bytes). For an 8B model that is about 14.9 GiB against 3.8 GiB for the base weights alone.
Does LoRA reduce activation memory?
No. The forward pass still runs through the whole model, so activations are the same as in a full fine-tune. Gradient checkpointing is what cuts them: it keeps only each layer's input — 2 bytes instead of 34 per layer, token and hidden unit — and recomputes the rest.
Why is there no KV cache here?
Training processes whole sequences in one pass and does not cache keys and values between steps the way text generation does, so the KV cache term from the inference calculator does not apply.

Go deeper