OPTIMIZATION GUIDE8 min read • Updated 2026-10-25

How to Fine-Tune 7B–32B LLMs Locally on an RTX 4090 with Unsloth (QLoRA VRAM Guide)

Exact VRAM budgets, batch sizes, and Unsloth 4-bit QLoRA configurations for training Qwen, Llama, and DeepSeek on a single 24GB GPU.

Alex Vance

Inference Systems Engineer

Fine-tuning open-weights LLMs used to require multi-A100 clusters. With Unsloth's custom Triton kernels and 4-bit QLoRA gradient checkpointing, a single 24GB NVIDIA RTX 4090 or RTX 3090 can fine-tune 8B to 32B models locally. Here is the exact VRAM math for weights, LoRA adapters, optimizer states, and activations.

Key Architectural Takeaways

Unsloth reduces QLoRA training VRAM by ~60–70% compared to standard Hugging Face TRL + FlashAttention-2 via custom Triton backpropagation kernels.
An 8B–14B model fine-tunes comfortably in 9GB–14GB VRAM at 8K sequence length on a single RTX 4090.
A 32B model (Qwen 2.5 Coder 32B / DeepSeek-R1 32B) fits inside 24GB VRAM using 4-bit NormalFloat (nf4), rank r=16, batch_size=1, and Unsloth gradient checkpointing.
After training, export directly to GGUF (Q4_K_M or Q8_0) for Ollama or vLLM deployment.

1. Training VRAM vs. Inference VRAM: The QLoRA Math

During inference, your GPU only stores frozen quantized weights and the KV cache. During fine-tuning, VRAM must hold four components simultaneously:

Base Model Weights (4-bit NF4): ~0.55 GB per billion parameters (e.g., ~17.8 GB for a 32B model).
LoRA Adapter Weights + Gradients (BF16): Typically 1%–2% of parameter count (~0.4–0.8 GB at rank r=16).
8-bit AdamW Optimizer States: Paged AdamW offloads spikes to system RAM if VRAM crosses 95%.
Forward Activations: Scales linearly with max_seq_length × per_device_train_batch_size. Unsloth's use_gradient_checkpointing = "unsloth" offloads activations to system RAM with <1% throughput penalty.

Model Size	4-Bit Base Weights	Peak QLoRA VRAM (4K Seq)	Peak QLoRA VRAM (16K Seq)	24GB RTX 4090 Fit
8B–9B (Llama 3.1 8B / Gemma 2 9B)	5.2 GB	7.4 GB	10.8 GB	Fits Easily (Batch 4)
14B (DeepSeek-R1 14B / Phi-4 14B)	8.6 GB	11.5 GB	15.9 GB	Fits Easily (Batch 2)
27B–32B (Qwen 3.8 27B / Qwen 2.5 32B)	17.8 GB	21.4 GB	23.6 GB	Tight Fit (Batch 1 + Paged AdamW)
70B (Llama 3.3 70B)	39.5 GB	46.2 GB	54.0 GB	Requires H100 80GB or 2x 3090/4090

2. Production Unsloth QLoRA Script for 24GB RTX 4090

Use the following configuration to fine-tune up to 32B models on a single 24GB GPU without triggering CUDA OOM:

from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Qwen2.5-Coder-32B-Instruct-bnb-4bit",
    max_seq_length = 8192,
    dtype = None,
    load_in_4bit = True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = "unsloth",
)

Always set optim = "paged_adamw_8bit" and per_device_train_batch_size = 1 with gradient_accumulation_steps = 8 when training 27B–32B models on 24GB cards.

3. Exporting Your Fine-Tuned Model to GGUF & Ollama

Once training finishes, merge and quantize directly to Q4_K_M or Q8_0 for local inference:

# Export directly to 4-bit GGUF and generate Ollama Modelfile
model.save_pretrained_gguf("finetuned_model", tokenizer, quantization_method = "q4_k_m")

If your dataset requires 70B QLoRA training or full 16-bit LoRA weights that exceed 24GB VRAM, rent an on-demand NVIDIA H100 (80GB) or A100 (80GB) pod on RunPod or Vast.ai for 2–3 hours (~$4–$6 total training cost), then download the resulting GGUF back to your local RTX 4090.

INTERACTIVE BENCHMARK TOOL