Finetuning LLMs at Home

If you are already running models locally, you know the limits of prompting. You can steer a model with clever instructions, but prompts use up precious context space. If only the instructions could be baked into the model. LoRA and QLoRA make that possible. They let you tailor a pretrained model without needing expensive cloud servers or ongoing API bills. For smaller models, a single graphics card from a gaming PC is enough. Larger models still need serious hardware, but the barrier to entry has dropped far enough that most people can get started.

What is LoRA?

AI models contain billions of internal settings. Adjusting all of them requires enormous computing power. LoRA avoids the problem entirely.

Instead of changing every setting, LoRA freezes the original model and attaches small tunable weights to its layers. These additions are called adapters. Layers are the stacked processing stages inside a model, each transforming data before passing it along. During fine-tuning, only the adapter weights change. The original model stays locked. When you use the model afterward, the adapters steer its output toward the task they were trained on.

The result is the same: a model customized for your task. The difference is scale and speed. Because LoRA only trains a fraction of the parameters, it finishes in a fraction of the time. Adapters for a medium-sized model take up only megabytes of space, while the full model takes gigabytes.

What is QLoRA?

QLoRA is LoRA applied to quantized models.

Quantization is a general method for shrinking models. It reduces the precision of the numbers a model uses internally, like lowering the resolution of a photograph. The image is smaller and takes up less space, but still recognizable. Eight-bit quantization cuts model weights in half with minimal quality loss. Four-bit cuts weights to a quarter and still works. LoRA works with any quantization level. QLoRA specifically uses 4-bit NF4 because the original paper showed it is the smallest option that reliably preserves quality during fine-tuning. People use quantization to run larger models on consumer hardware and cut costs.

Attach the tunable adapter weights to a frozen quantized model, and you get QLoRA.

The adapter pieces still train at full precision. Only the frozen base model is quantized. This means the learning stays accurate even though the base model itself is heavily compressed.

QLoRA makes it possible to customize an AI model on a single powerful graphics card. Without quantization, that same model would need multiple cards just to load.

Comparison

	Full Fine-tuning	LoRA	QLoRA
Quality	Baseline (best)	Within a few % of full FT study	1-3% below LoRA paper
VRAM needed	~88 GB (Enterprise GPU)	~20 GB (Consumer GPU, with gradient checkpointing)	~8 GB (Consumer GPU)
Cost	$15–$23 (cloud GPU rental)	~$0.40 (local electricity)	~$0.30 (local electricity)
Training time	8-12 hours	6-9 hours	4-6 hours

Estimates based on Llama 3.1 8B, ~1K samples, 3 epochs, batch size 1, sequence length 512. LoRA and QLoRA VRAM figures assume gradient checkpointing, which trades compute for memory by recomputing activations during backpropagation. Actual values vary by dataset and hardware.

Use Cases

Teaching formats and style	Show a model how to follow your format or match your voice. Customer support bots, specialized assistants, branded tone. The most common use and clearest return. Style adaptation with LoRA.
Personalization	Tailor a model to individual user preferences. Each user gets a small customization trained on their data. Private and portable between systems. Domain adaptation guide.
Fast experimentation	Try ten different approaches in the time one full customization takes. LoRA makes trial and error cheap enough to actually explore.
Serving many customers	Host one base model, swap customizations per request. Serve dozens of tailored versions from a single GPU. Multi-tenant LoRA serving.
Specializing small models	A fine-tuned small model can outperform a larger, unfine-tuned one on its specific task. The LoRA Land study ran 310 fine-tuned models across 31 tasks and found 4-bit LoRA models beat GPT-4 by 10 points on average. LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4.

Limitations

Major new abilities	Teaching a model something completely new, like a language it has never seen, requires more than a small adjustment. Full customization or deeper fine-tuning is more appropriate.
Rank selection	LoRA approximates weight changes through low-rank matrices. Pick too low a rank and you lose precision; pick too high and you lose the efficiency advantage. The right value depends on task complexity and there is no universal default.
Layer coverage	You need adapters on all linear layers to match full fine-tuning quality. Targeting only a few layers produces noticeably worse results. The QLoRA paper proved this empirically.
Hyperparameter sensitivity	Beyond learning rate and epochs, LoRA adds rank, alpha, dropout, and layer selection. Getting these wrong is a frequent cause of poor results. LoRA Hyperparameters Guide.
Data quality dependency	LoRA amplifies whatever is in your training data. Bad data produces bad adapters. No technique compensates for that.

Where to go next

The tools are mature and well-documented. The hard part is preparing good data, not writing the code.

Libraries

PEFT	Hugging Face’s parameter-efficient fine-tuning library. Handles LoRA, QLoRA, and other PEFT methods.
TRL	Transformer Reinforcement Learning. Training loop with support for supervised fine-tuning, DPO, and PPO.
bitsandbytes	4-bit quantization (NF4) that powers QLoRA.
Accelerate	Distributed training and device placement. Handles multi-GPU setups without boilerplate.
Axolotl	Config-driven fine-tuning framework. YAML config defines the entire training run.
Unsloth	Optimized fine-tuning with 2x speed and lower memory. Drops in for PEFT.

Common Datasets

Alpaca	Stanford’s 52K instruction-following dataset. The original benchmark for LoRA fine-tuning.
OpenAssistant/Guanaco	Conversational data collected from real user interactions. Used in the QLoRA paper.
Databricks Dolly	15K instruction-response pairs across eight prompt categories.
LongForm	Extended-form responses for training models to generate detailed output.

---