What is fine-tuning?

What is fine-tuning?

Fine-tuning takes a model already trained on a large general dataset and trains it further on a smaller, task-specific one. The model adjusts weights it already has, instead of learning from scratch. The ULMFiT paper (Howard and Ruder, 2018) framed this as inductive transfer learning. A language model first learns general features of a language, then adapts to a target task.

The appeal is efficiency. ULMFiT showed that with only 100 labeled examples, its fine-tuned model matched the performance of training from scratch on 100 times more data on the IMDb dataset. That gap is why fine-tuning, not training from zero, became the default way to specialise a pretrained model.

Full fine-tuning versus adapters: what's the difference?

Full fine-tuning updates every parameter in the model. It gives the most flexibility but produces a complete new copy of the model for every task. The LoRA paper (Hu et al., 2021) points out why that becomes impractical at scale. Deploying independent fine-tuned instances of GPT-3, at 175 billion parameters each, is, in the authors' words, 'prohibitively expensive'.

Adapters offer an alternative: freeze the pretrained weights and train only small added modules. Earlier adapter designs, such as the one from Houlsby et al. cited in the LoRA paper, insert extra layers between existing ones. LoRA measured this cost directly. On GPT-2 medium with a batch size of one, adapter layers added as much as 20.7% to inference latency, because the added layers must be processed sequentially rather than in parallel.

ULMFiT's own ablations show full fine-tuning has a subtler problem even without adapters. Fine-tuning all layers at once on a small dataset like TREC-6 can cause catastrophic forgetting, where the model loses the general knowledge it started with. ULMFiT addressed this with three techniques: discriminative fine-tuning, which sets different learning rates per layer; slanted triangular learning rates; and gradual unfreezing, which unfreezes layers one at a time starting from the last.

How does LoRA work?

LoRA, short for Low-Rank Adaptation, freezes the pretrained weight matrices entirely. Instead it trains a pair of small matrices that represent the change those weights would need. The LoRA paper (Hu et al., 2021) calls this a low-rank decomposition. Rather than learning a full-size update, the model learns two much smaller matrices whose product approximates it.

The savings are large. On GPT-3 175B, LoRA reduced the number of trainable parameters by 10,000 times and cut GPU memory needed for training from 1.2TB to 350GB, according to the paper. Because the low-rank matrices can be merged back into the original weights after training, LoRA adds no inference latency, unlike adapter layers. Hu et al. also report a 25% training speedup on GPT-3 175B compared with full fine-tuning. LoRA matched or beat full fine-tuning on RoBERTa, DeBERTa, GPT-2 and GPT-3 benchmarks despite the far smaller parameter count.

LoRA vs QLoRA: why add quantization?

QLoRA, introduced by Dettmers et al. in 2023, combines LoRA with a quantized base model, one whose weights are stored in a lower-precision 4-bit format instead of 16-bit. Gradients are backpropagated through this frozen, 4-bit model into LoRA adapters, which remain in higher precision.

The effect on hardware requirements is large. The QLoRA paper reports that finetuning a 65-billion-parameter LLaMA model normally needs more than 780GB of GPU memory. QLoRA cut that to under 48GB, fitting on a single GPU, without degrading performance relative to a full 16-bit finetuning baseline, the authors say. QLoRA adds three techniques to get there: a 4-bit NormalFloat data type suited to the normal distribution of neural network weights; Double Quantization, which also quantizes the quantization constants to save about 0.37 bits per parameter; and Paged Optimizers, which use NVIDIA unified memory to absorb memory spikes during training.

One finding mattered for anyone applying LoRA broadly. The QLoRA authors found that default LoRA settings, which adapt only the query and value projection matrices, could not match full finetuning on larger models. Applying LoRA to all linear layers in the transformer block was necessary to recover full performance.

What is instruction tuning?

Instruction tuning fine-tunes a model on examples of instructions paired with the responses that follow them. The model learns to follow arbitrary user commands rather than just continue text. The QLoRA paper trained its Guanaco family of models this way, using eight instruction-following datasets including OASST1, Alpaca, and FLAN v2.

Dataset quality mattered more than quantity, the authors found. A 9,000-example dataset, OASST1, outperformed a 450,000-example subsample of FLAN v2 on chatbot performance, even though both targeted instruction following. The resulting Guanaco 65B model reached 99.3% of ChatGPT's score on the Vicuna benchmark, as judged by GPT-4, after 24 hours of finetuning on a single GPU, according to Dettmers et al. Strong performance on one benchmark did not imply strong performance on another: the paper notes that high scores on the MMLU benchmark did not predict high scores on the Vicuna chatbot benchmark, and vice versa.

What are the trade-offs and limits?

Full fine-tuning gives maximum control over the model's weights. But it costs the most in memory, storage, and risk of forgetting prior knowledge, as ULMFiT's experiments showed on smaller datasets. Adapter layers cut trainable parameters but add inference latency, since Hu et al. measured real slowdowns in low-batch, latency-sensitive settings.

LoRA removes that latency cost by merging back into the base weights. But Hu et al. note it is not straightforward to batch requests for different tasks together if the weights have already been merged. QLoRA extends LoRA's savings to much larger models by quantizing the base weights. Dettmers et al. caution, though, that most of the remaining memory cost in LoRA-style finetuning comes from activation gradients, not the adapter parameters themselves, so shrinking adapters further yields only minor savings.