Padding strategy matters when Fine tuning LLMs

A poor padding strategy can disrupt an LLM after fine-tuning, and the reasons are quite subtle. This is especially true since there is no pad_token in the Llama2 or Mistral tokenizers.

A neural network is a sequence of matrix multiplications. When working with text and variable sequence lengths, padding is required to normalize them to a common length to allow matrix representation.

This padding simply consists of filling the sequences with a special token until they reach the length of the longest sequence in the batch.

It turns out that most LLMs were pre-trained without padding. As decoders, they predict the next word based on previous words. It is then enough to split the training corpus into equal sequence lengths and train the model on it (even if sometimes the splitting is not very graceful). No need for padding, besides in many models like Mistral or Llama2, the padding token doesn't even exist in the tokenizer.

When fine-tuning these models on specific instructions, we provide the instructions and expected responses in a single block (we concatenate both). There is no reason for the sequences to be the same length. To process all of this in batches, padding is then required. Having no padding token, one way to get by is to use the eos (end of sequence) or unk (unknown) token instead.

Then, what do we do? Right or left padding?

• With eos, I'm personally not very comfortable doing left padding: the model is asked to stop the sequence even though it hasn't started. Doing right padding, why not.
• With <unk>, doing left or right padding should be fairly equivalent.

On the other hand for inference, you have in my opinion no choice but to do left padding. Simply because the decoder will rely on your last token to predict the next one; if it's padding, the quality of the response will be altered.

One last approach that seems quite elegant and effective for fine-tuning is to do as in initial training: no padding, concatenate the entire question-answer dataset and slice it into identical chunks of N tokens.

Curious to know your padding strategies.