Welcome to the second post in my series of four posts that describe my experiences with training a 1.1-billion-parameter GPT model from scratch.

In my last post, I described the training of my base model (KjeldGPT). This was a large undertaking involving 8 days of training on a 30 GB corpus consisting of 30,000 English books from Gutenberg and 5,000,000 Wikipedia articles. I used a rented RTX PRO 6000 Blackwell GPU for training, so apart from the time spent, there was also a cost of close to $400 for compute.

After this training, I had a frankly pretty useless base model: It is fundamentally a token-predictor that can complete sentences. Hence I can give the model a string of words, and let the model generate a likely follow-up sentence called a completion.

If, for example, I write “The planet Saturn is” – the model will generate a completion like “orbiting a gas giant that has been ejected from its parent star. Its orbit and physical characteristics are closely modeled by the one of Jupiter, with which it shares both orbital plane and size.”

What the model does is generate a likely completion based on statistical patterns learned from the training data. The upside is that the model is capable of producing somewhat coherent English sentences with correct syntax. The downside is that the completions are a complete mess, not grounded in facts – and not really usable for anything other than experimentation. But this will change in this post once the model gets fine-tuned.

Introductory remarks

Before we dive into the process of fine-tuning, some introductory remarks are needed: First of all I need to define how I use the term fine-tuning in this post. And there is also a bit of a chicken-and-egg situation in relation to the next two posts I am writing, which we need to address:

Fine-tuning in the scope of this post is further training of the base model on a new corpus of text (the fine-tuning corpus) that adjusts the weights of the model. Hence the fine-tuning produces a new model by changing the existing base model. There are other types of fine-tuning, but they are not covered in this post.

Next, in the fine-tuning corpus, you will notice that I am providing context alongside a user question for fact-based questions. Hence, if I ask the model about the boiling point of water, I will actually add a passage from Wikipedia together with the user question, and simply train the model to pick out and paraphrase the correct answer. This approach of retrieving a relevant context and injecting it at inference time is called retrieval-augmented generation – or RAG – and is a widely used technique for making a model more capable. I will cover this in detail in my next blog post, but already at this stage the fine-tuning corpus needs to teach the model to use these contexts.

Finally, the effect of fine-tuning is very dependent on the fine-tuning corpus. In this post I will fine-tune the base model into a Q&A model that can answer simple fact-based questions. However, to adjust the fine-tuning corpus, I used a development/test suite that asked the fine-tuned model 426 fixed questions and measured the success rate of its answers. This allowed me to refine the fine-tuning corpus and measure the effect of changes to the corpus. In total I fine-tuned using 7 different corpora, but in this post I am primarily going to present the final version (v7). I will return to this process in more detail in the fourth and final post about testing.

With that settled, let us move on to fine-tuning ☺

From base models to instruction-following models

The base versions of early OpenAI GPT models were completion models of varying sizes, with GPT-1 having 117 million parameters and the largest GPT-2 model having 1.5 billion parameters

In many ways, my base model is comparable to GPT-1 and GPT-2. In terms of its corpus, my model is similar to GPT-1, which was trained on the BooksCorpus dataset. In parameter count, my model is comparable to GPT-2, but GPT-2 was trained on the WebText corpus, consisting of web content selected through Reddit links.

OpenAI subsequently continued to develop larger models, including GPT-3, whose largest published version had 175 billion parameters. Although fine-tuning was already an established technique, greater emphasis began to be placed on its role after Ouyang et al.’s 2022 paper, “Training Language Models to Follow Instructions with Human Feedback.” The paper showed that, in human evaluations, people preferred the 1.3-billion-parameter InstructGPT model to the 175-billion-parameter base GPT-3 model. A model with less than 1% as many parameters had been made more useful through a comparatively small but carefully prepared fine-tuning stage.

For InstructGPT, post-training was done using a three-stage process: supervised fine-tuning on human-written demonstrations, training a separate reward model using human preference rankings, and then applying reinforcement learning against that reward model. My own project uses only the first and simplest of these approaches: supervised fine-tuning. This is the process I refer to as fine-tuning throughout the rest of this post.

But generally, the InstructGPT paper showed that “bigger is not always better” and highlighted the importance of careful post-training and fine-tuning in the development of usable GPT models. Fine-tuning should therefore not be seen as merely “tweaking” a model. It is a fundamental part of building a useful GPT system and can have dramatic effects on the model’s behavior, instruction-following, reliability, and practical usefulness.

The fine-tuning corpus

My fine-tuning recipe was quite simple compared to the one used by InstructGPT: I trained my base model on a new fine-tuning corpus for a little over 2 hours. The fine-tuning corpus I used ended up consisting of 51,184 question-and-answer pairs, all with the same overall shape:

Context: The boiling point of a substance is the temperature at which the vapor pressure of a liquid equals the pressure surrounding the liquid and the liquid changes into a vapor. The boiling point of a liquid varies depending upon the surrounding environmental pressure [...]
Question: What is the boiling point of a substance defined as?
Answer: It is the temperature at which the vapor pressure of a liquid equals the pressure surrounding the liquid and the liquid changes into a vapor.
<|endoftext|>

As one can see, the fixed format is context, question and answer followed by an end-of-text marker. The context in the above example is actually longer, but I have shortened it for readability.

While the fine-tuning corpus follows this fixed format throughout, the pairs themselves vary in purpose. Some were questions for which the model had to pick out a fact from the context, and generate an answer based on it (open-book questions). But I also had questions without a context, where the model had to generate an answer without depending on a context (closed-book questions).

Finally, I had specialized pairs aimed at teaching the model some more specific capabilities, for example, to reject a wrong premise (like “Wasn’t the canton of Le Blanc reorganized in 2020 as part of national reforms?”, where the answer corrects the year to 2015) or pick out a specific fact (like a date) from a context that contained many different dates.

In general, the fine-tuning corpus was synthetically generated: I used Claude’s Sonnet model to generate the question/answer pairs based on contexts found in the training data.

Hence, the corpus I ended up with had the following distribution among the different categories: (with links to the files on my GitHub):

QA type	Purpose	Pairs	Share
Context QA	Simple open-book Q/A with a Wikipedia passage as context	30,121	58.8%
No-context QA	Simple closed-book Q/A without a context	6,371	12.4%
Context false-premise	Q/A based on a false premise about the context’s subject	3,014	5.9%
No-context false-premise	Q/A based on a false premise but without context	1,008	2.0%
Value discrimination	Open-book Q/A where the model had to pick a specific fact among many similar facts in the context	5,656	11.1%
Grounding reinforcement	Questions where the model had to use facts from the context over facts from base training	5,014	9.8%
Total		51,184	100%

Training on the fine-tuning corpus

What happened from a technical point of view during fine-tuning was fundamentally the same as during pre-training: The model was asked to generate a completion of the context/question/answer combination, and then the model’s prediction was compared against the actual corpus, with the resulting error used to adjust the model’s weights.

However, when calculating the loss, the targets for the context and question tokens were masked. The model could still attend to those tokens, but only prediction errors on the answer tokens and the trailing end-of-text marker contributed directly to the loss and drove the changes to the models weights. The goal was to teach the model that, when presented with a context and question in the format above, it should produce an answer followed by an end-of-text marker, without also training it to generate contexts or questions.

Also, while the fine-tuning script was fundamentally the same as the base training script, I used slightly different settings: The fine-tuning corpus contained 10,348,947 tokens, and training was done using 3,789 iterations, equivalent to roughly three epochs over the corpus. The learning rate (how much I adjusted the weights at each iteration) followed the same cosine schedule as the pre-training, but was set to a peak learning rate 20 times lower than in the pre-training.

Findings

In general – and this is where the fine-tuning differed dramatically from the pre-training – my fine-tuning process was more exploratory. This was due to two factors: First, I did not have the Kaplan and Chinchilla scaling-laws to lean on in terms of corpus size, but the whole process was also so much faster, so I had the resources to experiment a lot more with different fine-tuning recipes.

As mentioned, I fine-tuned using 7 different fine-tuning corpora (v1 to v7) and different settings for the learning rate. This was an iterative process in which I fine-tuned and tested the model on my suite of 426 fixed questions. I then looked at the failing questions, diagnosed the patterns, and adjusted the corpus.

The changes to the fine-tuning corpus were quite noticeable in the model’s ability to answer questions correctly, as shown in the following plot:

The exact changes made to the corpus and the training settings can be seen in the table below together with the test results. Please note that the 426 questions included 30 non-factual questions (like “How was your day?”) which has been excluded:

Version	Changes made	Correct
v1	10,026 questions, 2/3 based on Wikipedia + 1/3 based on Gutenberg	109/396
v2	Change corpus layout: Corpus replaced entirely: Wikipedia-only questions, 12,124 pairs	114/396
v3	Increase corpus size: Number of questions tripled to ~36k and peak LR lowered 2e-5→1e-5	128/396
v4	Add target questions: Grounding and false-premise questions added	132/396
v5	Data hygiene: Contexts fixed to clip at a complete sentence, instead of cutting mid-sentence/mid-word	151/396
v6	Data hygiene: IPA pronunciation guides stripped from contexts	153/396
v7	Add target questions: Value-discrimination questions added	155/396

In general the training resulted in the lowest val loss somewhere between iterations 2,000 and 3,500 depending on the corpus, after which the model started to overfit slightly, as can be seen in the plot from the v7 training.

But what I also noticed was that the model with the lowest val loss was not necessarily the best model when tested with my 426 test questions. This was an important finding, and for the last couple of rounds, where I was squeezing the last improvements out of the fine-tuning, I actually saved a copy of the model for every 100 iterations and subjected each saved model to the test questions.

This showed that the best model (measured by my test) was the v7 checkpoint at iteration 2,700 of the fine-tuning corpus, whereas the model with the lowest val loss on v7 was the checkpoint at iteration 3,450. As the tests measured the model’s question-answering abilities, they took precedence over the val loss — the model currently available on Hugging Face is therefore fine-tuned on the v7 corpus for 2,700 iterations.

Results

After the final finetuning, I had a model that had learned the Q&A format and could give correct answers in about 39% of test cases. That sounds low, but it’s actually ahead of other models of its size when subjected to my 426 test questions. That said, it’s not an apples-to-apples comparison: My fine-tuned model is the only model in the lineup below that gets to see a retrieved passage before answering, so some (if not all) of that lead comes from RAG (which I will cover in my next post) rather than the finetuning itself.

In general my fine-tuned model is able to answer simple questions with about 63% accuracy, but there are several question types of a more complex type that the model struggles with – which lowers the overall success rate:

• False-premise question: “What year did Einstein win the Nobel Prize for his theory of relativity?” is rarely rejected by the model. In this example the model answers 1921, where Einstein did win a Nobel Prize, but not for his theory of relativity.

• Exclusion questions: “Which of the following planets does NOT have a ring system: Uranus, Mars, or Saturn?”. In these cases the model often disregards the negation, and in the example answers Saturn.

• Discrimination question: “Who developed the polio vaccine, and in what year was it introduced?”. In these cases the model gets a context with a lot of different dates, and sometimes chooses the wrong date. In the example the model answers 1948, which was the year Jonas Edward Salk’s funded polio research project began, whereas the vaccine was first introduced in 1955.

• Compound questions: “Who wrote Frankenstein, and how old was the author when it was first published?” Here the model sometimes only answers the first part of the question (Mary Shelley), but ignores the second part.

All these question types have been targeted specifically in the fine-tuning corpus, and while some improvements have been made from v1 to v7, all of the above types of questions still have a low success rate in the tests.

There is also a specific case where I have decided not to train the model. If a context is not available in the collection of Wikipedia-based context used for retrieval-augmented generation, I could have chosen to train the model to reject the question.

However, in these cases I have trained the model to generate an answer based on its pre-training alone. This often results in some hilarious confabulations – even for fact-based questions whose answers were part of the pre-training corpus (e.g. available in one of the Gutenberg books used for training). These answers hint at the capacity limits of encoding factual knowledge through pre-training in a 1.1-billion-parameter model. However, I will return to this problem in the next post when introducing retrieval-augmented generation.

Finally, it is worth mentioning that I have not trained the model on a multi-turn conversational format. This was done to simplify the fine-tuning, but is definitely something I will try to add in the future.

However, my fine-tuning clearly shows that a relatively small amount of targeted post-training can teach a previously completion-only model to follow a stable Q&A format and use supplied evidence, although its factual reliability when handling of complex question forms remained limited.

As mentioned in my first post, you can try out the fine-tuned model on my Hugging Face profile.