I hope the loss function is still fresh in your mind. The lower the loss value, the better the model is at predicting the next token.

From a purely theoretical standpoint, the “best” loss value is whatever your model can get that is closest to zero. This absolute best possible number is what we call the Global Minimum.

The lowest loss value the model actually reaches during training is called a Local Minimum (assuming the training process was allowed to run long enough to settle there). We’ve been using the apartment analogy a lot on this journey, but it’s time we got outside and touched some grass — at least in our minds (I don’t have the budget for a group camping trip!).

Imagine we’re out in the countryside at night. There are rolling hills and grass fields everywhere. We know the sea level is the lowest point possible, and we’d love to reach it… but we have a problem.

We’ve got an absolutely awful flashlight that only shines on where we just were. We can’t shine it forwards. So we can take a small step and immediately check if we’re now a little lower than before — but we have no idea whether the next step will take us even lower or start climbing again.

We start in a random spot on this dark landscape and need to find the lowest point we can. Here’s how it works in practice: You try moving a tiny bit in one direction. Only after the step can you check: am I lower than I was?

• If not, you step back and try another direction.
• If yes, you keep going that way.

You repeat this over and over until, no matter which direction you try, every possible next step takes you uphill.

In machine learning and AI terms, this whole process of always stepping in the direction that takes you lower is called Gradient Descent. (You’re literally descending the gradient of the hill — or in our case, the loss landscape.)

At that moment you’re standing in a little ditch, surrounded by hills in every direction. When the sun suddenly rises, you can see you’re not actually at sea level (the true Global Minimum). But you are at the lowest point you could reach by always walking downhill from your starting spot. That’s a Local Minimum.

If you were super lucky, your random starting point might have led you straight to sea level — in which case the Local Minimum and the Global Minimum would be the same place.

Depending on how many things the learning process is adjusting (all the weights across every attention head and neural-net layer), this theoretical landscape can have thousands or even millions of little ditches. But there is only ever one true Global Minimum.

This is why even the best-trained models can be really good at a lot of things, yet still have odd blind spots or surprising mistakes. They’ve found a very comfortable ditch — just not necessarily the absolute lowest point possible. That’s one of the reasons we keep doing extra rounds of training and human feedback later on.

That wraps up global and local minima. Now we can finally talk about how training an LLM actually works.