We’ve covered a lot so far, like tensors, vectors, tokens, embedding spaces, arrays, dimensions.. That is a lot of the foundation information that you need to know so that we can move forward and learn more things.

To that end, it’s time that we start to talk about token prediction and how this actually works within the embedding space and what a token prediction really is.

When you give an input to an LLM that is specifically for text generation (a simple chatbot), the system takes the string (the big chunk of typing you gave) and splits it up into tokens that it already knows. Once done, the system then converts all the tokens into the vectors for the locations of each token in the embedding space (this space is the vocabulary of all the tokens/words that the model knows).

Refreshing your view of a generic array (which is also a tensor to the token for the word in the vocabulary that is the embedding space), we might have:

[82, 12, -42, -2, 0.43, 21, 99, 102, 32, 0.6]

And remembering that this would be stacked together with a bunch of other vectors for all of the other tokens in the input, we end up with a lot of tensors, like this:

[23, 16, -41, -1, 0.92, 42, 63, 98, 28, 0.9] [82, 12, -42, -2, 0.43, 21, 99, 102, 32, 0.6] [55, 5, -63, -5, 0.04, 12, 71, 142, 37, 0.1] [21, 64, -4, -8, 0.10, 24, 69, 78, 31, 0.23] [83, 22, -63, -4, 0.52, 63, 53, 99, 24, 0.69] [12, 95, -21, 0, 0.27, 91, 77, 101, 12, 0.05]

Obviously, we would see many more than this in a true input, but we can conceptualise the system more easily with a smaller set of tensors.

So, let’s imagine the attention head and neural net layers stack running all the calculations with all the model weights that it had from the training (we will cover the actual training process in another post). The calculations are actioned through all the inference processes, and our model lands at the following tensor for the next predicted token:

[24, 78, -27, -9, 0.64, 87, 59, 112, 19, 0.8]

So, with this, the LLM system will now take that tensor and using it like directions on a very complicated map, will get to a point within the multi-dimensional embedding space and find one of two things.

Either there is a token at that exact point, or, that tensor “address” leads to a point within the embedding space that is empty (more likely to be the case than landing exactly on a token ID).

So how does our theoretical LLM work out what token/word to write? Well, to explain that, I need you to consider a regular-world, two dimensional map.

You are wanting to visit Starbucks, so you open your map and see that you are in fact, not currently stood in a Starbucks. So, you look on the map for Starbucks locations, and you find three that are relatively nearby. Logically, you figure out which one is closest, and then travel to it.

In machine learning, we might reflect this type of “nearest is good enough” approach with an ML algorithm called KNN, or K Nearest Neighbours.

In this algorithm, when used in regular machine learning, we might take a few of the nearest neighbours of the available data points (where K is the number of neighbours nearby that we are sampling). In the LLM world, we are essentially using K=1 for this sampling, and finding the token that is closest to the point in the embedding space that we got from our calculated tensor from the inference process.

There are a few other factors in the inference process, like temperature and randomisation (which is where the seeming creativity of an LLM comes from), but for the basic understanding, this explanation should suffice.

With our theoretical LLM though, we only have one token to write next; so how do we proceed to get the rest of the tokens to write out a (hopefully) coherent response to the user? Well, we take their input, and the token we already predicted, and we do the process again. And again. And again. At some point, the token that is predicted as the best one to use will be an “end of sequence” token, or EOS. This EOS token tells the system to stop running the tensors through the inference process, and that the message provided to the user seems to be complete.

Please Note: This Starbucks / nearest-neighbour predictive explanation is a teaching simplification to get the spatial intuition across. The actual mechanism uses per-token scores (logits) + softmax — but we’ll cover that properly in an future post.