So far, we’ve covered some basic terms and explained them in ways that are more normal and what you see in the regular world. We’ve spoken about tokens being a representation of an apartment, or apartment and car park space (in LLMs, tokens are representations of words, or parts or a word structure).
 We covered arrays to get a vector (in multi-dimensional spaces for machine learning, vectors are the direction and angle in that space to get to the point you are referencing, in this case, a token).
 Also, we covered tensors - specifically Rank-1 tensors - which are a better way to describe the direction and angle, without needing to go from a specific point in that space, to get to a token.

For this post, we are going to cover Attention Heads, which are a core part of the transformer model approach that LLMs use. We’ll go into more detail later on, after we’ve covered the terminology, because I want you to understand the terms, so that we can be more direct when discussing the technical side of things.

LLMs are built on stacks of layers of neural networks, as has been the case for years. However in a paper in 2017 (“Attention Is All You Need”), a proposed approach was put forward to the machine learning community called Transformers. What this does, effectively, is allow the model to “look” at the entire input string at one time. This seems weird because, remember, we are converting inputs to token representations, so what does this actually mean?

 First of all, the news that might shock you (depending on your stance on LLMs and AI); LLMs are just math machines. They are taking the input you give (you might think that it is a question, but really it’s just the starting point), and then is going to predict the next token that would likely come next (with a few tweaks). This gives the impression of understanding, because it’s no longer just a chain of words/tokens (like in the first LLMs), instead we use transformers and, specifically, the attention head processing to allow the model to seemingly infer context.

For example, if I say the sentence “The cat chased the mouse because it was hungry”, and then I asked you what the word “it” was referring to, you would likely say the cat. The “it” is contextual in the sentence, because the cat was DOING the thing (the thing being the chasing). So how do we get a model to compute this understanding, especially when all the words vanish into tokens inside the model’s multidimensional space?

 We use the attention head. This is a very simple, but also complex, calculation where the attention head layer (for the ease of explaining) will use a mathematical process to compare each token in the sentence to the other tokens. This computation results in a score for relevance between each token.

In our example, we would expect to see a high relevance, or correlation, between “cat” and “it”. Allowing the mathematical processing to “understand” that the two words are linked.
 This process is first done during the learning stage of the model, where the embedding and model weights are built up, using the code. We will look into this code later on, with the python library PyTorch (the same library that ChatGPT and other LLMs are often coded with).

 For our apartment analogy, we might be able to ascertain that the number of bedrooms is connected to the floor area of the apartment; they are related values, so a much higher floor area without an increased number of bedrooms, would signify a more spacious apartment.
 Similarly, the car parking spaces would be likely tied to the number of bedrooms, and a single bedroom apartment would normally not have multiple car parking spaces included in the title.
 Of course, we might add another dimension to say how close the apartment was to the top floor of the building, and THEN we might expect that an apartment with zero floors above it, would have a higher number of car parking spaces, and more floor area, than the number of bedrooms would suggest; because it’s the penthouse.

The attention head would allow our model to start to “understand” the factors, given a high number of quality features in the tensor, to be able to more likely guess the price of a given apartment (assuming we had price as a feature, of course).