I’ve briefly touched on the many layers within an LLM, and in the last post I spoke about the attention head/layer. This layer was, as I hope you can recall, responsible for looking at the different tokens within an input (in usual terms, a “prompt”, but it’s a bit deeper than that), and attempting to figure out what token might be more related to another token in the input.

After that layer is done, the outputted values (again, still just vectors/tensors as to the location of the tokens in the mathematical space) are passed along to the next type of processing in the architecture of the LLM; the neural network layer. We will be discussing this in a VERY high level here, I may go further into the topic later on, but whilst we are just covering the basics, there’s no need to get lost in the weeds.

To give this layer its proper name, we can say that it is a Multi-Layer Perceptron, or MLP. These are the things that you will often hear called neural networks, or how they are made up of “brain like neurones”.

So what are they? Simply put, they are a series of layers, where a bunch of multiplications occur. That’s pretty much it. Kind of…

The first layer in an MLP is, not surprisingly, called the “input layer” this is (in our case of being after an attention head) where all the values from the previous processing layer. At the end of the MLP we get the “output layer” (well thought out names, to be sure), where the values resulting from all the multiplications are outputted to wherever the architecture sends them.

In between the input layer, and the output layer, are a bunch of other layers where the number crunching occurs. We don’t get to see WHY the MLP picks these values during the training process of back-propagation, so we inventively called these layers, “hidden layers”.

Within each hidden layer is a node (or neuron) which has a fixed calculation that is performed. It takes in the value from its preceding layer, and multiplies it by its (the neuron’s) value, and outputs the result. We call these neuron values “weights”.

During the back-propagation training phase, these individual weights are adjusted to get ever closer to the answer that is expected (remembering that with the training phase, the architecture can check if it predicted the right value(s) that were outputted, and if it was wrong, this back-propagation adjustment flows backwards through the layers in a particular way, minutely adjusting each neuron’s weight a little, until the outputs become more and more accurate when tested with known data.

Granted, this is extremely high level, but I just needed you to understand that within the architecture of an LLM are a series of layers alternating between attention heads, and MLPs full of neuron weights, which are responsible for taking numbers, doing a sum, and passing on the output.

Within the final model (Sonnet, ChatGPT, Gemini, etc), what we are actually talking about (generally speaking) is a series of weights for all the layers within the chain of attention heads and MLPs. That’s basically it. In the data centres in which these models run, the individual GPU chips are assigned a layer of the model (again, this is simplified), and all that GPU does is run the calculations on the numbers passed to it. The chips do this, at 100% load, 24 hours a day. Until they burn out and get replaced.

It is extremely important that, at this moment, you understand that the magical inferring that you experience when using an LLM, when it seems to express thought, or intent, is just raw mathematics, ran on a graphics card, that is good at matrix multiplications.

That’s not to say that the systems and models aren’t impressive, they absolutely are, but they are not magical thinking machines. 
They do not understand what you are inputting, they are not (despite the names given to the models) “thinking”, nor are they “reasoning”. They are just processing numbers.

In the next post, we are going to start bringing this all together, so as to make it clear what a prompt actually is, and we’ll be going back to the apartment analogy we started with.