Welcome back! Hopefully, you’ve read the posts up to now, and looked at how we covered two-dimensional spaces (the floor area and price of the apartments) and how they are ‘directions’ to get to the ‘location’ of the token that represents the apartments in that 2D space.
Let’s move onto adding more dimensions to our space, with the analogy of one of the rooms in the apartment, with a single light fixture, dangling down on a cord. If we were to find the location of the light bulb/globe in the room, we could start in a corner of the room and measure along one wall, and then along another wall (on the two-dimensional space of the floor), until we were directly below the light bulb/globe. Then we could measure vertically to get to the location of the bulb itself.
In the same concept as this, we could add another feature (or dimension) to the apartments that we’ve been discussing.
Let’s add the number of bedrooms. We can add the number of bedrooms after the floor area as an integer, representing the number of bedrooms, with 0 being a studio apartment.
We will also add a value for whether there is a pool and gym in the building. The value will be 0 if there isn’t a pool and gym, and 1 if there is. This yes/no, or true/false, value is a boolean. This is going to be put after the bedroom integer value.
Let us also add another feature. This will be a boolean as well, like the pool and gym, and it’s going to be whether there is a balcony space. This will be put after the pool/gym value.
- 2/12 Smith Avenue, 83627, 59, 1, 0, 1, 630000
- 72/204 Roberts Street, 99263, 57, 0, 0, 1, 595000
- 220/92 Thompson Boulevard, 13946, 60, 1, 1, 0, 650000
- 1/42 Universe Road, 62240, 72, 2, 1, 1, 750000
OK. Now let’s take a look at our first property, and we’ll put the details into an array, like we’ve seen before.
2/12 Smith Avenue OR the token 83627: [59, 1, 0, 1, 630000]
This represents: [area, number of bedrooms, a pool/gym, a balcony, price]
Because the values are in the same order as the other apartments in our list of properties, we can use these features as a comparison. If we are putting these apartments into a multi-dimensional space, these five value features in the array are still a vector, or tensor, giving us a direction from a starting point (like the corner of the room to find the light bulb/globe), to locate the token for the apartment, in that space.
But how on earth do we have a five-dimensional space? Well, within LLMs and other mathematical systems, we refer to these things as spaces because it’s a way to imagine the data in a way that is organised. But, because it’s a mathematical space, and not a physical one, we are not bound by the need to physically represent it. So, if we imagine one direction of this space being the floor area, and another direction being the number of bedrooms, and another direction being if there is a pool/gym or not, we start to understand how the space is representing the properties that we are putting into the space.
The process of putting these properties into the space can be called ‘embedding’. We are fixing the token, representing the property, into the space. Its embedded location within the space is dependent upon its features (or dimensions), and the resulting location is able to be described by the rank-1 tensor that tells us how to get to the token in the space.
Furthermore, if you imagine that we try to navigate that space by using this tensor: [100, 1, 0, 1, 630000] we can see that whilst four of the dimensions describe 2/12 Smith Street (or more specifically its token of 83627), because one of the features (the area size) is wildly different, we would actually arrive at a place within the space that doesn’t hold the token for that property. We would be in a blank space.
We are able to look through some models that are available as open-source, and we know that tokens within the GPT3 model have ~12,000 dimensions. Therefore, each token has 12,000 different numbers within its tensor, to describe its location within the mathematical space of that model.
We aren’t going to expand our apartment analogy to that level, as this serves our purposes for the explanation for now.
So, to recap, you’ve now got vectors, tensors, multi-dimensional spaces that are more than our 3D world, and what embedding generally is. Next up, we are going into what a Transformer model is, and how it works.