The mathematical mechanics behind GPT-3 architecture

From words to vectors

To process human language, GPT-3 must first convert text into a format computers can manipulate. The system maintains a vocabulary of 50,257 tokens, where each token is assigned a unique numerical ID. For example, the word 'Aardvark' might be 0 and 'aaron' might be 1.

Each word is initially represented as a one-hot encoding vector. This is a list of 50,257 numbers where only the index corresponding to the word's ID is a 1, and every other position is a 0. Because these vectors are mostly zeros, the model uses an embedding function to project them into a more efficient space.

This embedding function is a neural network that maps each word to a vector of 12,288 numbers. These dimensions can be thought of as abstract properties that help the model store the meaning of a word. By multiplying the sequence of one-hot encodings by a learned weight matrix, the model creates a 2,048 x 12,288 matrix of embeddings.

Adding spatial context

At the embedding stage, the model has no way of knowing the order of words or their relative positions. It treats the sequence as a bag of words where each vector is processed independently. To fix this, the architecture incorporates positional encodings to give the model a sense of sequence.

The system takes the position of each token, which is a number between 0 and 2,047, and passes it through 12,288 sinusoidal functions. Each function has a different frequency, creating a unique set of values for every position in the 2,048-word sequence. These values are then added directly to the word embeddings.

The result is a matrix where each row contains both the semantic meaning of the word and its specific location in the sentence. This combined data allows the model to distinguish between different word orders. As dugas.ch reported, the exact reason why sinusoidal functions work is not fully clear, but they provide the necessary relative-position information for the model to function.

The attention mechanism

The core of the GPT architecture is the attention mechanism, which determines which parts of the input are most relevant to the current word. For every output, the model learns to focus on specific input tokens and decides how much weight to give them. This is the only part of the process where different words in the sequence actually interact with one another.

The model creates three different linear projections for the embeddings: queries, keys, and values. By multiplying the queries and keys together, the model produces an importance matrix. This matrix is normalized to show how much each token should influence every other token in the sequence. The values are then multiplied by this importance matrix to create a weighted mix of information.

GPT-3 specifically uses multi-head attention, repeating this process 96 times with different weights. This allows the model to attend to different types of relationships simultaneously across the text. The results from all 96 heads are concatenated and processed through a final linear projection to produce a refined output for each position.

Feed-forward and output

After the attention layers, the data passes through a feed-forward block. This is a standard multi-layer perceptron that consists of two linear projections with a ReLU activation function in between. While the input and output shapes remain the same, the hidden layer expands the size of the data to 4 times the embedding dimension.

To ensure the model remains stable during training, the architecture uses residual connections and layer normalization. The output of the attention and feed-forward blocks is added back to the original input of that block. This structure, common in deep learning, helps information flow more effectively through the 96 layers of the network.

Once the data has passed through all layers, the model produces a final 2,048 x 12,288 matrix. To turn these numbers back into human language, the model reverses the initial embedding mapping. It applies a softmax function to the results, which provides a probability for each of the 50,257 words in the vocabulary.

Generating the next word

While the model technically produces a guess for every position in the 2,048-word sequence, we usually only care about the very last one. When generating a story or a chat response, the model looks at the final probability distribution to pick the next word. This creates a loop where the new word is added to the sequence, and the process repeats to find the following word.

Users can control this selection process using a parameter called top-k. This limits the model's choices to the k most likely words predicted by the probability distribution. For instance, if top-k is set to 1, the model will always choose the single most likely word to follow the current sequence. This simple cycle of prediction and addition is what allows the model to generate long, coherent strings of text.

By combining these matrix multiplications and learned weights, the architecture creates a sophisticated system for pattern recognition. It doesn't 'know' facts in the human sense; instead, it calculates the most statistically likely next token based on the massive amount of data it was trained on. This mathematical framework is what powers the conversational abilities of modern AI.