In this blog post, Giles continues his journey through understanding deep learning concepts by delving into self-attention mechanisms in neural networks specifically for language models (LLMs) from scratch. He has recently overcome a challenging hurdle involving matrix manipulations and trainable weights related to self-attention. With that obstacle cleared, he moves forward to explore two more topics: causal self-attention and multi-head attention.
Causal self-attention mimics human reading behavior by not considering future tokens while processing a given one in LLMs. Multi-head attention is less complex than initially perceived but still crucial for improving model performance through parallelization of information across different spaces or “heads.”
The author notes how the names of matrices (query, key, and value) hint at their roles inspired by database systems but acknowledges that it may not be immediately apparent. He expresses curiosity about what batches do to all these matrix operations since even with a single input sequence in LLMs, full matrices are used already.
Giles also mentions the need for higher-order tensors when dealing with multiple input sequences simultaneously due to increasing complexity beyond simple scalar-, vector-, and matrix-based calculations. He concludes by inviting readers’ thoughts, questions, or suggestions while looking forward to exploring these topics further in subsequent posts.
In summary (no pun intended), Giles has conquered complex self-attention concepts related to LLMs’ neural network architecture using matrices with trainable weights and prepares for tackling causal self-attention and multi-head attention next. He also anticipates discussing batch processing implications in future posts while highlighting the relevance of GPUs due to their efficiency in handling matrix operations common across various applications including deep learning models like LLMs.
—
1. It’s worth noting that this is absolute position embeddings — there are also relative ones, but they’re not covered in the book. [↩](#fnref-1 “Return to footnote 1 in the text.”)
2. This, of course, is one of the reasons why GPUs – which were built to accelerate 3D graphics in games – are so useful for neural networks. They were designed to be super-efficient at matrix multiplications so that game developers could easily manipulate and transform objects in 3D and 2D space, but their efficiency is a general thing – it’s not tied just those matrix multiplications needed for graphics. [↩](#fnref-2 “Return to footnote 2 in the text.”)
3. This feels like something that would be best understood by trying some training runs with and without the scaling and seeing what happens – it’s an engineering fix rather than something mathematically obvious. [↩](#fnref-3 “Go back to footnote 3 in the text.”)
Complete Article after the Jump: Here!