What is the difference between self-attention and multi-head attention?

Self-attention is the fundamental mechanism where each token computes relevance scores against all other tokens using Query, Key, and Value projections, producing a contextually informed representation. Multi-head attention runs this mechanism h times in parallel with different learned projection matrices, allowing the model to capture multiple types of relationships simultaneously — for example, one head might focus on syntactic dependencies while another focuses on semantic similarity. The outputs of all heads are concatenated and projected through a learned output matrix. Multi-head attention is strictly more expressive than single-head attention with the same total parameter count.

Why is the scaling factor √d_k used in the attention formula?

The scaling by √d_k (the square root of the key dimension) prevents the dot products from becoming very large when the embedding dimension is large. Without this scaling, the dot products Q K^T grow proportionally with d_k, pushing the softmax function into regions where it produces very sharp distributions — one token receives nearly all the attention weight, and gradients become extremely small. By dividing by √d_k, the attention scores are kept in a range where the softmax produces useful probability distributions and gradients flow well during training. This is especially important for large models where d_k may be 64 to 128 or higher.

What is causal masking and why is it needed in decoder-only LLMs?

Causal masking prevents each token in a sequence from attending to future tokens during training. In a decoder-only model trained on next-token prediction, if token at position i could attend to token at position i+1, the model would simply copy the answer rather than learn to predict it — the training objective would be trivially solvable. The causal mask is implemented by setting scores to −∞ (which becomes 0 after softmax) for all positions j > i before the softmax operation. At inference time, causal masking is not explicitly needed because tokens are generated one at a time and future tokens do not yet exist, but it must be applied during training on full sequences.

How does Rotary Positional Embedding (RoPE) differ from sinusoidal positional encoding?

Sinusoidal positional encoding adds a fixed position-dependent vector to the token embedding at the input to the Transformer — position is encoded as an absolute value. RoPE instead applies a position-dependent rotation to the Query and Key vectors inside each attention head before computing dot products. The critical advantage is that the dot product of query at position m with key at position n naturally encodes the relative distance (m − n) rather than the absolute positions m and n. This relative encoding provides better generalization to sequence lengths longer than those seen during training (context length extrapolation) and consistently improves downstream task performance. RoPE is now the standard positional encoding in most open-source LLMs including LLaMA 2 and 3, Mistral, and Qwen.

What is Byte Pair Encoding (BPE) and how does it affect LLM cost?

BPE is a subword tokenization algorithm that builds its vocabulary by iteratively merging the most frequent adjacent character pairs in a training corpus. Starting from individual characters, it adds common subwords like "ing", "tion", "un" and frequent whole words, producing a vocabulary of typically 32,000–100,000 tokens that balances vocabulary size with sequence length efficiency. BPE directly affects LLM cost because API pricing is per token: a typical English prompt costs 1.3 tokens per word, so a 1,000-word document costs about 1,300 tokens. Code, non-English text, and mathematical notation typically tokenize less efficiently, increasing cost. Understanding tokenization helps engineers optimize prompts and control API costs.

AI & Automation·10 min read·June 19, 2026

🧠 Transformer Architecture Deep Dive: How Self-Attention, Positional Encoding, and Tokenization Work

A thorough technical explanation of the Transformer architecture behind ChatGPT, Claude, and other large language models — covering the encoder-decoder design, multi-head self-attention with Q/K/V matrices, scaled dot-product attention, sinusoidal and RoPE positional encoding, BPE tokenization, and the key differences between GPT and BERT architectures.

📖AI Roadmap for Electrical Engineers (Full Access)

View →

What Is a Transformer and Why Did It Change AI?

The Transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al., replaced recurrent neural networks (RNNs) as the dominant approach for sequence modeling in natural language processing. RNNs processed tokens sequentially — one at a time — making them slow to train and prone to the vanishing gradient problem that degraded long-range dependencies. Transformers process all tokens in parallel using a mechanism called self-attention, which allows every token to directly attend to every other token in the sequence regardless of distance. This change unlocked the ability to train much larger models on much more data, and ultimately produced the large language models (LLMs) that power ChatGPT, Claude, Gemini, and LLaMA.

Topics covered

transformer architecture explainedself-attention mechanismmulti-head attentionpositional encoding transformersBPE tokenizationGPT vs BERT architecturescaled dot-product attentionquery key value attentiondecoder-only transformerrotary positional embeddings RoPElayer normalization transformerfeed-forward network LLMWordPiece tokenizationtransformer encoder decoderhow ChatGPT workslarge language model architectureattention is all you needcausal masking LLM