179 questions
No questions match those filters.
Why did RoPE (rotary position embeddings) win out over learned absolute position embeddings?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansSelf-attention has no inherent sense of order — it’s permutation- invariant unless you inject position information. Learned absolute position embeddings do this by giving every position index its own learned vector, which works fine up to the longest sequence length seen in training, but has literally nothing to say about position 8,193 if training only ever saw sequences up to 8,192 tokens.
RoPE instead rotates the query and key vectors by an angle that’s a function of their position, so that when you compute the dot product between a query and a key, what falls out is a function of their relative distance, not their absolute positions. That relative framing generalizes far more gracefully to longer contexts than anything seen in training — it’s one of the main reasons long-context extension techniques exist at all, since you’re extrapolating a continuous rotation rather than looking up an embedding that was never learned.