179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Mixture of Experts· Loss Functions
Explain the Switch Transformer auxiliary loss and why i...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansThe Switch Transformer auxiliary loss is defined as αN ∑ i fiPi, where fi is the hard token fraction (non-differentiable) and Pi is the soft router probability (differentiable). The gradient ∂L/∂Pi = αN fi flows through Pi to the router embeddings, using fi as a load-proportional multiplier. Because no gradient passes through the discrete top-K decision itself, the loss remains differentiable.