179 questions
No questions match those filters.
← All questions
LLM FoundationsHard· Model Architecture· FLOP Budgeting· Parameter Efficiency
If SwiGLU needs three weight matrices and GeLU only two...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansWhen transitioning from a GeLU-based MLP to a SwiGLU-based MLP, the architectural change involves moving from two weight matrices to three. To maintain parity in both parameter counts and FLOP budgets compared to a standard GeLU baseline where the hidden dimension is set to 4d, the SwiGLU hidden dimension must be reduced. By setting dff = 8d/3, the total number of operations (3 * d * dff for SwiGLU vs 2 * d * 4d for GeLU) is balanced, ensuring that the model maintains the same computational footprint.