179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Model Architecture· Attention· MLP
Walk me through the forward pass of a 7B decoder-only m...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansA 7B decoder-only model’s forward pass involves specific projections within the attention and MLP layers. The attention block utilizes four projection matrices, while the SwiGLU MLP utilizes three. Quantitatively, this results in approximately 67M parameters for the attention block and 135M for the MLP, explaining why the MLP is roughly twice as large.