179 questions
No questions match those filters.
← All questions
LLM FoundationsMedium· Model Architecture· FFN
How do you calculate the FFN dimension for a SwiGLU model with a dmodel of 8192?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansTo determine the FFN dimension for a SwiGLU model with dmodel=8192, multiply by 8/3 to get 21845.3. Rounding this to a multiple of 128 gives 21888. When finalizing this number, you must ensure that the dimension is divisible by your tensor-parallel degree to maintain efficient computation.