179 questions
No questions match those filters.
← All questions
LLM FoundationsEasy· Model Architecture· FFN
Why is the FFN expansion ratio 4x commonly used instead...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansEmpirical evidence suggests that the loss landscape is relatively flat for FFN expansion ratios between 2x and 8x, meaning there is no significant quality signal distinguishing these values. The community has converged on 4x as a focal point, which provides a standard for comparing model architectures and conducting ablations across different research papers.