179 questions
No questions match those filters.
← All questions
System DesignHard· Model Architecture· Scaling Laws
If you increase the FFN dimension (dff) of a 7B model, what must change to keep the total parameter count constant?
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansIncreasing dff from 11008 to 16384 adds approximately 66 million parameters per layer, totaling roughly 2.1 billion extra parameters for a 32-layer model. To maintain a 7B parameter budget, you must compensate by reducing either the number of layers or the dmodel. Reducing layers negatively impacts the aspect ratio, while reducing dmodel affects attention head dimensions and embedding sizes, demonstrating that FFN dimension is tightly coupled to all other architectural hyperparameters.