179 questions
No questions match those filters.
← All questions
System DesignVery Hard· Hybrid Architecture· Inference Optimization· Layer Composition
How would you design the layer composition for a 70B hy...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansFor a 70B hybrid model with 32k context, a recommended design uses 24 Mamba blocks and 8 full-attention layers. The attention layers should be placed at positions 4, 8, 16, 20, 24, and 28, concentrating them in the second half of the stack where long-range context is most critical. This composition balances the efficiency of Mamba states with the retrieval capabilities of full attention, keeping the total KV cache and state memory usage under 20GB.