179 questions
No questions match those filters.
← All questions
Training & Fine-tuningMedium· Knowledge Distillation· Model Pruning
Why is distilling from a pruned initialization better t...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansDistilling from a pruned initialization is superior to training from scratch because the surviving weights in the pruned model are already the best-performing weights from the teacher’s full pre-training. Consequently, the student begins the training process near the target loss. This significantly reduces the required training effort, often requiring 6-8x fewer tokens to reach equivalent quality compared to a model initialized randomly.