What loss function do you use for distillation and why...

While cross-entropy against hard one-hot labels is standard for supervised learning, it discards valuable information about the teacher’s uncertainty. Forward KL divergence allows the student to learn from the teacher’s full probability distribution. This provides a much richer gradient signal, especially in early training stages, as the student learns from all high-probability tokens rather than just the single ground-truth token.