Given a compute budget of 1024 FLOPs, how do you choose...

To determine the optimal model size and token count, start with the compute formula C = 6ND. Conduct an isoFLOP sweep to identify the point where performance is maximized for your specific budget. While the training-optimal allocation is N ≈ √C/120 and D = 20N, it is critical to distinguish between training-optimal and inference-optimal objectives. Inference-optimal models often favor smaller parameter counts to reduce serving costs, so you should always ask the interviewer which metric is the priority.