You are 20 days into a 30-day run and discover a GPU ha...

First, quantify the contamination window by comparing canary loss to its expected trajectory. If the divergence is minimal (within two standard deviations of noise), the model may be salvageable from the day-5 checkpoint. If the drift is significant, a 15-day rollback is necessary. Regardless of the path, you must isolate the affected hardware, verify the day-5 checkpoint using an independent forward pass on clean hardware, implement cross-replica gradient audits before resuming, and file a formal hardware incident report.