← All questions
EvaluationHard· Interpretability· Causal Tracing

An engineer says: we located this model’s induction heads, and on this answer the top one attends to chunk 4, so the answer is grounded in chunk 4. Do you accept that?

The attention weight is not exclusive evidence of use. Because the final output is a margin over 1,024 head terms plus the direct path, one well-behaved addend does not determine the final sign. The definitive experiment is to remove the chunk and re-run the model to see if the output changes, as interpretability claims must be validated through causal intervention.

Code playground (Python)