A genuinely relevant chunk lands at position 11 of a 20...

When you cannot retrain the model, you can use inference-time interventions to adjust how the model weights specific positions in the context. Attention calibration and MS-PoE (Mean-Shifted Position-of-Interest Encoding) are techniques that manipulate the attention mechanism directly. These methods allow you to force the model to attend more heavily to specific chunks (like the one at position 11) by adjusting the attention logits or positional encodings, effectively bypassing the model’s inherent positional biases without changing its weights.