A colleague argues that as instruction tuning and RLHF keep improving, models will get better at ignoring injected instructions, so this risk shrinks on its own over time. Do you agree?

Instruction tuning and RLHF are designed to make models more responsive to instructions provided in the input. Because an indirect prompt injection attack relies on the model treating text found in the context as an imperative instruction, these training techniques inadvertently strengthen the model’s tendency to follow the attacker’s injected commands. Unless the training process explicitly incorporates a privilege boundary to distinguish between trusted system instructions and untrusted user-provided context, general capability gains will not resolve this vulnerability.