← All questions
Training & Fine-tuningHard· SFT· Model Behavior· Context Utilization

The correct passage is in the context window and the mo...

When a model ignores provided context in favor of its internal memory, it indicates a failure in instruction-following behavior rather than a lack of knowledge. This issue is primarily addressed during the SFT phase of model development.

To correct this, the training dataset must be carefully curated. It should contain a mix of items where the gold document is provided and the model is trained to utilize it, balanced with a deliberate fraction of examples where the document is removed. This forces the model to learn the conditional dependency between the provided context and the output, rather than relying solely on pre-trained weights.

Code playground (Python)