What happens when you run a RoPE model trained with a 4...

When a RoPE-based model is extended beyond its trained context length, the model encounters rotation angles that were not present during training. This results in a gradual rise in perplexity as the position index increases. The degradation is not uniform; slow-frequency pairs (large k) are typically the first to degrade because they have the least coverage in the training distribution. This can be mitigated using techniques like NTK-aware base scaling or continued pretraining.