Accuracy on the original set is 60.0% and on the permut...

Aggregate accuracy is a poor measure of robustness because it obscures individual instance-level changes. If ‘p’ is the rate of correct-to-wrong flips and ‘q’ is the rate of wrong-to-correct flips, the accuracy delta is q - p, while the instability is p + q. A 0.2% delta is consistent with high levels of instability. You should instead measure Variation of Prediction (VP) on a permutation of the existing evaluation set to quantify true model invariance.