We handle router uncertainty by escalating to the expensive path. Isn't that enough?

A single scalar for confidence conflates two different things: distance from the decision boundary and distance from the training data. An out-of-distribution query can be far from the training data while being positioned far from the decision boundary, meaning it will be ‘confidently wrong’ and bypass your uncertainty gate. The fix is to use a method that fails independently of the classifier, such as k-NN distance or centroid distance on the input, thresholded at a training percentile.