179 questions
No questions match those filters.
Your offline metrics improved but the online numbers di...
This is one of the questions in the full AI/ML interview bank. Pro unlocks all 1789 questions; Premium includes the same bank plus the highest daily Practice limit.
See plansBecause offline evaluation measures the model; online outcomes measure the whole system, including the humans using it. Those are different things, and a gap between them is common enough that it should always be an expected possibility, not a surprise.
The usual causes, roughly in order of how often I’ve seen each:
- Distribution shift — the offline evaluation set no longer resembles live traffic.
- The wrong metric — the number that improved isn’t actually the thing users care about.
- A trust or UX problem — the model got measurably more correct, but users don’t act on the output even when it’s right, because the interface doesn’t give them a reason to trust it.
The fix is to instrument human behavior directly — acceptance rate, override rate, edit rate, escalation rate, abandonment — because those measure trust, and trust is ultimately what determines whether the system survives contact with real users, independent of how good the model itself has gotten.