n0rds3c.dev / notes / draft
Graceful failure when the model is confident and wrong
. Draft. Not finished; the argument is here, the polish is not.
A confidently wrong answer looks identical to a confidently right one. Same fluency, same formatting, same lack of hedging. Any interface that uses the model's tone as a proxy for reliability has already lost, because tone is the one thing the model controls perfectly.
So the design problem is not “how do we detect when it is wrong.” It is “what happens to the person when it is wrong and nobody detected it.” That reframing moves the work out of the model and into the workflow, where you have actual levers.
The levers I keep coming back to: show the sources next to the claim, not behind a click. Make the consequential step a separate action with a separate confirmation, so a wrong answer costs a reread rather than a rollback. Keep the human gate in front of anything that leaves the system, and do not let the model's certainty skip the queue. And write the whole exchange to the trail, so the wrong answer is a thing you can find, count, and learn from, instead of a rumor.
None of this makes the model more accurate. It makes the product survivable when it is not, which is the property people are actually buying.