A stub. I will keep editing this one.
Most of the argument about AI agents is about what they can do. I am more interested in what they should decline to do, and whether they can tell the difference.
Here is the situation I keep coming back to. An agent is about to act on a record. Maybe it is merging two customers. Maybe it is approving something. The agent is confident. The question nobody asks is: confident about what, exactly?
There are two different things being confused.
Confidence in the answer. The model says 0.98. Fine.
Confidence that it is answering the right question about the right thing. Whether the entity in front of it is the entity it thinks it is.
The second one almost never gets measured, and it is the one that hurts. A model can be beautifully calibrated about a person who is not the person you meant.
What refusal actually requires
You cannot refuse without three things.
A notion of how sure you are, expressed as something other than a vibe. A threshold somebody chose on purpose and wrote down. And somewhere for the uncertain cases to go, which usually means a human, which usually means someone has to pay for that human.
That last one is why abstention is rare. It is not a technical problem. It costs money, and the cost is visible while the harm from a bad merge is not.
Things to work out
- What does an agent need to carry with it so that a refusal is auditable later
- Is there a useful difference between “I am unsure” and “I am sure this is out of scope”
- Who is accountable when a system correctly refuses and someone overrides it