How should an AI system handle uncertainty?

Separate the signals

A language model predicts a distribution over the next token given its context. A high probability for one token means the model strongly preferred that continuation at that position. It does not directly mean that the completed answer is factually correct, that cited evidence supports it, or that an external operation will succeed.

Several things called “confidence” need separate names:

• token probability: preference among possible next token IDs;
• sequence score: an aggregate over token probabilities, strongly affected by wording and length;
• semantic consistency: whether repeated samples converge on the same meaning;
• self-report: text such as “I am 90% confident,” generated through the same language mechanism;
• grader score: a separate instrument applying a rubric to visible evidence;
• empirical calibration: how often a defined prediction was correct among comparable cases assigned a given score;
• system confidence: an application decision combining evidence, dependencies, policy, and cost of error.

Fluency is not on the list. A polished answer can be unsupported, while a hesitant style can accompany a correct one. The harness should not infer reliability from tone.

Ask for the fact the policy does not contain

A fictional assistant retrieves an employee reimbursement policy, but the user asks about a contractor. The passage is clear; its scope does not answer this case.

“I'm 90% confident” supplies none of the missing evidence. State the concrete gap and a next step. These are authored policy decisions, not calibrated percentages or a claim that a model recognizes every unsupported answer.

Calibration is a population property

Suppose a classifier emits 0.7 for many comparable cases. It is calibrated at that level if roughly 70% of those cases are correct under the declared label and data distribution. Calibration does not say which individual case is correct, and it does not transfer automatically to a new language, user group, task, model revision, or traffic shift.

flowchart LR
  S[Raw signal] --> C[Calibration on held-out labels]
  C --> P[Estimated risk for a declared slice]
  P --> D{Policy threshold}
  D -->|low risk| A[Answer or act]
  D -->|unclear| V[Retrieve, verify, or ask]
  D -->|high risk| H[Abstain or escalate]

For fixed-label tasks, reliability diagrams compare assigned score bands with observed correctness. Proper scoring rules such as log loss or the Brier score reward useful probabilities and penalize confident errors. A single expected-calibration-error number can hide small severe slices, so keep the underlying counts and plots.

Choose a threshold without hiding its denominator

The experiment contains thirty authored cases: ten each with scores 0.60, 0.80, and 0.95. In the matched population, 6, 8, and 9 cases in those respective groups are correct. In the shifted population, the scores stay the same while the correct counts become 3, 4, and 5.

These scores represent an illustrative instrument's estimate of a defined correctness event, not token probabilities or an LLM's self-report. The labels are independent authored outcomes. The scenarios provide no evidence about a real model's calibration.

Raise the threshold and inspect accepted errors, coverage, and accuracy among accepted cases. Then switch population while keeping the same threshold. The reliability plot always uses all thirty cases, so filtering accepted cases cannot make the plotted population look better.

At threshold 0.95, the matched fixture accepts ten cases and gets nine right. The shifted fixture accepts the same score group but gets only five right. A high score's interpretation depends on the population and the quality of the scoring instrument.

At threshold 1, no case qualifies. Accuracy among accepted cases is then undefined: the denominator is zero. Reporting 100% would disguise a policy that answered nothing. Coverage and accepted errors must remain visible beside conditional accuracy.

The threshold chooses which cases to accept; it does not recalibrate their scores. The optional inspector calculates a Brier score and a count-weighted calibration gap for the three declared score groups on the whole fixture. Those population quantities remain fixed when only the threshold changes.

In a real system, estimate and validate calibration using appropriately separated labelled data and relevant slices. Thirty invented examples are enough to reveal the arithmetic and its boundaries, not to certify a deployment.

Agreement still needs independent evidence

Free-form generation is harder. Many token sequences express the same answer, and longer sequences naturally accumulate lower probability. Sampling several answers and grouping them by meaning can reveal semantic disagreement; research on semantic uncertainty shows this can detect some confabulations. It still does not detect every error—several samples can agree on the same false claim.

Locate what is uncertain

An application may face different uncertainties:

• the request is ambiguous or missing constraints;
• relevant knowledge is absent, stale, or outside the model;
• retrieved sources conflict or do not support the claim;
• a tool result is missing, malformed, or from an unknown state;
• the model produces several incompatible interpretations;
• the request differs from the calibrated evaluation population;
• a grader is unavailable or disagrees with deterministic evidence;
• the real-world effect is irreversible or has an unknown outcome.

These conditions call for different responses. Ask a clarifying question for ambiguity. Retrieve current evidence for missing knowledge. Re-run an idempotent read when a tool result is unavailable. Require approval for consequential effects. Escalate a high-risk domain decision. Preserve “unknown outcome” after an ambiguous write rather than retrying blindly.

Abstention belongs to the system contract

Abstention means deliberately not making a claim or effect under specified conditions. It can be a refusal, a request for missing information, a limited answer with named uncertainty, a handoff to a person, or a safe no-op. Define which form applies and what the user can do next.

Thresholds should reflect consequences. A slightly uncertain movie recommendation and an uncertain medication dose cannot share one policy. Compare at least three rates by slice: correct actions, harmful or incorrect actions, and abstentions or escalations. Increasing abstention can raise accuracy among answered cases while making the product useless; reducing it can improve completion while increasing severe false actions.

Evaluation metrics influence behaviour. A benchmark that scores only exact correct answers and treats abstention exactly like a wrong answer rewards guessing whenever there is any chance of success. Score confident errors, useful abstentions, and unnecessary refusals in a way that matches the actual decision cost.

Do not delegate the final decision to a prompt saying “answer only if confident.” The model can contribute a signal, but the harness owns evidence checks, calibrated thresholds, permissions, routing, and the allowed terminal states.

Communicate without false precision

Show users what matters: the evidence used, its date and scope, unresolved conflict, assumptions, actions taken, and a clear next step. A naked “87% confidence” suggests a universal frequency unless it was calibrated for that exact kind of claim and population.

Prefer statements such as “The retrieved policy does not cover contractors, so I need the employment type” or “The write timed out after submission; its outcome is unknown, so I will check the transaction before retrying.” These expose the missing evidence and safe response rather than imitating a feeling.

Confidence is not permission

Uncertainty is not one scalar stored inside a language model. Entropy is not the probability that a sentence is false. Repeated agreement is not independent verification. A model saying “I don't know” is not automatically calibrated, and a citation is not evidence until the cited material is retrieved and shown to support the claim.

Calibration is also not permanent certification. It depends on labels, scoring rules, slices, thresholds, and the system version. Change the model, prompt, tools, context construction, or population and the evidence must be checked again.