“The AI said it was 95% confident” is not a safe policy for deciding whether work can go out without review.
A model can sound certain, assign itself a high score, and still produce an incomplete, unsupported, or simply wrong answer. The opposite is also true: a useful output may include cautious language because the model was trained to hedge. Neither polished tone nor self-reported confidence tells you enough about whether the output is ready to act on.
The practical problem is not whether to use human review. It is how to use it without turning every workflow into a queue of meaningless “Approve” clicks.
The answer is a routing matrix built on two separate variables: task impact and measured uncertainty. Impact sets the maximum autonomy a task can have. Uncertainty determines whether a lower-impact item can proceed automatically, go into a sampled audit queue, or stop for review.
Why one confidence threshold fails
A universal rule such as “auto-send anything above 90% confidence” creates false safety. It assumes that the score means the same thing across tasks, inputs, models, and consequences. It does not.
More importantly, confidence is not impact. An incorrect internal meeting-note summary is annoying and usually reversible. An incorrect client quote, payment instruction, account-permission change, or legal claim can cost money, damage trust, expose private information, or be hard to undo.
Generative AI also has a specific failure mode: it can present false content with great confidence. That is why a model’s wording, token probabilities, or self-assessment should be treated as one weak signal, not the decision-maker.
Instead, ask two questions for every workflow:
- If this is wrong, what happens?
- What evidence says this specific output may be wrong, incomplete, or outside normal conditions?
The first question measures impact. The second measures uncertainty. Keep them separate.
Axis one: set an autonomy ceiling by impact
Impact is the potential blast radius of a bad action. Score the action itself, not just the text the AI generated. A draft becomes more consequential when it is sent, published, used to change a record, or triggers a real-world action.
Low impact
Low-impact work is reversible, internal, and unlikely to harm a person or materially affect a business if wrong. Examples include formatting internal notes, tagging non-sensitive documents, creating a first-pass content outline, or categorising a known set of files.
These tasks can be eligible for automatic action when their checks pass.
Medium impact
Medium-impact work is externally visible or operationally important, but can usually be corrected without serious harm. Examples include product-description drafts, routine support replies within a narrow approved policy, or CRM enrichment that does not overwrite important fields.
These tasks may proceed with sampled audits when uncertainty is low and the workflow has a proven error record.
High impact
High-impact work affects money, rights, privacy, safety, reputation, compliance, or an irreversible decision. Treat client-facing prices, contract language, invoices, payments, deletions, permission changes, medical or financial guidance, and consequential decisions about people as high impact by default.
High-impact work requires qualified human approval or should not be automated. A low uncertainty score does not remove that ceiling. It can help the reviewer work faster, but it does not turn approval into an optional checkbox.
Axis two: measure uncertainty from evidence
Do not ask the model, “How confident are you?” and route based on its reply. Build a task-specific evidence score from things your workflow can observe and test.
Your score does not need to be mathematically sophisticated on day one. Start with a checklist. Give each signal a pass, fail, or flag status, then define what combination sends work to review.
- Source support: Is every material claim supported by the approved input or retrieved material?
- Coverage: Did the retrieval step find enough relevant information, or are key fields and required sections missing?
- Contradictions: Do supplied records, retrieved materials, or independent validation steps disagree?
- Rule checks: Did the output meet required schema, formatting, price, policy, or business-rule constraints?
- Validator disagreement: Do independent runs or purpose-built checks reach different conclusions?
- Novelty flags: Is this input materially different from the examples and cases the workflow normally handles?
- Task history: Has this type of task produced a higher reviewer-caught error rate than usual?
A failed hard rule should usually override everything else. For example, an incomplete required field, an unverified price, or missing evidence for a claim is not a reason to lower confidence slightly. It is a reason to hold the item.
If you are mapping a process from scratch, the SOP-to-Automation Mapper can help identify the handoffs, checks, and human decision points before you build the automation.
The 3-by-3 AI routing matrix
Use impact to define the row. Use uncertainty to define the column. Then assign a plain-language action.
Low impact
- Low uncertainty: Auto-act and log the result.
- Medium uncertainty: Auto-act only if it enters a spot-check queue.
- High uncertainty: Hold for human review.
Medium impact
- Low uncertainty: Use sampled audits, with a clearly defined audit rate.
- Medium uncertainty: Hold for review, or allow a tightly bounded action with rapid rollback.
- High uncertainty: Mandatory human review before action.
High impact
- Low uncertainty: Qualified human approval required.
- Medium uncertainty: Qualified human approval required, with the flagged evidence visible.
- High uncertainty: Hold, escalate, or decline to automate the action.
This is deliberately conservative at the top of the matrix. The point is not to make every system look autonomous. The point is to give routine work room to move while preserving real control where errors matter most.
Three examples you can apply this week
1. Internal meeting notes
An AI turns a recording into a standard meeting-note template. If the workflow has passed privacy checks, uses a fixed format, and finds all required sections, it is a low-impact, low-uncertainty task. Auto-create the draft in the internal workspace and log it. Sample a small portion each week to make sure quality has not drifted.
2. Product descriptions
An AI drafts descriptions from approved product data. This is often medium impact because the copy is public and errors can create customer confusion. Check that features, dimensions, price references, and claims match approved data. When those checks pass, publish through sampled audits. If source details are missing or conflicting, hold the draft for review.
3. A client-facing price quote
Even if the AI has retrieved the right rate card and passed every validation step, this is high impact. A person with authority should confirm the scope, price, terms, and customer context before it is sent. The AI can assemble the quote and flag exceptions; it should not have final authority.
Use the AI Automation Opportunity Finder to rank candidate workflows, then apply this matrix before selecting what to automate first.
Calibrate thresholds against real reviewer outcomes
“Calibration” means checking whether your uncertainty bands match what actually happens. A low-uncertainty item should have a lower reviewer-caught error rate than a medium- or high-uncertainty item. If it does not, your signals or routing rules need work.
Start with a reviewed pilot set for each task type. Run representative inputs through the workflow, record the signals that fired, and have a qualified reviewer label material errors. Then compare error rates by routing band.
- Choose one narrow task with a stable input format.
- Define hard-stop conditions and evidence signals.
- Review an initial batch of outputs.
- Compare actual errors with the assigned uncertainty band.
- Set your audit rate and review threshold based on your tolerated error rate and review capacity.
- Repeat whenever the model, prompt, tools, data source, or task context changes.
Do not borrow a threshold from another team, model, or use case. A score is only useful if it has been tested against outcomes in your own workflow.
Make human review meaningful
A person clicking “Approve” after scanning a long AI output is not reliable oversight. Reviewers need enough context to challenge the system and the authority to stop it.
For each review item, show:
- the original request and relevant input data;
- the proposed output or action;
- the supporting evidence used by the workflow;
- which checks passed, failed, or were unavailable;
- the triggered uncertainty flags; and
- clear options to edit, reject, override, escalate, or stop the workflow.
Reserve mandatory approval for consequential gates. Routine, repetitive prompts create approval fatigue, which encourages people to approve without thinking. Bounded autonomy and sampled audits are often stronger controls for low-risk work because they keep attention available for decisions that deserve it.
Turn the matrix into an operating loop
Your first routing policy will not be perfect. Treat it as a working control, not a one-time configuration.
For every item, log the impact band, uncertainty signals, route selected, action taken, reviewer outcome, override, and any incident. Review the data on a regular cadence. Tighten routing after a material failure. Reassess when you add new tools, change models, expand the task, or handle a new customer context.
The goal is not to prove that AI is confident. It is to build a workflow that earns the right to automate specific work, observes when conditions change, and knows when to stop.
For a structured implementation brief, use the AI Project Scope Generator. If you need help designing the workflow, checks, and review gates around a real business process, Work With Parker.