How to know an AI‑run process is working

Write down what a correct result is before the process starts.

Most arguments about whether an automated process works are really disagreements about what correct means. Settle that in writing, in terms the person who does the work would accept. Say the process drafts replies to supplier emails. A correct reply gives the right delivery date and does not promise anything the company has not agreed to.

Add the cases that count as wrong however polite the reply is. A short written standard lets two reviewers reach the same verdict on the same result, and that agreement is what makes any later measurement mean something.

Review a random sample every week.

Checking the results that happen to catch someone's eye tells you little, because those are the unusual ones. Pull a random selection each week and have a person judge each against the written standard. The size depends on volume. Twenty results a week is enough to notice a problem on a process that handles hundreds.

Keep the reviewer's notes. When a result is wrong, write down why in a sentence. After a month those sentences show which kind of mistake is common, and that is more useful than a single score.

Measure a few numbers that someone actually reads.

A short list is easier to keep up than a dashboard nobody opens. These four cover most processes:

• The share of sampled results judged correct.
• The share of cases the process passed to a person.
• The time from arrival to completion.
• How often a person changed a result before it went out.

Put the numbers next to the same figures from before the process was automated, if you have them. Without a baseline, a number can only be called good or bad by feel.

Treat every exception as information.

An exception is any case the process could not handle or handled wrongly. Each one needs somewhere to go, and that should be a named person with a queue they check daily. Once it is resolved, record what kind of case it was.

Exceptions then lead to a decision. Either change the instructions so the process handles that case next time, or add the case to the list of things that always go to a person. Some are rare enough to leave alone. The record tells you which fits, and it stops the same problem being fixed twice.

Agree in advance what makes you slow down.

Set thresholds before launch, when nobody is defending the project. For instance: if the share of correct results in the weekly sample falls below an agreed level, more of the work goes to a person until the cause is found. Deciding this in advance turns a bad week into a routine step.

Reduce checking slowly and only when the record supports it.

After several steady months you can sample less often or review a smaller share. Make that decision on the numbers, and make it for one process at a time. Keep some sampling permanently. The inputs change, suppliers change their wording, and a process that was right in spring can drift by autumn without anyone noticing.