The One Number Everyone Asks For First
The first question a mortgage operations leader asks about an AI agent is what percentage of the work it does without a human. It is a fair question and it is the wrong one to lead with, because the automation rate is the single metric that can climb while the deployment gets worse. You raise it by lowering the bar at which the agent stops and asks for help, and lowering that bar is also how a confident wrong answer starts flowing into a loan file unreviewed. A number that improves when you weaken the control is not a number you manage toward. It is a number you watch, with three others standing guard around it.
I build the measurement layer for our mortgage agents, and the thing I spend the most time explaining to a new lender is why I will not optimize the automation rate directly. The rate is a byproduct of getting the harder things right. Lead with it and you end up tuning the one dial that trades safety for throughput, which is the trade a regulated lender can least afford to make quietly.
Automation Rate and Why It Is a Trap on Its Own
Start with a definition, because the number gets quoted without one. The automation rate, or the straight-through-processing rate, is the share of items the agent completed to the end without routing to a person. On an income desk it is the fraction of files where the qualifying figure was produced and cleared without an underwriter touching it. On a conditions desk it is the fraction of conditions cleared on the evidence alone.
Here is the mechanism that makes it dangerous. Every agent that produces a decision also produces a confidence in that decision, and there is a threshold, the confidence floor, below which the agent stops and hands the item to a human as a named exception. The automation rate is almost entirely a function of where that floor sits. Drop the floor and more items clear automatically, so the rate goes up. The items you just let through are, by definition, the ones the agent was least sure about. So the honest reading of a rising automation rate is ambiguous: either the agent genuinely got better, or someone moved the floor and the extra throughput is the agent's own uncertainty, now unreviewed. The number alone cannot tell you which, and a vendor quoting it to you has no incentive to say.
That is why I treat the automation rate as an output of the system rather than a target for it. The target is the floor being set correctly for the cost of each decision being wrong, and the rate is whatever falls out of that. A lender who manages the rate directly is managing the one variable that improves by removing the safeguard.
The Exception Rate Is the Number That Tells the Truth
The measurement I actually watch is the inverse of the one everyone asks for, and it carries far more information. The exception rate is the share of items the agent stopped on and routed to a person, and the useful part is not its size but its composition. Two deployments can both run a twelve percent exception rate and be in completely different health, and the difference is what is in the twelve percent.
A healthy exception rate is concentrated in genuinely hard files: the self-employed borrower with three income sources, the bank statement with a layout the reader has not seen, the condition that turns on a document that does not exist yet. An unhealthy exception rate of the same size is spread evenly across ordinary files, which means the agent is uncertain about work it should handle cleanly, and that is a signal of drift or a mismatch between what the agent was validated on and what the lender is actually sending it. So I slice the exception rate by file characteristic and by reason, and I read the distribution, not the headline. A flat exception rate that suddenly develops a spike in one document type is the earliest warning I get that something upstream changed, an employer that switched payroll vendors, an investor overlay that moved, and it shows up here before it shows up in a defect.
The exception rate also has an honest floor that it should never be driven below, because some decisions in a mortgage are supposed to reach a human no matter how confident a machine is. The ability-to-repay determination is the creditor's, and the adverse-action reason is a consumer-protection obligation under ECOA. An agent that drove its exception rate on those to zero would not be impressive. It would be out of compliance.
The Two That Decide Whether the Rate Is Earned
The automation rate is only as good as the accuracy of the work it let through, so the two measurements that give it meaning are both about the items the agent did not escalate.
The first is the confident-error rate, which is the share of automatically cleared items that were wrong. This is the measurement that matters most and the one a throughput dashboard never shows, because a confident error does not announce itself. The agent cleared it, nobody looked, and the first time anyone learns it was wrong is a QC finding, a repurchase demand, or a borrower dispute. We measure it the only way it can be measured, with a human re-review sample layered on top of the full-population automated work, specifically to estimate how often the agent was confidently wrong on the items it handled alone. That sample is not the production process. It is the control that proves the production process is safe, and it is the same posture the pre-funding full-population QC program takes when it keeps a human re-review sample running over the top of the machine.
The second is the human-override rate, the share of the agent's escalated recommendations that the human reverses when they look. This one reads in two directions, which is what makes it useful. A very high override rate means the agent is escalating with bad recommendations and wasting the reviewer's attention. A very low override rate over a long period is its own warning, because it usually means the humans have stopped actually reviewing and are rubber-stamping whatever the agent proposed, which quietly turns a human-in-the-loop control into a human-shaped decoration. I watch for both failure modes, because an override rate that is too low is a governance problem wearing the costume of a good result.
Why These Are Controls, Not Dashboard Tiles
The reason to define these four with this much care is that a supervised lender cannot treat an AI agent in the credit path as a productivity tool with a nice chart. It is a model, and SR 11-7, the model-risk guidance the Federal Reserve and the OCC issued, expects ongoing monitoring of a model's performance across its lifecycle, with benchmarks and thresholds set in advance and breaches that trigger action rather than a note in a file. That is exactly what these four measurements are for. The confident-error rate has a threshold that, when crossed, pulls a task back to human-only. The exception-rate distribution has an expected shape, and a departure from it opens an investigation. The guidance does not care about the automation rate. It cares that the lender knows when the model has left the envelope it was validated in, and the automation rate alone cannot tell you that.
The NIST AI Risk Management Framework gives the same discipline its vocabulary in the Measure function, which asks an organization to define metrics that capture the ways an AI system can fail and to track them against the context the system runs in, rather than reporting a single aggregate that flatters the deployment. A confident-error rate sliced by borrower and document type is a Measure-function metric. A headline automation rate is the aggregate the framework warns you not to mistake for an answer. We run both frames, SR 11-7 for the monitoring backbone an examiner expects and NIST AI RMF for the AI-specific failure surface, because the agent fails in ways a scorecard does not, and the metric set has to be able to see those failures.
The Number I Will Give You, With Its Caveat
Operations leaders want a figure, so here is how I give one honestly. An illustrative target we hold internally on mature tasks is a confident-error rate on automatically cleared items held in the low single digits of a percent or below, with the human re-review sample sized to detect a breach quickly, and the floor set per task so that the highest-consequence work, anything feeding the credit decision, carries the tightest threshold and the lowest tolerance for a confident error. The automation rate that results from those settings varies by task and by lender and by how clean the incoming files are, and I will not quote it as a headline because it is the output, not the promise. The promise is the confident-error rate and the re-review that proves it, because that is the number that protects the borrower and the lender, and it is the one a lender can hold us to.
The tasks do not all deserve the same settings, which is why a single global automation rate across an agent is a fiction to begin with. A document-checklist task and an income-calculation task carry different costs when they are wrong, so they get different floors, different confident-error tolerances, and different re-review intensities. Averaging them into one rate hides the only thing that matters, which is how the agent behaves on the work where being wrong is expensive.
The Honest Read
The automation rate is the number every mortgage AI conversation opens with and the one that tells you the least on its own, because it rises when you lower the confidence floor, and lowering the floor is how confident errors reach the loan unreviewed. The measurements that carry the real information are the exception rate and its composition, the confident-error rate on the work the agent cleared alone, and the human-override rate that reveals whether the reviewers are still reviewing. Those are the four we report, and the automation rate is what we let fall out of setting them correctly rather than the dial we turn.
At Sei we treat these as controls under the model-risk discipline SR 11-7 and the NIST framework describe, with thresholds set before go-live and breaches that act rather than annotate, because a mortgage agent in the credit path is a model and the metric that flatters it is not the metric that governs it. When a vendor leads with an automation rate and cannot tell you their confident-error rate or how they measure it, they have shown you the number that improves when the control gets weaker and hidden the one that improves when the agent actually gets better. Ask for the second one. It is the only one worth buying.
Ramkumar Venkataraman
CTO & Co-Founder