Who This Is For
Model risk managers, Chief Risk and Compliance Officers, and heads of underwriting or QC who are being asked to let an AI agent clear conditions or findings without a person reading every line, and who want to know what stands between the model's answer and the loan file.
Mortgage Already Runs on Maker-Checker
Nobody in mortgage trusts one person's read of a file. The processor assembles, the underwriter decides, pre-funding QC re-checks a sample, post-closing QC re-checks another, and the investor re-checks whatever it buys back. Each layer exists because a single competent reviewer still misses things, and the misses are expensive. Separation of duties is how the industry turned individual judgment into something an examiner can rely on.
Then AI arrived, and a lot of it skipped that lesson. The typical design is one model that reads the document, produces an answer, and attaches a confidence score it computed about itself. If the score is high, the answer goes into the LOS. That is a maker with no checker, grading its own work. It would not pass a QC audit if a person did it, and it should not pass one when a model does.
The verification layer in UMOS, which we call Triple Check, is our answer to that. It has three layers, each of which can stop the work on its own, and each of which is logged. This post is about how they are built and why they are built that way, because the design choices are where the actual protection comes from.
Layer One: The Maker Reasons Against the Rulebook, and Cites or Does Not Act
The first layer is the agent doing the work. When the Conditions Manager decides whether a condition is satisfied, it works from a specific rule from the Rulebook (a Selling Guide section, an FHA Handbook paragraph, an investor overlay, the lender's own SOP) applied to specific evidence in the Loan Graph, rather than from a model's general memory of mortgage lending, and its output has to name both: the rule, by citation, and the evidence, by document and page.
That requirement is a check in itself, because a maker that has to cite cannot bluff. If the agent cannot point to the rule it is applying, or cannot point to the page that satisfies it, it does not clear the item. The item becomes a condition written in plain language with the failing rule named. A large share of what other systems call hallucination never gets past this layer, because an answer with no citation is not an answer the system will accept. We wrote about the grounding and citation architecture behind this separately.
Layer Two: The Checker Is a Different Model, and It Does Not See the Maker's Reasoning
The second layer is an independent checker. Three choices in how it is built matter more than anything else in this post.
First, it is a separate model from a different model family than the maker. Two instances of the same model tend to make the same mistakes on the same inputs, because they share training data and failure modes. If the maker misreads a handwritten date on a gift letter, a second copy of the same model is more likely to misread it the same way. Using a different model family does not guarantee independent errors, but it lowers the correlation, and the correlation is the whole point.
Second, the checker sees the rule, the evidence and the maker's conclusion. It does not see the maker's chain of reasoning. The reason is a known weakness of language models: shown a fluent argument for a conclusion, a model tends to find the argument persuasive and agree with it. A reviewer who reads the first reviewer's notes before looking at the file is not an independent reviewer, and the same is true here. So the checker re-derives the answer from the source, and we compare conclusions.
Third, the checker goes back to the source document, not to the maker's extraction of it. This is the failure mode we design hardest against. If both models work from the same extracted text, an OCR error in the extraction becomes an error both models agree on, and agreement on a wrong input looks exactly like confidence. So for the fields that drive a decision, the checker re-reads the relevant region of the page image. Shared inputs are the quiet way two independent checks turn into one.
Disagreement Is Never Settled by a Vote
When maker and checker disagree, the item is held and surfaced to a person. It is not resolved by a third model breaking the tie, and it is not resolved by taking the answer with the higher confidence. Both of those are tempting because they keep throughput up, and both throw away the most useful signal the system produces. A disagreement means at least one of two independent reads got it wrong, and the item is, by definition, one where the evidence or the rule is less clear than usual. That is exactly what a human underwriter should look at.
Every disagreement is logged in the Evidence Trail with both conclusions, the rule and the evidence, and the human's resolution is logged beside them. Those resolved disagreements are the best test cases we have, so they go into the golden-file library that the agent is regression-tested against before every change ships. When the same kind of disagreement keeps recurring, it usually means the rule itself is ambiguous, and the fix belongs in the Rulebook, written precisely enough that both models and both underwriters would read it the same way.
Layer Three: The Confidence Floor Is a Number the Lender Sets
The third layer is confidence scoring, and the floor is a number the lender sets, not one we pick. Anything above the floor clears automatically, with the citation trail attached. Anything below it goes to the lender's own underwriter or QC reviewer as a named exception, with the question, the evidence and both models' reads.
A confidence number is only worth something if it is calibrated: when the system says 95%, it has to be right about 95% of the time on that kind of item. We measure that against the lender's own history before go-live, by running the agent in shadow against loans a person already decided and plotting stated confidence against actual agreement. The floor is set per item type, not once for the whole file, because the cost of a wrong answer is different for each. A misnamed document in the wrong eFolder is a nuisance. A wrongly excluded liability is an ability-to-repay problem. We recommend stricter floors on anything that lowers risk on paper, such as an excluded debt, a sourced deposit or a cleared fraud flag, than on anything that adds a condition.
Calibration drifts. A new loan product, a new document layout from a large employer's payroll vendor, a change in the borrower mix, and the stated confidence can stop meaning what it meant at launch. So calibration is monitored continuously against the items that humans review, and a widening gap between stated and actual accuracy moves the floor or pulls the item type back to full review until the cause is found.
What This Did to Review Time
Three layers exist so that it is safe to stop reading the items that clear. In our pre-close QC deployment at Better Mortgage, on Conventional, FHA, VA and HELOC files spanning over twenty document types, 92% of conditions checks run end to end without a person, 78% of items clear at high confidence with a citation trail, and human review per loan file fell from 4 hours 50 minutes to 47 minutes. The reviewers stopped reading the 78% and spent their time on the exceptions, which are exactly the items where maker and checker disagreed or where confidence fell under the floor.
The reviewer's job changes from line-reading every file to adjudicating exceptions, which is a better use of an experienced underwriter and, frankly, a better control, because a person reading their two-hundredth clean paystub of the day is not giving the two-hundred-and-first one their full attention.
How This Maps to SR 11-7 and the GSE Requirements
None of this is a new idea in model risk. The Federal Reserve's SR 11-7, adopted by the OCC as Bulletin 2011-12, makes "effective challenge" a guiding principle: critical analysis by objective, informed parties who can identify model limitations and assumptions and produce appropriate changes. It also expects validation to involve a degree of independence from model development and use. An independent checker built on a different model, blind to the maker's reasoning, re-reading the source, is effective challenge applied to every item instead of to a validation sample once a year. The human adjudication of disagreements is the part of effective challenge that still needs a person with authority.
The NIST AI Risk Management Framework puts the same idea under its Measure function, which treats measurement of validity and reliability as ongoing, not a launch gate. The continuous calibration monitoring is that function in practice.
For agency sellers and servicers the requirement is now explicit. Freddie Mac's Bulletin 2025-16 added AI and machine learning governance requirements to its Seller/Servicer Guide, effective March 3, 2026, including for AI embedded in vendor platforms the lender did not build. An examiner or a GSE reviewer asking how an AI agent's output is checked before it reaches the loan file deserves a better answer than "the model was confident." The answer we give is the three layers above, with the log for any item they want to pull.
What Triple Check Does Not Do
It does not make the agent right about things the Rulebook gets wrong. If an overlay is encoded incorrectly, maker and checker will both apply the incorrect rule faithfully, which is why Rulebook changes are themselves reviewed and versioned with an effective date. It does not replace post-closing QC or the investor's review; it moves defects earlier, while they are still conditions. And it does not remove the human decision on anything the lender has decided a human owns, such as a credit decision, a fraud referral or a SAR. Those stay with named people, and the agent's job is to put the evidence in front of them.
What it does do is give every item the AI clears the same thing mortgage has always required of a person's work: a second independent read, a stated standard, and a record of who decided what.
Ramkumar Venkataraman
CTO & Co-Founder