The Layer Underneath Every Other Agent
A mortgage loan file does not arrive as structured data. It arrives as a stack of PDFs and phone photos: paystubs, W-2s, two months of bank statements, two years of tax returns, a driver's license, a purchase contract, an appraisal, a title commitment, and a dozen things that are none of those but got uploaded anyway. Before any of the work people associate with AI underwriting can happen, something has to read that stack and turn it into data. It has to decide that these three pages are one bank statement and those two are a different one, that this page is a W-2 and not a 1099, and that the number sitting in a particular box is annual wages and not year-to-date withholding.
That layer, document classification and field extraction, is underneath every other agent in the pipeline. The income calculation agent cannot compute qualifying income until something has decided which documents are the income documents and pulled the right figures off them. The condition-clearing agent cannot match a document to a condition until something has classified the document. Everything downstream inherits whatever this layer got right or wrong, which is exactly why it is the layer teams underestimate. It looks like plumbing. It is a model, and its errors do not announce themselves.
How the Reader Fails, and Why a Spot-Check Misses It
A human processor reading a loan file makes errors that are mostly obvious in hindsight, because a person who misreads a document usually produces a result that looks wrong. A machine reader fails differently, and the difference is the whole governance problem. It produces a confident, plausible, wrong answer, and nothing about the output looks off.
Two failure modes from our own shadow-mode testing make this concrete. The first was a self-employed borrower's bank statement where the extraction pulled a year-to-date deposit total and handed it downstream as if it were a single month's deposits, because the statement's layout put the YTD figure where a monthly figure usually sits. The number was real, it came off the right document, and it was wrong by a factor of twelve, and no downstream check that trusted the field would have caught it. The second was an upload where two borrowers on the same loan had paystubs merged into one PDF, and the classifier treated the whole thing as one document belonging to one borrower, attributing one person's income to the other. Both errors were confident. Both would have survived a human spot-check of a sample, because the sampled fields looked fine, and the failure was in the files nobody sampled.
That is the thing that makes this layer dangerous in a way ordinary software is not. A deterministic parser either works or throws. A learned reader degrades, and it degrades on the documents that are slightly off from the training distribution, which in mortgage is a huge fraction of real files, because every employer's paystub and every bank's statement looks a little different. Average accuracy on clean documents tells you almost nothing about the behavior that matters, which is what the reader does on the messy edge.
Confidence Is an Output, and the Threshold Is a Control
The design decision that governs this layer is what the reader does when it is not sure. A reader that returns only an answer is a reader you cannot control. A reader that returns an answer and a calibrated confidence in that answer is one you can build a control around, and the control is the threshold at which the agent stops trusting itself and routes the page to a person.
We treat extraction confidence as a first-class output, calibrated so that the number means what it says, and we set the routing threshold by the cost of the specific field being wrong, not by a single global cutoff. A field that feeds the income calculation, where an error flows straight into the qualifying decision, gets a high threshold, so the agent routes to human review on anything short of near-certainty. A field that is informational and will be re-verified anyway can run at a lower one. The threshold is not a tuning parameter the engineering team picks for throughput. It is a risk control, set where the expected cost of a confident error crosses the cost of a human review, and it is documented and owned as a control rather than buried in a config.
The straight-through-processing rate that results is a byproduct, not a target. An illustrative way we hold it internally is that we would rather the reader send a clean W-2 to a human because the scan was skewed than confidently extract a skewed W-2 and be wrong, because the first mistake costs thirty seconds of review and the second mistake can corrupt a credit decision. Teams that optimize the STP rate directly end up lowering thresholds until the confident errors start getting through, which is optimizing the one number that should never be optimized in isolation.
Why This Is a Model Under SR 11-7
It is tempting to classify document reading as IT infrastructure and keep it outside model governance, and that is the mistake. The moment the reader's output feeds a credit or eligibility decision, which it always does, it is processing input data into estimates that a decision relies on, which is what SR 11-7, the interagency model-risk guidance from the Federal Reserve and the OCC, treats as a model across its full lifecycle. An extraction that says this borrower's monthly income is a particular number is an estimate, and the underwriter who relies on it is relying on a model whether or not anyone called it one.
Governing it as a model means the things SR 11-7 asks for apply here, not just to the scoring model everyone remembers to govern. The reader has a documented development record: what document types it handles, what it was trained and evaluated on, and where its known limitations are, the skewed scans and the merged uploads and the layouts it has not seen. It gets independent validation, tested by people who did not build it, against a held-out set of real files with known-correct answers, and the validation reports accuracy at the field level and on the hard document classes specifically, not a single headline number. And it has ongoing monitoring, because the document distribution drifts as employers change their payroll vendors and banks redesign their statements, and a reader validated last year can quietly fall out of its envelope this year.
The NIST AI Risk Management Framework covers the parts specific to a learned reader that a traditional model-risk process does not have language for: the ways a document model can be wrong that a scorecard cannot, including confident misclassification and extraction from an adversarially malformed document. We run both, SR 11-7 for the model-risk backbone an examiner expects and NIST AI RMF for the AI-specific failure surface, because the reader is the model most likely to be treated as plumbing and therefore the one most likely to be ungoverned. This is the same model-governance posture we apply to anything in the credit path, extended down to the layer people forget is in the credit path.
The Fair-Lending Edge Nobody Draws
There is a fairness dimension to this layer that does not get discussed because the layer is seen as mechanical. If the reader's accuracy is not uniform across document types, and if particular document types correlate with particular borrower populations, then an accuracy gap becomes a disparity. A reader that handles a standard salaried W-2 near-perfectly but struggles with the irregular documentation of a gig worker or a self-employed borrower is a reader whose errors fall harder on those borrowers, and errors that fall harder on a protected-class-correlated population are a fair-lending exposure even when the model has no idea who the borrower is.
So our validation measures accuracy sliced by document type and by the documentation patterns that correlate with non-traditional borrowers, and a material accuracy gap is a finding we fix before deployment, not a performance footnote. The point is not that the reader knows anyone's protected characteristics. It is that uneven accuracy across document types produces uneven treatment across borrowers, and the control for that is measuring the slices rather than the average. A reader governed only on aggregate accuracy can be systematically worse for exactly the borrowers fair-lending enforcement is meant to protect.
Where the Human Stays in the Loop
The architecture that makes all of this governable is the one where the reader produces reviewable outputs and never acts on its own reads alone. Every extracted field carries its source, the document and the location it came from, so a human or a downstream check can see not just the number but where it was pulled, which is what turns the YTD-as-monthly error from an invisible corruption into a flagged, checkable claim. Low-confidence pages route to people. The documents that feed the highest-stakes fields get the tightest thresholds. And the whole thing is logged, so the record shows which document was classified as what, with what confidence, and who reviewed the ones the agent was unsure about, which is the same audit trail a pre-funding QC review relies on.
Keeping the human on the uncertain pages is not a lack of confidence in the reader. It is what makes the reader's confident answers trustworthy, because a reader that routes its genuine uncertainty to people is one whose unrouted answers you can actually rely on. A reader that never asks for help is a reader that is wrong confidently, and confidently wrong is the one failure mode in the credit path that costs the most.
The Honest Read
The document reader is the quietest model in the mortgage pipeline and one of the most consequential, because every other agent inherits its output and its errors are confident and silent. It fails on the messy edge rather than the clean average, and a spot-check of a sample will not catch the YTD figure read as a month or the two borrowers merged into one file. Those are the errors that corrupt a credit decision while looking completely normal.
Treating the reader as plumbing is how it ends up ungoverned, and ungoverned is where it does damage. At Sei we build the classification-and-extraction layer as a governed model: calibrated confidence as a real output, thresholds set as risk controls by the cost of each field being wrong, independent validation on the hard document classes and sliced for fairness, source attribution on every extracted field, and the uncertain pages routed to people. The reader that knows when it does not know is the one you can build a mortgage on. The reader that is always sure is the one that funds the loan with the wrong number in the box.
Ramkumar Venkataraman
CTO & Co-Founder