Skip to content
Sei

The Loan File Arrives as a Pile of PDFs: Document Classification, Field Extraction, and Why the Model That Reads the Stack Is Governed Like the One That Scores It

7 min readRamkumar Venkataraman
Share

The Layer Underneath Every Other Agent

A mortgage loan file does not arrive as structured data. It arrives as a stack of PDFs and phone photos: paystubs, W-2s, two months of bank statements, two years of tax returns, a driver's license, a purchase contract, an appraisal, a title commitment, and a dozen things that are none of those but got uploaded anyway. Before any of the work people associate with AI underwriting can happen, something has to read that stack and turn it into data. It has to decide that these three pages are one bank statement and those two are a different one, that this page is a W-2 and not a 1099, and that the number sitting in a particular box is annual wages and not year-to-date withholding.

That layer, document classification and field extraction, is underneath every other agent in the pipeline. The income calculation agent cannot compute qualifying income until something has decided which documents are the income documents and pulled the right figures off them. The condition-clearing agent cannot match a document to a condition until something has classified the document. Everything downstream inherits whatever this layer got right or wrong, which is exactly why it is the layer teams underestimate. It looks like plumbing. It is a model, and its errors do not announce themselves.

How the Reader Fails, and Why a Spot-Check Misses It

A human processor reading a loan file makes errors that are mostly obvious in hindsight, because a person who misreads a document usually produces a result that looks wrong. A machine reader fails differently, and the difference is the whole governance problem. It produces a confident, plausible, wrong answer, and nothing about the output looks off.

Two failure modes from our own shadow-mode testing make this concrete. The first was a self-employed borrower's bank statement where the extraction pulled a year-to-date deposit total and handed it downstream as if it were a single month's deposits, because the statement's layout put the YTD figure where a monthly figure usually sits. The number was real, it came off the right document, and it was wrong by a factor of twelve, and no downstream check that trusted the field would have caught it. The second was an upload where two borrowers on the same loan had paystubs merged into one PDF, and the classifier treated the whole thing as one document belonging to one borrower, attributing one person's income to the other. Both errors were confident. Both would have survived a human spot-check of a sample, because the sampled fields looked fine, and the failure was in the files nobody sampled.

That is the thing that makes this layer dangerous in a way ordinary software is not. A deterministic parser either works or throws. A learned reader degrades, and it degrades on the documents that are slightly off from the training distribution, which in mortgage is a huge fraction of real files, because every employer's paystub and every bank's statement looks a little different. Average accuracy on clean documents tells you almost nothing about the behavior that matters, which is what the reader does on the messy edge.

Confidence Is an Output, and the Threshold Is a Control

The design decision that governs this layer is what the reader does when it is not sure. A reader that returns only an answer is a reader you cannot control. A reader that returns an answer and a calibrated confidence in that answer is one you can build a control around, and the control is the threshold at which the agent stops trusting itself and routes the page to a person.

We treat extraction confidence as a first-class output, calibrated so that the number means what it says, and we set the routing threshold by the cost of the specific field being wrong, not by a single global cutoff. A field that feeds the income calculation, where an error flows straight into the qualifying decision, gets a high threshold, so the agent routes to human review on anything short of near-certainty. A field that is informational and will be re-verified anyway can run at a lower one. The threshold is not a tuning parameter the engineering team picks for throughput. It is a risk control, set where the expected cost of a confident error crosses the cost of a human review, and it is documented and owned as a control rather than buried in a config.

The straight-through-processing rate that results is a byproduct, not a target. An illustrative way we hold it internally is that we would rather the reader send a clean W-2 to a human because the scan was skewed than confidently extract a skewed W-2 and be wrong, because the first mistake costs thirty seconds of review and the second mistake can corrupt a credit decision. Teams that optimize the STP rate directly end up lowering thresholds until the confident errors start getting through, which is optimizing the one number that should never be optimized in isolation.

Why This Is a Model Under SR 11-7

It is tempting to classify document reading as IT infrastructure and keep it outside model governance, and that is the mistake. The moment the reader's output feeds a credit or eligibility decision, which it always does, it is processing input data into estimates that a decision relies on, which is what SR 11-7, the interagency model-risk guidance from the Federal Reserve and the OCC, treats as a model across its full lifecycle. An extraction that says this borrower's monthly income is a particular number is an estimate, and the underwriter who relies on it is relying on a model whether or not anyone called it one.

Governing it as a model means the things SR 11-7 asks for apply here, not just to the scoring model everyone remembers to govern. The reader has a documented development record: what document types it handles, what it was trained and evaluated on, and where its known limitations are, the skewed scans and the merged uploads and the layouts it has not seen. It gets independent validation, tested by people who did not build it, against a held-out set of real files with known-correct answers, and the validation reports accuracy at the field level and on the hard document classes specifically, not a single headline number. And it has ongoing monitoring, because the document distribution drifts as employers change their payroll vendors and banks redesign their statements, and a reader validated last year can quietly fall out of its envelope this year.

The NIST AI Risk Management Framework covers the parts specific to a learned reader that a traditional model-risk process does not have language for: the ways a document model can be wrong that a scorecard cannot, including confident misclassification and extraction from an adversarially malformed document. We run both, SR 11-7 for the model-risk backbone an examiner expects and NIST AI RMF for the AI-specific failure surface, because the reader is the model most likely to be treated as plumbing and therefore the one most likely to be ungoverned. This is the same model-governance posture we apply to anything in the credit path, extended down to the layer people forget is in the credit path.

The Fair-Lending Edge Nobody Draws

There is a fairness dimension to this layer that does not get discussed because the layer is seen as mechanical. If the reader's accuracy is not uniform across document types, and if particular document types correlate with particular borrower populations, then an accuracy gap becomes a disparity. A reader that handles a standard salaried W-2 near-perfectly but struggles with the irregular documentation of a gig worker or a self-employed borrower is a reader whose errors fall harder on those borrowers, and errors that fall harder on a protected-class-correlated population are a fair-lending exposure even when the model has no idea who the borrower is.

So our validation measures accuracy sliced by document type and by the documentation patterns that correlate with non-traditional borrowers, and a material accuracy gap is a finding we fix before deployment, not a performance footnote. The point is not that the reader knows anyone's protected characteristics. It is that uneven accuracy across document types produces uneven treatment across borrowers, and the control for that is measuring the slices rather than the average. A reader governed only on aggregate accuracy can be systematically worse for exactly the borrowers fair-lending enforcement is meant to protect.

Where the Human Stays in the Loop

The architecture that makes all of this governable is the one where the reader produces reviewable outputs and never acts on its own reads alone. Every extracted field carries its source, the document and the location it came from, so a human or a downstream check can see not just the number but where it was pulled, which is what turns the YTD-as-monthly error from an invisible corruption into a flagged, checkable claim. Low-confidence pages route to people. The documents that feed the highest-stakes fields get the tightest thresholds. And the whole thing is logged, so the record shows which document was classified as what, with what confidence, and who reviewed the ones the agent was unsure about, which is the same audit trail a pre-funding QC review relies on.

Keeping the human on the uncertain pages is not a lack of confidence in the reader. It is what makes the reader's confident answers trustworthy, because a reader that routes its genuine uncertainty to people is one whose unrouted answers you can actually rely on. A reader that never asks for help is a reader that is wrong confidently, and confidently wrong is the one failure mode in the credit path that costs the most.

The Honest Read

The document reader is the quietest model in the mortgage pipeline and one of the most consequential, because every other agent inherits its output and its errors are confident and silent. It fails on the messy edge rather than the clean average, and a spot-check of a sample will not catch the YTD figure read as a month or the two borrowers merged into one file. Those are the errors that corrupt a credit decision while looking completely normal.

Treating the reader as plumbing is how it ends up ungoverned, and ungoverned is where it does damage. At Sei we build the classification-and-extraction layer as a governed model: calibrated confidence as a real output, thresholds set as risk controls by the cost of each field being wrong, independent validation on the hard document classes and sliced for fairness, source attribution on every extracted field, and the uncertain pages routed to people. The reader that knows when it does not know is the one you can build a mortgage on. The reader that is always sure is the one that funds the loan with the wrong number in the box.

Ramkumar Venkataraman

Ramkumar Venkataraman

CTO & Co-Founder

Related Articles

Mortgage

When Rates Drop, Your Servicing Portfolio Gets Called by Someone: Refinance Recapture With an AI Voice Agent, and the Tests That Decide Who It Calls

A rate move puts thousands of loans in the money in the same week, and the servicer has the data to reach those borrowers first. Before an AI agent dials any of them, it has to know which loans can actually close: VA's seasoning and recoupment tests under 38 U.S.C. 3709, the early payoff window on loans you sold, which consent covers a marketing call, and what the agent may and may not say about savings. What we learned running a voice agent on a lender's funnel, applied to the recapture campaign.

6 min read
Read more
Mortgage

A Fraud Alert on the Credit Report Is a Legal Stop: What FCRA Section 605A Requires Before a Mortgage Proceeds, and Who Makes the Call

An initial, extended or active duty alert on a tri-merge report restricts the lender by statute, and the verification has to go to the phone number in the alert, which is not always the one on the application. Address discrepancy notices carry their own duty under Regulation V. A technical account of how our agents read alerts on every credit pull, what they compare, and why we decided the verification call is made by a person.

6 min read
Read more
Mortgage

Social Security, Pensions and the Gross-Up: Calculating Nontaxable and Retirement Income Without Counting It Twice

The same monthly benefit produces a different qualifying income under Fannie Mae, FHA and VA, and the gross-up is applied to the whole check far more often than the rules allow. A technical account of how our income agent reads the taxable fraction off the tax return, reconciles the award letter against the deposit, tests continuance from documented terms rather than age, and where Regulation B draws the line on what an underwriter may ask.

5 min read
Read more

You Ain't Seen Nothin' Yet

Book a Demo

Pack up some of your complex historical files — any loan type, any investor. We run them through intake, income and condition clearing, and in 30 minutes you see every condition we created and cleared efficiently for your own team, and why.

  • Any loan type, any agency guideline or custom investor overlays.
  • Every finding cited to the guideline or document it came from

Case study highlight

Loan review fell from 4 hours 50 minutes to 47 minutes per file, and 78% of items now clear at high confidence — so QC touches only the exceptions.

Pre-Close QC Agent · Better Mortgage— read the case study

Please provide your full name so we know how to address you.

Tell us which company you represent so we can personalise our response.

Use your work email. One of our experts will get back to you.

Which desks would you like to discuss?*

Choose the desks you’d like us to cover. Pick “Not sure yet” if you’d rather we worked it out on the call.

Roughly, so we bring the right person to the call.

Complete the verification to submit the form.