Validating a Classifier Is Easy. This Is Not a Classifier.
There is a comfortable way to validate a model, and it does not work here. You hold out a labeled test set, you run the model, you compute accuracy, precision, recall, maybe an AUC, and you have a number you can put in front of a model risk committee. That approach is sound for a scoring model, a thing that takes a fixed input and returns a prediction, because the unit of behavior is a single call and you can sample those calls and measure them.
A long-horizon agent is not that. The unit of behavior is not a call. It is a loan worked across weeks, during which the agent reads documents, calculates income, creates and clears conditions, re-plans when the file changes, and stops and escalates when it hits its confidence floor. "Accuracy on a test set" does not describe that, because there is no single prediction to score. There are dozens of decisions, in sequence, each depending on the state the prior decisions produced, and the thing you actually need to know is not whether the agent gets one answer right. It is whether the whole trajectory is right, and whether it fails safely when it is wrong. We argued in the long-horizon versus copilot piece that an agent that runs for thirty days is a different kind of thing than a feature that runs for a fraction of a second, and testing is where that difference stops being rhetorical and starts being an engineering problem.
Golden Files: The Loans We Know the Answer To
The foundation of the harness is a library of golden files, which are real loan files, de-identified, each paired with the correct outcome and the reasoning behind it. Not a synthetic input designed to exercise a code path, but an actual loan with actual documents that a senior underwriter has already worked and whose correct disposition we know. The golden library is curated to cover the distribution that matters, which is not the easy middle of the pipeline but the edges: the self-employed borrower with a complicated cash-flow analysis, the file with an undisclosed debt that appears late, the gift with a broken transfer trail, the condo project that fails warrantability, the income that looks sufficient until you read the fourth document.
The edges are the point, because an agent that handles clean files is not interesting and clean files are not where the losses come from. We build the golden library the way a good QC function builds a defect taxonomy, from the failure modes that actually occur, and every time the agent gets something wrong in production or a human catches an escalation that should have been a decline, that file becomes a golden file. The library grows toward the failures, which is the only direction worth growing in, because a test set that is all easy cases gives you a high number and tells you nothing about the loans that will hurt you.
Running the Whole Trajectory, Not Just the Endpoint
Scoring a golden file means running the agent through the entire file and checking two things, which are not the same thing. The first is whether it reached the right disposition, approve, decline, condition, escalate. The second, and the one that matters more, is whether it got there for the right reasons. An agent that clears a file to close for the wrong reason is more dangerous than one that declines it, because the wrong-reason clear will generalize to files where the reason actually matters and produce a defect you will not see until it funds.
So the harness checks the reasoning path, not only the endpoint. Did the agent source the large deposit, or did it clear the file without noticing the deposit was unsourced and happen to land on approve anyway? Did it calculate the income from the correct documents, or did it reach a plausible number by a path that will break on the next file? Because every decision the agent makes is in the Evidence Trail, cited to the document that drove it, the harness can compare not just the answer but the derivation against what the senior underwriter did on the same file. This is closer to grading a proof than to scoring a prediction, and it is the only way to catch the agent that is right for reasons that will not hold.
Backtesting Against the Decisions a Human Already Made
Before an agent touches a live file, we run it in shadow against the recent historical book, which means feeding it loans a human already decided and comparing the agent's decisions to the ones the institution actually made, without the agent's output affecting anything. This is the backtest, and it does the thing a golden library cannot do alone, which is measure the agent against the full distribution of real production volume rather than a curated sample.
The disagreements are the signal. Where the agent and the human agreed, you learn little. Where they disagreed, you learn where the agent's judgment diverges from the institution's, and every disagreement gets adjudicated by a senior reviewer into one of three findings: the agent was wrong, the human was wrong, or the rule was ambiguous and both were defensible. The first finding fixes the agent. The second is the uncomfortable and valuable one, because it means the agent caught something the human missed, which is evidence of lift but also a reason to look harder at the historical book. The third sends us back to the Rulebook to make the rule precise enough that the ambiguity does not recur. This adjudication is labor, and it is the labor that actually produces a trustworthy agent, because an agent validated only against cases where it agrees with history has been validated against its own blind spots.
What SR 11-7 and the NIST Framework Ask For Here
This is not a methodology we invented in a vacuum. It is what the supervisory guidance already asks for, read honestly. SR 11-7 describes model validation as resting on conceptual soundness, ongoing monitoring, and outcomes analysis, and it emphasizes effective challenge, which is critical review by people with the competence and the independence to find the problem. The golden library and the backtest are outcomes analysis. The adjudication of disagreements by an independent senior reviewer is effective challenge. The guidance was written for statistical models, and the commentary that it is hard to apply to agentic systems is half right: the specific tests differ, but the three pillars map cleanly, and an operation that skips them because "the guidance predates LLMs" is skipping the part that actually keeps the thing safe.
The NIST AI Risk Management Framework adds the function the point-in-time validation misses, which is Measure as an ongoing activity rather than a launch gate. An agent is not validated once. It is validated continuously, because the world it operates in moves, the guide changes, the borrower population shifts, and a validation that was true in March can be false by September without a single line of the agent changing. We built the first-90-days deployment controls around standing the monitoring up before the agent goes live, because the go-live gate is the easy part and the ongoing measurement is the part that tells you whether the agent is still the agent you validated.
Ongoing Monitoring: Catching Drift Before It Becomes a Finding
Once the agent is live, the harness does not retire. It runs continuously against new production, and it watches for the things that degrade quietly. Confidence calibration is the first: when the agent says it is ninety percent sure, is it right ninety percent of the time, or has its stated confidence drifted away from its actual accuracy? A miscalibrated agent is dangerous in a specific way, because the whole bounded-autonomy design rests on the confidence floor being meaningful, and a floor set against a confidence number that no longer means what it did is a floor in the wrong place.
Drift is the second. When the document mix changes, when a new loan product enters the pipeline, when an investor overlay shifts the population, the agent is now seeing inputs its validation did not cover, and the monitoring has to notice that the inputs have moved rather than waiting for the outcomes to go bad. We tie this to the real outcome signal wherever we can get it, which means the agent's decisions get checked against what the loans actually did, the same early-payment-default feedback loop that tells a QC function whether its pre-funding review was right. A loan that the agent cleared and that defaulted on payment one is the most direct outcomes-analysis signal there is, and it feeds back into the golden library as a new case the agent has to get right next time.
The Regression Suite, Because Fixing One Thing Breaks Another
Every change to the agent, a new capability, a tuned threshold, a Rulebook update, a model upgrade, runs against the full golden library before it ships, exactly as a software team runs its test suite before merging. This is the unglamorous part and the part that prevents the most common failure, which is fixing the behavior on one kind of file and silently breaking it on another. Without a regression suite, every improvement is a gamble, because you cannot see the loan you just broke until it funds. With one, a change that fixes the self-employed income calculation and quietly degrades the gift-funds verification gets caught before it leaves the building, because the gift-funds golden files are in the suite and they fail.
The regression suite is also what makes a model upgrade survivable. When a better underlying model becomes available, you do not get to assume it is better on your actual work just because it is better on public benchmarks, because your work is mortgage files and the benchmark is not. You run the new model against the full golden library and the recent backtest, you look at what moved, and you ship only if it improved on the cases you care about without regressing the ones you already got right. That is the discipline that lets an operation adopt a better model without betting the pipeline on a vendor's benchmark, and it only exists if you built the harness first.
The Honest Read
Testing a long-horizon agent is more work than validating a classifier, and the work does not compress, because the thing you are validating is a trajectory of dependent decisions rather than a single prediction, and the failures that matter are the wrong-reason successes that a surface accuracy number hides. The golden library, the trajectory scoring, the backtest against real human decisions, the continuous monitoring, and the regression suite are not optional rigor on top of the product. They are the product, in the sense that an agent you cannot test to this standard is an agent you cannot responsibly put on a loan file, and no amount of impressive behavior in a demo substitutes for knowing how it fails on the four hundred edge cases you have already seen.
We built the evaluation harness at Sei before we built most of what it tests, because the supervisory guidance is right that a model touching credit decisions has to be validated by people independent enough to break it, and an agent is a model that got ambitious. The agent that runs for thirty days is a colleague, and you would not put a colleague on live files without checking their work against cases where you already know the answer. The harness is how we check the work, continuously, against the loans we know cold, so that the ones we do not know cold get a decision we can stand behind.
Ramkumar Venkataraman
CTO & Co-Founder