Latent Legal AI Errors, Agentic Benchmarks, and Court Sanctions

Erick S. Robinson

The central question today: as legal AI moves from single prompts to layered agentic workflows, how do latent synthesis errors slip past a basic citation check, and how are rules and sanctions assigning supervisory responsibility to the lawyer? This is education, not legal advice. And a caution up front: some figures in the brief we're working from, including a November 2026 benchmark and a New York rule effective June 1, 2026, are time sensitive. Verify them against primary sources before you rely on them.

Dr. Angela Liu

I'll add the litigator's version of that question. If the case name is real, the brief looks fine. But what if the case is real and the proposition is wrong? Or the case is real, the proposition is right, and it's from the wrong jurisdiction? That's the failure I worry about.

Erick S. Robinson

Start with the independent evidence on retrieval. Daniel Ho and colleagues published an empirical study in the Journal of Empirical Legal Studies in 2025. They found proprietary legal research tools hallucinated between 17 percent and 33 percent of the time, even with retrieval grounding. As summarized in the brief, Westlaw AI-Assisted Research hallucinated in roughly 33 percent of responses, and Lexis+ AI was accurate about 65 percent of the time. Those are the figures as reported to me. Check the study itself for definitions and test conditions, because hallucination there covers more than invented cases.

Dr. Angela Liu

Right, and that's the layer distinction. A base model inventing a case name is one error type. A retrieval tool pulling a genuine but irrelevant authority, or accepting a false premise in your question, is another. The brief calls these retrieval confounding and false premise adoption. A citation checker catches neither.

Erick S. Robinson

Now the agent layer. Harvey's Legal Agent Benchmark, version HLAB-Vals-1.1, tracked by Vals AI in November 2026, reports frontier models passing 93 to 98 percent of individual criteria. But the full task pass rate falls to roughly 23 to 27 percent. The examples given are Gemini 4 Argon at about 27 percent and Muse Spark Open at about 25 percent. One caveat: this benchmark comes from a vendor, tracked by a third party. I'd treat it as informative, not as settled independent evidence.

Dr. Angela Liu

The arithmetic is intuitive once you've built a claim chart. Ninety five percent of the checklist can be right and the deliverable is still unusable. In patent work, the misses tend to sit in negative claim limitations, or antecedent basis tracking across dependent claims. One miss there can change an infringement or invalidity position.

Erick S. Robinson

And in procedure, the brief gives an example: a motion falsely asserting a deadline is jurisdictional, despite contrary precedent like Kontrick v. Ryan. The citation might be perfectly real. The doctrine is what's drifted. That's the latent jurisdictional drift we've been discussing since the earlier episodes on hallucination rates and sanctions.

Dr. Angela Liu

So what are courts doing? The tracking database in the brief counts over 1,800 hallucination incidents globally, nearly 69 percent in the U.S. Pro se litigants account for about 59 percent of occurrences. But licensed attorneys account for 97 percent of formal sanctions. Courts expect more from us, and they should.

Erick S. Robinson

The consequences have moved past nominal Rule 11 fines. The brief lists briefs struck, a default judgment in Flycatcher Corp. v. Affable Ave., disqualification in Johnson v. Dunn, and a formal suspension in Florida Bar v. Neusom. I'd read the opinions directly for the exact holdings; I'm relying on the brief's characterizations.

Dr. Angela Liu

And New York's 22 NYCRR Part 161, effective June 1, 2026 per the brief, is interesting. It doesn't impose blanket disclosure of AI use. It does require affirmative, independent attorney verification of all cited authorities. The duty is non delegable. Use the tool if you like, but you own the output.

Erick S. Robinson

Which defines what verification is not. It's not re-prompting the model to confirm itself. It's not confirming a case exists in a reporter. It means running Shepard's or KeyCite, reading the primary text, and checking that the doctrine fits your facts and your jurisdiction.

Dr. Angela Liu

For firms, I'd say build that into the workflow: a named reviewer, a record of what was checked, and extra scrutiny on the agent outputs that look cleanest. For vendors, publish error rates by task and error type, not one headline number. For courts, focus on the duty of verification more than on disclosure.

Erick S. Robinson

What's unknown matters too. Will multi step reasoning agents replace blatant fabrications with subtle statutory misreadings? That would be much harder for junior lawyers and clerks to spot, and we don't yet have independent data on it.

Dr. Angela Liu

Bottom line: a real citation proves nothing about the proposition. Verify the law, not the link.

Erick S. Robinson

Reliable legal AI isn't purchased with a license. It's designed into the workflow, with human verification at every layer that matters.