Why Legal AI Fails: Benchmarks, Error Layers, and Workflow Safeguards

Erick S. Robinson

Welcome to The AI Law Podcast. I'm Erick Robinson, and with me is Dr. Angela Liu. Quick note first: this is educational and technological analysis, not legal or patent advice. Today's question is why legal AI keeps failing at measurable rates even as models improve. Our answer is that hallucination isn't one glitch. It's several different failures, living at different layers of the tool.

Dr. Angela Liu

And that matters because when a partner says, we bought the legal tool, so we're safe, they're treating three layers as one. There's the foundation model, the retrieval layer, and then the application guardrails around it. Each one fails differently.

Erick S. Robinson

Start with the foundation model. The Stanford paper Large Legal Fictions, by Dahl and colleagues, tested 2023 general purpose models on over 800,000 verifiable legal questions. Hallucination rates ran from 58 to 88 percent. GPT four was at 58, GPT three point five at 69, Llama 2 at 88. So newer models did better, but the baseline was bad.

Dr. Angela Liu

Then retrieval was supposed to fix it.

Erick S. Robinson

Partly. Magesh and colleagues, in the Journal of Empirical Legal Studies in 2025, tested proprietary retrieval tools on more than 200 preregistered queries. Lexis+ AI hallucinated 17 percent of the time and was accurate on 65 percent. Westlaw AI Assisted Research hallucinated 33 percent and was accurate on 42 percent. Thomson Reuters Ask Practical Law AI gave incomplete answers, meaning refusals or ungrounded responses, on more than 60 percent of queries. General purpose GPT four came in at 43 percent hallucination in that comparison.

Dr. Angela Liu

Now, I want to flag a trap. Someone will line up 58 percent from Dahl next to 43 percent from Magesh and say retrieval gave a fifteen point improvement. Different studies, different questions, different model snapshots. That's not a clean comparison.

Erick S. Robinson

Agreed, and there's a bigger caveat. The Stanford team tested May 2024 snapshots, and the study was published in 2025. Vendors update continuously without public version notes. Products like Lexis Protégé have moved since. So treat these as evidence about error patterns, not a current scoreboard.

Dr. Angela Liu

The Vals report from October 2025 is more recent, and it complicates the vendor story. Top legal AI tools scored 78 to 81 percent accuracy. General purpose models with web search got about 80. Both beat the unassisted human lawyer baseline of 69 percent.

Erick S. Robinson

Right, so the tool layer isn't magic. What seems to matter is access to curated, verified sources with citators. But look at the other number from Vals: accuracy fell 14 percentage points on complex multi jurisdictional surveys across fifty states. Even a good tool degrades as the task gets harder.

Dr. Angela Liu

And the Schwarcz randomized trial shows the workflow side. 127 law students. Those using the retrieval based Vincent AI had roughly the same hallucination rate as students using no AI. Students using o1 preview, a reasoning model without retrieval, produced better analytical work but introduced hallucinations. Both gained productivity. Only the retrieval tool did it without adding errors.

Erick S. Robinson

Now the error types, because a fabricated case is the easy one. First is sycophancy. The Stanford study named it as one of four major error types. If your prompt contains a wrong legal premise, like find cases holding X, the tool may build convincing support, or cite real cases that it mischaracterizes, instead of correcting you.

Dr. Angela Liu

This is where I get nervous as a patent litigator. Think about claim construction. You prompt, show me where the file history limits this term, and the tool wants to please you. It may confabulate a limitation or misread claim scope. Same with prior art invalidity searches. To be clear, the sources we reviewed don't measure patent tasks specifically. That's my inference from the sycophancy findings, so test it on your own matters.

Erick S. Robinson

Second is jurisdictional fragility. In Place Matters, by Curran and colleagues in 2025, hallucination rose from 45 percent for Los Angeles, to 55 for London, to 61 for Sydney. On a specific Australian Residential Tenancies Act scheme it hit 100 percent. Large Legal Fictions found models do best on Supreme Court cases and worst on district court metadata.

Dr. Angela Liu

Third, time. In the study AI Gets Its First Law School A+s, OpenAI's o3 applied Chevron in an administrative law exam. Its cutoff was May 2024, and Loper Bright came in June 2024. And the Stanford testing found Lexis+ AI treating Casey's undue burden standard as good law after Dobbs. That's a good law failure, and a citation check wouldn't catch it, because the case is real.

Erick S. Robinson

Fourth, the confidence paradox. Large Legal Fictions found no correlation between a model's expressed confidence and its accuracy. And research suggests post training with human feedback can make models more overconfident, because benchmarks reward definitive answers over honest uncertainty. So tone is not a signal.

Dr. Angela Liu

Courts are done being patient. The count of documented incidents where lawyers or self represented parties filed AI hallucinations is approaching 1,000. In Johnson v. Dunn, in the Northern District of Alabama on July 23, 2025, the court disqualified counsel after hallucinated citations in a discovery filing. It reasoned that monetary Rule 11 fines, like the $5,000 in Mata v. Avianca, no longer provide enough deterrence. I'd read the opinion itself, not just my summary of it.

Erick S. Robinson

So what do firms do? First, match verification to layer and error type. Citation checking catches fabrications, not mischaracterized holdings or stale doctrine. Someone has to read the authority and run a citator for current treatment. Second, write neutral prompts. Never ask a tool to support a position you haven't verified.

Dr. Angela Liu

Third, escalate scrutiny for state and local law, lower courts, recent changes, and complex synthesis. Fourth, ask vendors hard questions: which model version, when was it last independently tested, does it use live retrieval, can it decline to answer? Refusing can be a feature. And for clients and courts, disclose how AI was used and who verified the output.

Erick S. Robinson

Bottom line: the best legal tools in the evidence still err somewhere around one in five answers, depending on the study and the metric, and the failures are patterned. Reliability gets designed into workflow and human verification. It doesn't arrive with the license.

Dr. Angela Liu

And what we don't know: how today's tool versions perform, how any of this plays out in patent tasks specifically, and whether disqualification becomes the norm. Until independent testing catches up, assume the vendor number is a starting point. Thanks for listening.