Legal AI Agents: Criterion Scores vs. Task Completion

Erick S. Robinson

Welcome to The AI Law Podcast. I'm Erick Robinson, and this episode is education, not legal advice. Here's our central question. When a legal AI agent scores ninety five percent on accuracy, how often does it actually finish the job? Three takeaways. First, criterion level scores can hide task level failure. Second, the evidence differs by tool layer and by error type. Third, courts in 2026 are not asking what the vendor promised.

Dr. Angela Liu

And I'm Angela Liu. A quick bridge from earlier episodes. Stanford's 2024 study, testing tools in May 2024, found Lexis+ AI hallucinating more than 17% of the time, Westlaw's AI assisted research around 33%, and GPT-4 around 43%. That was one question, one answer. Today we're talking about agents that read a matter file, run code, draft a deliverable, and check themselves. That's a different layer, so it needs a different yardstick.

Erick S. Robinson

Start with the Vals AI board, HLAB Vals 1.1. It runs Harvey's Legal Agent Benchmark on a held out set of 120 tasks. The top entry, Gemini 4 Argon, passes 96.30% of individual criteria. But its task pass rate is 27.08%. Wonder Full passes 96.13% of criteria and resolves 23.33% of tasks. The reason is the grading rule. A task counts as resolved only if every single criterion passes.

Dr. Angela Liu

So ninety six percent sounds like an A, and roughly three tasks in four still fail a strict review. As a litigator, that tracks. A brief where ninety five percent of the propositions are right is not a ninety five percent brief. One fabricated cite is the whole problem.

Erick S. Robinson

Here's an intuition, and I want to be clear it's my illustrative arithmetic, not a Vals finding. If a task had twenty independent checks, each passed at 95%, the chance of passing all twenty is about 36%. Real criteria aren't independent, and tasks have different numbers of criteria. But it shows why an average and an all pass result diverge.

Dr. Angela Liu

Harvey's own May 2026 baselines show the same shape under that strict standard. Frontier models complete less than 10% of tasks in aggregate. Claude Opus 4.7 leads at 7.1%, Sonnet 4.6 at 5.4%, Opus 4.6 at 4.2%, GPT 5.5 at 2.1%, Gemini 3.5 Flash at 0.8%. The top configuration costs about fifty dollars and ninety cents and roughly twenty two minutes per task.

Erick S. Robinson

And disclose the conflicts. Harvey built this benchmark and sells legal AI. Vals says it ran Harvey's harness with its own fixes, and that its results are not comparable with Harvey's board or with Artificial Analysis's. Different models, different judges. So read each table directionally, and don't line them up against each other.

Dr. Angela Liu

One finding I like: no single model leads every practice area. In Harvey's data, GPT 5.5 leads retrieval heavy groups, Opus 4.7 leads synthesis heavy corporate work, and Sonnet 4.6 leads privacy and tax. Harvey calls it jagged intelligence. Their trace analysis also found agents that revise after validating improved, while drafting without review fell by 1.2 points. That's just what a good associate does.

Erick S. Robinson

Now error types, because they sit at different layers. At the architecture layer, the Schwarcz randomized trial gave 127 law students either vLex's Vincent AI, a retrieval tool, OpenAI's o1 preview reasoning model without retrieval, or no AI. Both AI groups were faster. The retrieval group hallucinated at about the same rate as the no AI group. The reasoning group wrote stronger analysis but introduced hallucinations.

Dr. Angela Liu

Then there's sycophancy. Stanford's study and Anna Guo's 2025 work found that when a prompt carries a false premise, tools often build plausible support instead of correcting it. And when a liability cap was redacted, some tools fabricated an answer. For litigators that's the dangerous one, because a lawyer typically asks, find me cases supporting my position.

Erick S. Robinson

Add geography. The Place Matters study found hallucination rates of 45% for Los Angeles, 55% for London, and 61% for Sydney, and 100% on one local Australian tenancy statute. Add temporal blindness: one model applied Chevron after Loper Bright because its cutoff predated the decision. And Stanford's Large Legal Fictions found no correlation between a model's confidence and its accuracy. Those studies tested earlier generation tools, so they show patterns, not today's exact rates.