🔍 Read the full analysis: Why Z.ai’s Legal AI Benchmark Rank Outpaces Its Pass Rate on ThorstenMeyerAI.com
Get the little things that make your day delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal scored 19.17% on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh among 81 models. The task score is all-or-nothing, while the model passed 92.5% of individual criteria, explaining why its rank can look stronger than its end-to-end pass rate. The benchmark offers a narrow measure of performance, not proof that the model can safely handle legal work without human review.
The figures refer to different levels of scoring, not a contradiction. A benchmark task may require an agent to review a matter and produce a work product that meets numerous criteria, including factual findings, legal conclusions, citations, deadlines and formatting. Under Harvey’s all-pass grading, one failed criterion is enough for the entire task to count as incomplete. The 19.17% result means the model met every criterion on roughly one in five tasks in the evaluation.
Vals AI’s archived leaderboard lists GLM Legal at 19.17% of tasks, seventh among 81 models, at a reported $0.15 per test. It also records a 92.5% criterion pass rate; Vals’ version notes that a criterion counted as passed only when both of its judges agreed. The leaderboard score is the average task pass rate from two language-model judges, GPT-5.5 and Claude Sonnet 4.6. The evaluation used a private holdout of 120 tasks, with internet access disabled.
Z.ai’s documentation describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including research, contract review and drafting. The company says it was adapted through legal-domain training and reinforcement learning on tens of thousands of task environments. Z.ai reported a 12.5-percentage-point gain over the base model and called the result state of the art among open-source models in the evaluation. The archived leaderboard’s figures support the stated difference: GLM Legal scored 19.17%, against 6.67% for GLM-5.3-Flash, the base model listed there.
Why Task Scoring Changes the Rank
The distinction matters to anyone interpreting legal AI rankings. A high criterion pass rate can indicate that a system gets many individual details right, while its much lower all-pass rate shows how often it completes a full assignment without any rubric-defined miss. In legal work, a correct summary with one wrong citation or missed deadline could still require correction before use. The benchmark’s strict scoring makes that kind of failure visible, but it also means the task percentage should not be read as a simple measure of how much useful work the model can do.
For law firms and legal departments, the result is evidence about one controlled evaluation, not a substitute for checking outputs in practice. Z.ai recommends connecting the model to external services for laws, cases and company information. Buyers would still need to examine how those sources are selected, how client information is handled, and who verifies citations and other consequential details. The leaderboard’s low per-test price is a reported evaluation cost, not a full measure of deployment or review expenses.
Products Worth Considering
TCL 65 Inch QM8L Series SQD-Mini LED QLED 4K Smart Google TV
Dyson Car+Boat Cordless Handheld Vacuum
Philips Hue Secure Starter Kit with Contact Sensors, Motion Sensor, Bridge, and Color Smart Bulbs
As an affiliate, we earn on qualifying purchases.
How Harvey Grades Legal Agents
Harvey released its Legal Agent Benchmark as an open-source evaluation on May 6, 2026. It contains 1,250 tasks spanning 24 practice areas, assessed against more than 75,000 expert-written binary criteria. Harvey says a task is complete only if every criterion passes, describing this method as all-pass grading. Its example of a deal-team report makes the rationale clear: identifying some risks while missing others may leave the work product materially incomplete.
The public leaderboard uses a private holdout set designed to reflect the benchmark’s task mix. Vals AI says that set contains 120 tasks and is graded by two AI judges, whose task pass rates are averaged. Results depend on the benchmark version and evaluation setup. Harvey’s own May baseline, for example, reported a best all-pass rate of 7.1% among the frontier models it tested; that is a separate run and should not be treated as a direct comparison with Vals’ later leaderboard.
Z.ai’s AutoClaw blog announced the model on October 13, 2026, with beta access by application; Vals lists the release as October 12. Z.ai’s documentation page does not carry a date. The company says the legal model is built on GLM-5.3-Flash and lists API pricing at $0.37 per million input tokens and $1.25 per million output tokens, separate from Vals’ per-test figure.
““A task is marked complete only if every criterion passes.””
— Harvey, describing its Legal Agent Benchmark grading rule
Limits of the Published Results
The leaderboard establishes a score under a particular benchmark setup; it does not show how GLM Legal performs across all legal matters, jurisdictions or live firm workflows. The 120-task holdout is a limited sample, and the result depends on automated judging. The available material does not establish whether the same ranking would hold under other task sets, human review or different scoring rules.
Z.ai’s description of the model as state of the art among open-source models is a company claim. The source material does not specify the comparison set, and it does not identify published model weights for GLM Legal. A separate four-task spot check described by the source author found that the model cited rulings the author could not locate; that small check is not a benchmark result and cannot establish a general error rate. The materials also do not settle how client data would be processed in a particular deployment or who would be responsible for checking each citation.
What Firms Should Verify Next
The next useful evidence would include further evaluations on different legal tasks, clearer disclosure of the comparison set behind Z.ai’s open-source ranking claim, and practical information about data handling and deployment controls. Z.ai’s announced beta is available by application, but the source material does not give a wider release date or report a subsequent rollout.
For organizations considering the model, the immediate next step is to test it against their own work under human supervision. That means checking citations against primary sources, reviewing deadlines and factual claims, and confirming how documents and other client information are handled. Until further evidence is available, the leaderboard supports a measured conclusion: GLM Legal performed strongly in rank relative to the listed field, but completed all benchmark criteria on fewer than one in five tested tasks.
Key Questions
What does GLM Legal’s 19.17% score mean?
It means the model met every required criterion on 19.17% of tasks in the Vals AI leaderboard evaluation. A task with even one failed criterion counted as incomplete.
Why is the criteria pass rate much higher?
The 92.5% figure measures individual rubric criteria, while 19.17% measures whole tasks. A model can pass most details but still fail a task if it misses one required item.
Where did GLM Legal rank?
Vals AI’s archived leaderboard listed it seventh among 81 models, at $0.15 per test in that evaluation.
Does the result prove the model is safe for legal work?
No. It is a score on a defined benchmark, not proof of reliable performance across legal matters or permission to use outputs without review. The available results do not resolve how the model performs in live workflows or how client data is handled.
Is Z.ai’s 12.5-point improvement supported by the leaderboard?
The archived figures show GLM Legal at 19.17% and the listed base model at 6.67%, a difference of 12.50 percentage points. That supports the arithmetic for this leaderboard version, though it does not establish results on other evaluations.
Source: ThorstenMeyerAI.com
