🔍 Read the full analysis: Understanding Z.ai’s Legal AI Ranking Despite Its Benchmark Pass Rate on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s Harvey Legal Agent Benchmark leaderboard, ranking seventh of 81 models in the archived version. The model passed 92.5% of individual criteria, but a task counted as complete only when every criterion passed. Z.ai’s claimed 12.5-percentage-point gain over its base model matches the leaderboard figures; its claim of state-of-the-art performance among open-source models has no comparison set specified in the cited documentation.
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Vals AI’s leaderboard for Harvey’s Legal Agent Benchmark, placing seventh among 81 models in the archived results. The score means the model met every required criterion on about one in five benchmark tasks—not that it achieved only 19.17% accuracy across all legal criteria.
The distinction comes from the benchmark’s strict scoring rule. Harvey says a task is complete only if every criterion passes. A task can therefore fail because of one missed deadline, citation or other required detail, even if the system satisfies most of its rubric. On the same Vals leaderboard version, GLM Legal passed 92.5% of individual criteria, a much higher measure that does not mean it completed 92.5% of tasks.
The archived Vals AI Version 1 leaderboard, last updated October 18, lists GLM Legal at 19.17% across a 120-task private holdout, with a listed cost of $0.15 per test. The base GLM-5.3-Flash model scored 6.67% on that table. The difference is 12.50 percentage points, matching Z.ai’s stated improvement. The leaderboard also placed Muse Spark 1.2 at 25.42%, Muse Spark 1.3 Max at 23.75%, and Gemini 4 Argon at 19.58%.
Z.ai describes the model as optimized for legal document understanding, reasoning and task execution, including research, contract review and drafting. Its documentation says the model received legal-domain mid-training and post-training, including reinforcement learning across tens of thousands of task environments modeled on legal workflows. Z.ai recommends connecting it as an agent to services for laws, cases and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens.
A High Criteria Score, Few Complete Tasks
The two pass rates answer different questions for legal teams. The 92.5% criteria pass rate indicates that the model often meets individual rubric requirements, while the 19.17% task rate shows how often it satisfied all the requirements in a work product. For workflows where a single omitted issue or incorrect citation can make a document unreliable, the all-criteria result is the more demanding measure of end-to-end completion.
Cost is another point of interest, though the figures need care. Vals listed $0.15 per test for GLM Legal, below the listed costs for the higher-ranked models in that archived table. That comparison may matter to organizations testing systems at scale, but benchmark expense alone does not establish the total cost of using a system in practice. The available information does not detail the cost of human review, integration, data controls or correcting errors.
The leaderboard is not evidence that the model can independently provide reliable legal advice. Its results come from a defined set of benchmark tasks and grading rules. Firms considering it would still need to check its outputs, assess how client information is handled and determine whether the model fits their professional and operational requirements.
Popular right now
Dyson Car+Boat Cordless Handheld Vacuum
Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Black
Dreame G10 Pro Wet and Dry Vacuum Mop
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Tasks
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. It contains 1,250 tasks across 24 legal practice areas, with more than 75,000 expert-written rubric criteria. Tasks pair an instruction with materials for a client matter and ask an agent to produce a work product for review. The criteria cover items such as facts, conclusions, citations, recommendations, deadlines, dollar amounts and formatting.
The public leaderboard results use a private 120-task holdout that Vals says mirrors the public task mix; internet access is disabled during testing. Vals reports that two language-model judges, GPT 5.5 and Claude Sonnet 4.6, grade submissions and calculate task pass rates, with the reported score averaging their results. In its benchmark description, Harvey argues that a deal-team report identifying eight of ten risks is still materially incomplete—an explanation for the all-pass rule.
Z.ai’s October 13 AutoClaw blog post announced the model and opened a beta by application; Vals lists the release date as October 12. Z.ai’s documentation says it is built on GLM-5.3-Flash. The source article also notes that Harvey’s May 26 baseline reported a best all-pass result of 7.1% among the frontier models tested in that run. That earlier result is a separate benchmark run and should not be treated as a direct comparison with the later Vals leaderboard.
“A task is marked complete only if every criterion passes.”
— Harvey, describing its Legal Agent Benchmark
Claims and Limits of the Results
The benchmark establishes a result on a particular test set; it does not establish performance across all legal work or jurisdictions. Z.ai’s claim of state-of-the-art performance among open-source models is not fully assessable from the cited documentation because it does not define which models count as open-source or provide a comparison set. The source material also reports no published weights for the legal model, leaving its openness unclear.
The source article describes a four-task spot check in which the legal model cited rulings the reviewer could not find. The author says the spot check was not a benchmark, so it cannot establish a general error rate. The same check found that two Claude models cited a real ruling for the opposite of what it held, while the OpenAI and Claude models in that sample invented none. These observations are limited and should not be read as a systematic comparison.
It is also not clear from the cited Z.ai pages who processes client data when the model is used, which rules govern that processing, or what checks are required before generated material is used. The source article says the pages describe outputs as drafts for legal professionals and do not carry a “not legal advice” notice. That absence does not, by itself, establish how Z.ai or its customers address legal, privacy or professional obligations.
Questions Before Legal Deployment
The next practical step for firms and other users is to validate the model against their own tasks, review how citations and factual claims can be checked, and establish a human review process before any generated work product is relied on. Procurement teams will also need information about client-data handling, applicable rules and system integrations, particularly if the model is connected to internal company records or external legal sources.
Z.ai’s AutoClaw post described a beta available by application. The source material does not give a broader release timetable, further benchmark results or details of independent testing beyond the Vals leaderboard. Any wider assessment will depend on additional disclosures, later leaderboard updates and evaluations that show how the model performs on the specific work firms intend to use it for.
Key Questions
What does the 19.17% score mean?
It is the share of tasks on the Vals leaderboard’s 120-task private holdout for which the model passed every required criterion, averaged across two AI judges. A task failed if even one criterion did not pass.
Why is the criteria pass rate 92.5%?
The 92.5% figure measures individual criteria, not fully completed tasks. A model can pass most items in a task and still fail the task under Harvey’s all-criteria grading rule.
Did GLM Legal improve on Z.ai’s base model?
Yes, on the archived Vals Version 1 leaderboard: GLM Legal scored 19.17%, compared with 6.67% for GLM-5.3-Flash. That is a 12.50-percentage-point difference, consistent with Z.ai’s rounded 12.5-point claim.
Does the benchmark show the model is ready to give legal advice?
No. The result measures performance on a defined benchmark, not reliability across all legal matters. The source material describes the model’s work as drafts for legal professionals; firms would need to verify outputs and evaluate data handling and review procedures before use.
Source: ThorstenMeyerAI.com
