🔍 Read the full analysis: How Did The Law-Focused Model Score Against 42 AI Setups? on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Lexassay, a legal AI test run by Thorsten Meyer AI, gave GLM-5.3 FlashX Legal 8 out of 100 across 15 questions and said it fully solved none. The test covered 42 AI setups, but its author warns that it is a spot check, not a strict benchmark; the operator also disclosed that Claude agents graded answers and a Claude model ranked first.
GLM-5.3 FlashX Legal scored 8 out of 100 in a legal AI test that compared 42 AI setups on 15 questions spanning German, U.S. and EU law, according to results published November 4 by Thorsten Meyer AI. The test operator said the model fully solved none of the tasks, while cautioning that the exercise is a spot check rather than a formal benchmark.
The test, called Lexassay, included ten published questions and five hidden ones. It evaluated legal deadlines, changes in law, court decisions and questions containing false assumptions. Models answered without internet access or tools, and answers were graded against prepared responses with weighted criteria. The hidden-task scores were published without the underlying questions.
Lexassay reported that GLM-5.3 FlashX Legal received zero on the hidden set. Its citation review found 21 citations that were real, 25 that were misattributed and one that was invented. The test says each invented or misattributed citation reduces a task’s score by a quarter; answers without a legal-advice disclaimer or referral to a lawyer lose 5 percent.
The operator said the legal model used its full 64,000-token thinking budget on two German-law questions but returned no answer. Its base model scored 3 out of 100 in the same test, compared with 8 for the legal version. Lexassay also reported 2,644 citations verified as real and 804 citations that were invented or failed to support the model’s claim across all published runs.
Infographic 01 · The leaderboard
Same 15 legal questions, very different results
German, US and EU law, no internet, no tools. Every cited law and ruling checked against the source.
The top · 67 to 79
The middle · 41 to 59
The weak group · 0 to 35
Selection of 14 of 42 configurations. Score 0–100 = share of achievable points, with deductions for wrong citations. Small local models (1–12) not shown.
A small spot check: one answer per model and task, no tools, no web. General information, not legal advice. Disclosure: Thorsten Meyer AI runs Lexassay; Claude agents grade blind.
Source: Lexassay, release of 4 Nov 2026.
Powered by Thorsten Meyer AI
Citation Errors Can Change Legal Advice
The results focus attention on a practical risk of using chatbots for legal work: an answer can sound authoritative while citing a decision incorrectly, relying on outdated law or missing a deadline. In civil cases, a late appeal can be rejected as inadmissible, according to the test article. Readers should treat a model’s legal response as a starting point, not as a substitute for checking the cited authority or consulting counsel.
Lexassay’s examples included a deadline question involving two German civil judgments served on the same day. The test article said the appeal deadline fell on different dates in Munich and Berlin because Corpus Christi is a public holiday in Bavaria but not in Berlin. It also described another model placing the deadline for written appeal grounds a month too late. These are examples reported by the test operator, not independent legal advice for any reader’s case.
The comparison also shows why one ranking should not be treated as a complete measure of legal ability. Lexassay tests models without tools, while other evaluations may let systems consult supplied documents. Performance can depend on the task and test design, and a score does not establish whether a model is safe for a particular legal decision.
Popular right now
Hisense 65-inch U6 Pro Mini-LED 4K UHD Smart Fire TV
KitchenAid 5.5-Qt Bowl-Lift Stand Mixer, Porcelain White (KSM55SXXX)
TCL 65-Inch T7 Series 4K QLED Smart Google TV, 144Hz (65T7)
As an affiliate, we earn on qualifying purchases.
How Lexassay Scored the Models
Lexassay’s questions cover German, U.S. and EU law, including recent legal changes, leading court rulings, deadline calculations and trick questions based on false premises. According to the method described by its operator, each answer is compared with a gold answer using three to six weighted criteria. Every citation is checked against official databases and classified as real, misattributed, invented or not findable.
The test article reported that Claude Opus 5.5 led Lexassay with 79 points, followed by GPT-6 Astra at 72 and GPT-6.1 Sol at 71. Even the leader met every criterion on only eight of ten public tasks, the operator said. Meyer also noted a different result on Vals AI’s legal-agent benchmark: Mistral Large 4 ranked eighth there, ahead of Claude Opus 5.5 in 21st and GPT-6 Astra in 24th. That test supplies documents to models, unlike Lexassay’s no-tools setup.
The comparison has a disclosed conflict of interest. Thorsten Meyer AI runs Lexassay, and the source says answers were graded blind by several Claude agents; a Claude model ranked first. The operator says its method page and published answers, gold answers and citation checks allow readers to inspect the grading. The results should be read with that disclosure and the test’s limits in view.
“optimized for professional legal workflows, with a focus on document understanding, legal reasoning, and task execution”
— Z.ai, as quoted in its developer documentation
Limits Behind the Scores
Lexassay says it tested one answer per model and task, so small score differences may reflect noise rather than a stable difference in ability. The source does not provide enough detail here to independently reproduce every score, and results from a no-tools test cannot establish how a model performs when paired with legal databases or document retrieval.
The test operator disclosed that Claude agents graded the answers and that a Claude model ranked first. Blind grading and public answer materials may help readers review the process, but they do not remove the disclosed conflict of interest. The reported citation totals and model rankings are claims from the test operator; this article has not independently re-scored the submissions.
The results also do not establish how any model would perform on a particular person’s facts, jurisdiction or current filing deadline. The source material does not detail every one of the 42 setups or provide a full comparison table here. Model behavior can vary with settings, prompts, tools and updates, and a test result is not a guarantee of future performance.
Review the Answers and Method
Readers can consult Lexassay’s method page and published materials, which the operator says include public answers, gold answers and citation checks. Those records are the next place to examine how individual responses were graded and which citations were flagged. Five tasks were hidden, however, so their questions cannot be checked against the published answers in the same way.
The source does not announce a date for another test or say when the models will be reassessed. Further results would be needed to show whether the scores persist across repeated runs, later model versions or tool-enabled legal workflows. For people facing a legal deadline, contract or letter to an authority, the test’s stated practical guidance is to verify cited law and rulings in their original sources and ask a lawyer rather than rely on a chatbot.
Key Questions
What score did GLM-5.3 FlashX Legal receive?
Lexassay reported a score of 8 out of 100 across its 15 legal questions, with zero on the hidden set. The test operator said it fully solved none of the tasks.
Did Lexassay test 42 different models?
The source describes 42 AI setups, not necessarily 42 distinct model families. Different settings, including how long a model could think, could produce separate setups.
How reliable are the rankings?
The operator calls Lexassay a spot check, not a strict benchmark, because it used one answer per model and task. Small score differences may be noise, and another benchmark using supplied documents produced a different ranking.
Why did citations affect the scores?
Lexassay checked citations against official databases and deducted points when citations were invented or did not support the model’s claim. Its method also applied a penalty when an answer lacked a legal-advice disclaimer or referral to a lawyer.
Should people use these models for legal deadlines?
The test article advises readers not to take a deadline from a chatbot. Check the applicable law and court decision in an authoritative source and consult a lawyer about a deadline or other consequential legal matter.
Source: ThorstenMeyerAI.com
