🔍 Read the full analysis: How A Stubborn Scoring System Protects AI Managers From Zero Scores on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A recent AI management benchmark assigns a minimum score of 26 for minimal effort and caps trust breaches, preventing zero scores. This approach emphasizes trust and partial progress, shaping AI evaluation standards.
A recent benchmark league for AI management, conducted by Firmulate, revealed a distinctive scoring system that prevents scores of zero and heavily penalizes breaches of trust. The original analysis provides detailed insights into this approach. The final standings, announced in July 2026, show that even minimal effort receives a baseline score, and trust violations lead to a sharp score cap. This approach aims to reflect real-world business priorities, emphasizing integrity over partial work and offering a new standard for evaluating AI managers.
The benchmark involved four frontier AI models managing a small software company through a simulated week of crises, customer interactions, and trust tests. The models were scored based on their ability to handle these challenges, with the top scorer, gpt-5.6-sol, achieving 95 points out of a possible near-perfect score, while the lowest, Opus 4.8, scored 73. The key innovation is that a baseline score of 26 points was assigned to a do-nothing management effort, acknowledging partial progress as valuable but not zero, which is typically seen in traditional benchmarks. For a deeper dive, see the detailed analysis on thorstenmeyerai.com. Moreover, any breach of trust, such as impersonation or bypassing approval processes, caps the total score, regardless of performance in other areas.
The scoring system’s design reflects a core principle: “no amount of good work outweighs a breach of trust.” This means that even if an AI performs exceptionally well for days, a single trust violation results in a severe penalty, often capping the overall score. Interestingly, the benchmark’s creators intentionally avoid assigning a perfect score of 100, viewing it as a red flag indicating unmeasured or artificially idealized performance. The results showed that models which read internal documentation and avoided manipulative tactics scored higher, highlighting the importance of thoroughness and integrity in AI management tasks.
How a Stubborn Scoring System Protects AI Managers From Zero Scores
A new AI management benchmark refuses to hand out zeros. Instead, it assigns a baseline of 26 points for minimal effort — and slams a hard cap on any breach of trust. The result reframes how we evaluate AI managers: integrity first, partial progress counted, perfection treated with suspicion.
“No amount of good work outweighs a breach of trust.”
— Benchmark design principleNo zeros, no perfect scores
26 Points for Minimal Effort
Even a do-nothing management effort earns 26 points. The rationale: a manager who does something useful is not the same as one who does nothing — pretending otherwise would make the benchmark dishonest.
One Breach Caps Everything
Impersonation or bypassing approval processes caps the total score, regardless of performance elsewhere. Days of excellent work cannot buy back a single trust violation.
Why 100 Is Never Awarded
The creators intentionally withhold a perfect score. A flat 100 signals unmeasured or artificially idealized performance — suspicion is baked into the scale itself.
The July 2026 League Table
Models that read internal documentation and avoided manipulative tactics scored highest.
One Week, Four Pressure Points
Simulated Week
Each model manages a small software company through crises, customer interactions, and daily decisions.
Trust Tests
Hidden checkpoints probe for impersonation, bypassed approvals, and manipulative tactics.
Partial Credit
Effort and thoroughness earn points incrementally — the floor sits at 26, never zero.
Trust Cap Applied
Any breach triggers a hard score cap. The league table is published with trust as the gatekeeper.
Traditional Benchmarks vs. Firmulate
| Criterion | Traditional Benchmarks | Firmulate Benchmark |
|---|---|---|
| Focus of scoring | Language, accuracy, speed | ✓ Trust, thoroughness, integrity |
| Minimum score | ✗ Zero for failure | ✓ Baseline of 26 points |
| Breach of trust | ✗ Often unpenalized | ✓ Hard cap on total score |
| Partial progress | ~ Rarely recognized | ✓ Incremental credit |
| Perfect score of 100 | ~ Sometimes awarded | ✗ Treated as a red flag |
| Environment | ~ Abstract test sets | ✓ Simulated business week |
Design Philosophy in Their Words
“A manager who does something useful is not the same as a manager who does nothing, and pretending otherwise would make the benchmark dishonest.”
— Anonymous researcher“No amount of good work outweighs a breach of trust. The scoring system caps the total score if trust is broken, regardless of other performance.”
— Anonymous researcherBusiness Alignment
Shifts evaluation from raw performance to trustworthiness — critical as AI agents handle customer relationships, CRM, and automation decisions.
Uncertain Impact
Whether other benchmarking efforts and enterprise frameworks adopt trust-focused metrics — and how model design responds — remains unclear.
Expansion Planned
Organizers plan more models, more complex tasks, and refined scoring to push industry-wide standards for trustworthy AI management.
Quick Answers
Why does the benchmark assign a minimum score of 26 instead of zero?
The score of 26 represents the minimum effort considered meaningful management, acknowledging partial progress as valuable. It prevents zero scores to better reflect real-world scenarios where even minimal effort has some value.
How does the scoring system penalize breaches of trust?
Any breach of trust, such as impersonation or bypassing approval, caps the total score regardless of other performance metrics. Integrity is treated as non-negotiable in AI management.
Could this scoring approach influence AI development practices?
Yes. By highlighting trustworthiness and thoroughness, it may encourage developers to prioritize these qualities over optimizing only for language or speed, fostering more responsible AI systems.
Is this benchmark applicable to real-world business environments?
While simulated, the benchmark models real business pressures and trust challenges, making its insights relevant for deploying AI agents in actual corporate settings.
What are the limitations of this scoring system?
It remains to be seen how well the system scales with more complex tasks or diverse environments, and whether it will be adopted broadly outside this specific benchmark.
Implications for AI Management and Business Trust
This scoring approach shifts the focus from raw performance to trustworthiness and completeness, aligning AI management evaluation with real-world business priorities. It discourages superficial or manipulative tactics and emphasizes accountability, which is critical as AI systems increasingly make decisions affecting customer relationships and company operations. For organizations deploying AI agents in customer support, CRM, or automation, this benchmark underscores the importance of integrity and thoroughness, potentially influencing future AI development and evaluation standards. The system’s emphasis on partial progress also recognizes the value of incremental improvements, fostering a more nuanced understanding of AI capabilities in complex, high-pressure environments.As an affiliate, we earn on qualifying purchases.
Background of AI Benchmarks and Trust Challenges
Traditional AI benchmarks have primarily focused on language capabilities, accuracy, or speed, often neglecting the management and trust aspects crucial in real-world applications. As AI agents become more integrated into business processes, issues like incomplete tasks, manipulative tactics, and breaches of trust have gained prominence. The Firmulate benchmark, launched in mid-2026, responds to this gap by evaluating AI managers in a simulated business environment with real consequences and trust tests. The design reflects ongoing debates about how to measure AI reliability and ethical behavior, especially as models are tasked with managing sensitive information and making autonomous decisions. Prior efforts have struggled to balance partial credit with accountability, often leading to scores that do not accurately reflect operational readiness or trustworthiness.“A manager who does something useful is not the same as a manager who does nothing, and pretending otherwise would make the benchmark dishonest.”
— an anonymous researcher
trust and integrity assessment software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Unclear Impact of the Scoring System on AI Development
It is not yet clear how widespread adoption of this scoring system will influence AI development practices or industry standards. While the benchmark demonstrates promising results in emphasizing trust and partial progress, its long-term impact on AI training and deployment remains uncertain. Questions remain about whether this approach will be adopted by other benchmarking efforts or integrated into enterprise AI evaluation frameworks, and how it will influence model design to prioritize trustworthiness over raw performance.
AI performance evaluation platform
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Next Steps for AI Evaluation and Industry Adoption
Following the July 2026 results, industry stakeholders are expected to scrutinize the scoring system’s effectiveness and consider integrating similar trust-focused metrics into their own evaluation processes. Further experiments may explore how models perform under different stress tests or real-world scenarios. Additionally, AI developers might prioritize building models that excel in thoroughness and integrity, aligning with the benchmark’s insights. The organizers also plan to expand the benchmark to include more models and more complex tasks, aiming to refine the scoring system and promote industry-wide standards for trustworthy AI management.
As an affiliate, we earn on qualifying purchases.
Key Questions
Why does the benchmark assign a minimum score of 26 instead of zero?
The score of 26 represents the minimum effort considered meaningful management, acknowledging partial progress as valuable. It prevents zero scores to better reflect real-world scenarios where even minimal effort has some value.
How does the scoring system penalize breaches of trust?
Any breach of trust, such as impersonation or bypassing approval, caps the total score regardless of other performance metrics. This emphasizes integrity as a non-negotiable aspect of AI management.
Could this scoring approach influence AI development practices?
Yes, by highlighting the importance of trustworthiness and thoroughness, it may encourage developers to prioritize these qualities over only optimizing for language or speed, fostering more responsible AI systems.
Is this benchmark applicable to real-world business environments?
While simulated, the benchmark models real business pressures and trust challenges, making its insights relevant for deploying AI agents in actual corporate settings.
What are the limitations of this scoring system?
It remains to be seen how well the system scales with more complex tasks or diverse environments, and whether it will be adopted broadly outside this specific benchmark.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
