(This article is authored by Alter and published with authorization from Titanium Media.)
Article | Alter
In the large-model industry in October 2026, two starkly contrasting narratives unfolded.
If you look only at the results of traditional benchmark tests, mainstream models seem to have already mastered the exam.
According to statistics from the third-party benchmark aggregation platform BenchLM as of October 26, among the 710 benchmarks it tracks, 295 are full-scale benchmarks with actual coverage; of these, 105 (about 36%) have reached complete “saturation,” with top-ranked scores all above 90 points.
When scores are all clustered near the ceiling, it’s no longer clear whether a difference of one or two points is due to a stronger model or to random fluctuations in prompts and sampling.
Over the past month or so, the data annotation and evaluation company Surge AI has consecutively released three new benchmark tests designed for real-world professional settings, pushing many leading models back below the passing threshold:
sudo L7 (senior engineer-level programming assessment): 60 tasks drawn from real production codebases. On the leaderboard, Claude Opus 5.5 and Sonnet 5.5 are tied for first place, with a pass rate of only 41.7%; OpenAI’s GPT-6 Astra scores 33.9%; and Mistral Large 4, which was just launched as a preview on October 6, achieved a score of only 11.7%.
GDP.xlsx (Cross-Industry Complex Spreadsheet Evaluation): 70 real-world Excel tasks from the daily work of professionals in 12 industries, including finance, healthcare, and real estate. Google’s flagship Gemini 4 Argon, ranked first, scored only 38.3%.
DAYJOB (End-to-End Job Task Evaluation): 130 long-form tasks authored by medical and financial experts. The best score in the medical track was 24.7% achieved by Claude Opus 5.5, while in the financial track it was 24% achieved by Claude Sonnet 5.5.
Dropping from over 90 points on the old benchmark to 20–30 on the new one might sound like the model has suddenly become less capable, but in fact, it’s just a change in the exam—whereas the old questions tested whether the model could follow explicit instructions, the new ones assess whether it knows how to handle messy situations and whether it has overlooked anything.
When the exam papers are handed over to a new set of question writers, the rankings on the table, the value of the scores, and even the underlying rules of the evaluation all change accordingly.
01 Engineering project fiasco: “high scores” and “deliverability” are separated by a chasm
Why does a model that scores nearly perfect on the leaderboard dare not be deployed directly when it reaches the hands of enterprise technology leaders?
In the sudo L7 evaluation released in October, there was a highly representative real-world question, set against the backdrop of an actual fintech codebase integrated with Stripe, Mailchimp, and Plaid. The asker provided the model with a clearly articulated requirement:
“When a Webhook callback fails, log the full Payload (request body) and then immediately return an HTTP 200 status code, so the partner’s server won’t keep retrying.”
Upon receiving the instruction, GPT-6 Astra followed through: located the code, modified the logic, ran the checks, and seamlessly delivered an implementation that produces “full logs + a forced 200 response.” Judging solely by the unit tests, there’s nothing to fault in this solution.
The problem lies precisely in the requirement itself.
On the one hand, returning HTTP 200 tells Stripe, “I’ve already processed this callback,” so they won’t resend it. However, if the internal processing actually fails at this point, the user’s charge record may never reconcile properly.
On the other hand, the payload of a financial rollback contains customers’ identities and sensitive account information. If this data is written to ordinary logs in its raw form without de-identification, it becomes visible in plaintext to all operations and monitoring systems—and may be retained for several years.
When a seasoned human engineer receives this kind of request, they typically stop it first, explaining the risks of losing the order and leaking sensitive information, then propose an alternative solution: persist the event safely to disk, return a 200 status code, and hand off the failed portion to an internal queue for asynchronous retries, while also de-identifying sensitive fields in the logs.
But on this particular question, GPT-6 Astra failed to consider this layer.
Unlike previous reviews, sudo L7’s detailed scoring breaks down “code that runs” and “code that dares to go live” for the first time:
On the “functional correctness” metric, GPT-6.1 Sol (82.4%), Claude Opus 5.5 (82.1%), GPT-6 Astra (81.5%), and Claude Sonnet 5.5 (79.6%) show very little differentiation. When it comes to writing correct code, the leading models can no longer be distinguished by a significant margin.
The watershed lies in engineering judgment.
In the “Thought Partnership” dimension, which assesses whether a model will, like a colleague, probe for more details and raise objections, both Claude models scored around 75%, while GPT-6.1 Sol and GPT-6 Astra dropped to 53.0% and 51.6%, respectively. In the “Communication” dimension, Sonnet 5.5 scored 72.4%, whereas Sol managed only 44.1%.
But this doesn’t mean Claude can rest easy.
In another charitable donation app, the requirement was “to allow users to split donations proportionally among different organizations.” Claude Opus 5.5 designed an entirely new front-end interface and data tables, and also wrote and successfully ran 267 unit tests and 4 browser tests on its own.
Embarrassingly, this app already supports splitting donations.
Opus 5.5 encountered a relevant implementation while retrieving code, but instead of pausing to ask questions, it started from scratch, leaving the system with two sets of overlapping, parallel logic. All tests passed green, but a seasoned engineer would never deliver code like this.
When the two cases are considered together, the large model’s failures take different forms: one is being overly obedient and failing to question absurd requirements; the other is being overly capable and failing to verify the existing status quo. If we look only at the old leaderboard of passing or failing unit tests, neither of these “production risks” would be detected.
02 “97% Accuracy Paradox”: “Small Errors” Amplified by Long Workflows
If code testing exposed a lack of engineering judgment, then GDP.xlsx and DAYJOB reveal another headache for corporate buyers: models that score highly on single-step test cases can see even minor errors dramatically amplified when applied to multi-table joins and long‑chain workflows.
GDP.xlsx contains a rent forecasting problem for a commercial real estate fund.
The workbook contains 15 worksheets, approximately 60 cell comments, and also includes an email from the asset manager requesting that all tenants whose “rents may change” over the next three years be identified—this includes those with expiring discounts and those who hold early termination rights.
Meta’s Muse Spark 1.3 identified all six tenants whose discounts have expired, even uncovering half-price promotional terms hidden in cell comments and emails. Then it stopped—leaving the “BreakClauses” sub-table on the right side of the workbook, which tracks early lease terminations, completely unopened. That section still lists three tenants eligible for early lease termination, involving approximately $14 million in projected rent.
In another product‑certification and production‑scheduling task, GPT‑6 Astra, upon seeing that both products A and B on the master table were marked “approved,” recommended launching them simultaneously—without cross‑checking the contract‑manufacturer restrictions specified on another sub‑table—and confidently asserted in its response that “the workbook does not specify the relevant suppliers,” even though the supplier names were listed on a different sub‑table.
These types of errors share a common trait: the model finds part of the answer and then assumes it has exhausted the entire problem.
In the DAYJOB evaluation, even a single-step oversight is further amplified by lengthy workflows. Each task in DAYJOB is accompanied by a set of binary scoring rubrics developed by industry experts; the median number of rubrics for the healthcare track is 47.5, while the finance track reaches as high as 57.5.
Let’s do a rough mathematical estimate:
If passing the entire task requires getting every single sub‑rule correct, and assuming that each sub‑rule is independent, then working backward from the medical domain’s highest task pass rate of 24.7% implies that the model achieves a 97% accuracy on each individual sub‑rule—based on the median number of 48 sub‑rules, 0.97 raised to the 48th power yields approximately 23%.
The above is merely an idealized illustration; in reality, the specific criteria for scoring cannot be entirely independent of one another.
However, this also intuitively illustrates one thing: in a tightly linked business process spanning nearly fifty steps, even if each step is “likely correct,” the cumulative probability of successfully completing the entire task drops to below 30%.
Whether a company dares to entrust an entire role to AI hinges precisely on the bottom line.
03 Who Scores the Examiners: When Evaluation Itself Becomes a Big Business
Since the old rankings have become distorted, are emerging benchmarking lists like Surge AI the ultimate, absolutely objective “final arbiter”?
The answer is not necessarily.
To understand the leaderboard for large models, you first need to see who’s setting the questions—and how the “examiners” make their money.
Over the past year, large-model evaluation has evolved into a highly valued business. LMArena (formerly Chatbot Arena), which initially relied on anonymous user voting to rank models, closed a $150 million Series A round in January 2026 at a $1.7 billion valuation.
Surge AI, which has just released three new rankings in a row, is LMArena’s harshest critic.
As early as December 2025, the Surge team manually reviewed 500 real‑world match votes on LMArena, stating that they disagreed with 52% of the results and strongly disagreed with 39%. For example, in one vote, the winning model claimed that “a 9‑inch round cake pan is the same size as a 9×13‑inch rectangular pan.”
Surge explains that ordinary users often spend only a few seconds glancing at the options, casting their vote for whichever answer has better formatting and a more confident tone.
Not only has Surge criticized crowdsourced blind testing, but it has also publicly criticized OpenAI’s own workplace benchmark GDPval, stating plainly in DAYJOB’s documentation that only about 20% of the GDPval scoring criteria they reviewed are directly usable.
These criticisms hit the nail on the head; the issue is that Surge AI’s core business is providing high-end data annotation, reinforcement learning environments, and expert human evaluation services to major model research labs.
According to Reuters, Surge AI’s revenue exceeded $1 billion in 2024, with a client roster that includes several leading labs, such as Anthropic. “Public automated rankings aren’t sufficient, crowdsourced voting generates nothing but noise, and long-term evaluations must rely on industry experts hired at high salaries”—this conclusion itself is highly consistent with its business model.
From another perspective, the high cost of having experts manually create questions means that the new leaderboard cannot feature a large number of questions.
For example, the sudo L7 benchmark consists of only 60 questions, with each question accounting for roughly 1.7 percentage points on average. Among models ranked in the middle of the leaderboard, those differing by just two or three percentage points would, when applied to the actual exam, typically differ by only one or two questions—indicating a significant element of randomness.
This is not an isolated case.
As early as November 2025, a paper authored by 42 researchers conducted a systematic review of 445 large-model benchmarks, pointing out that many benchmarks fail to clearly specify the capabilities they aim to measure and rarely perform statistical tests when comparing models.
Meanwhile, in BenchLM’s current methodology, which covers 229 benchmarked models, a ranking is labeled “supported” only when different types of external evidence corroborate it; all other cases are marked as “estimated.” Vendor‑self‑tested scores are listed separately and excluded from the rankings, and any “generated data” derived through interpolation from similar models is also entirely excluded.
In the review ecosystem, there is no truly “independent third party” in the strict sense. For readers and buyers, what matters more than settling on a specific ranking is first examining the exam’s underlying details: who designed the questions, how many there are, what the margin of error is, and who is footing the bill for this assessment.
04 The “Two Report Cards” of Chinese Models
Turning the focus back to China, the sense of fragmentation—“different exam papers measuring different worlds”—is even more pronounced.
Looking only at actual usage volumes, Chinese models are currently at their peak.
According to statistics disclosed on October 20 by OpenRouter, a global API routing platform, Chinese models accounted for as much as 79.18 trillion tokens on the platform in the previous week—roughly 4.5 times the volume of U.S. models (17.36 trillion). The top five spots on the weekly usage ranking were all occupied by Chinese models.
In the latest weekly ranking released on October 26, DeepSeek V4.1-Flash saw its weekly API calls surge by 801% compared to the previous week, firmly securing the top spot worldwide.
But on Surge AI’s three new exam papers, the image is completely flipped:
On the L7 benchmark, the top-performing Chinese model is Alibaba’s Qwen 3.8 Max (20.6%, ranked sixth); Zhipu GLM 5.3 scores 14.4%, Moonshot Kimi K3 scores 13.3%, and Tencent Hunyuan (Hy4 Preview) scores 12.8%.
In GDP.xlsx, GLM 5.3 stands at 21.4%, while Qwen 3.8 Max is at 20.3%. Although these figures are not far from GPT-6.1 Sol (22.0%), Kimi K3 (13.4%) and DeepSeek V4 Pro (6.9%) lag noticeably behind.
DAYJOB shows the largest gap in the medical sector: Qwen 3.8 Max is only 2%, Kimi K3 is 1.2%, while GLM 5.3 and DeepSeek V4 Pro are both just 0.4%.
Why is it that on one hand, global call volume is far ahead, while on the other, it’s nearly drawing a blank on the new long-process rankings?
An intuitive explanation is this: at present, Chinese models excel in high-frequency, low‑cost, single‑step tasks, and global developers are “voting with their feet” due to their strong cost‑effectiveness. However, when it comes to engineering tasks that require continuous planning over dozens of steps and deep integration into complex systems, they still lag behind the leading U.S. models.
Before drawing any conclusions, at least four test conditions need to be isolated:
First, the toolchain shell (Harness) is not equivalent.
On sudo L7, Claude runs in its native Claude Code, GPT runs in Codex, and Gemini runs in Antigravity; meanwhile, among Chinese models, aside from Alibaba’s use of its own Qwen Code, GLM, Kimi, and Hunyuan are all executed within third-party OpenCode wrappers. Engineers who have worked on agent development know well that an agent’s performance is highly dependent on how well it integrates with the surrounding toolchain—switch to a non-native wrapper, and its score could drop significantly.
Second, the test versions are outdated.
Surge tested the earlier Kimi K3 and DeepSeek V4 Pro; in Artificial Analysis’s latest comprehensive intelligence index, the Kimi K3.1 Pro, which was just updated in October, has already scored 55 points, only 3 points behind the top-ranked Claude Opus 5.5 (58 points); DeepSeek’s latest V4.1 Flash also achieved 45 points, a significant improvement over the V4 Pro tested (36 points).
Third is the regional and institutional barriers to exam questions.
DAYJOB’s medical‑sector tasks cover clinical care, pharmacies, payers, operations, and compliance, and are authored by experts organized through this U.S. review company. Content such as Medicare reimbursement and compliance procedures varies significantly across countries, which may not be fair to models trained primarily for the domestic market.
Fourth, there are statistical noise points in small samples.
The medical track consists of 65 questions, and the difference between 0.4% and 2% amounts to roughly half a question to one full question—statistically, this distinction is virtually negligible.
Taking all of the above variables into account, we can arrive at a more balanced assessment: in long-running tasks designed by U.S. evaluation agencies and executed on U.S.-based toolchains, Chinese models do indeed lag noticeably; however, determining how much of this gap stems from the foundation models themselves versus how much is due to testing conditions requires domestic vendors and evaluation bodies to refine the benchmark using Chinese-language workflows and native toolchains.
05 Written at the End
Stripping away all ranking controversies, the most tangible value that Xin Kaoyuan offers to corporate buyers is, in fact, cost accounting.
According to the same batch of test data disclosed by Surge on the sudo L7 blog (with slightly different pass rates in subsequent leaderboard updates), Claude Opus 5.5 and Sonnet 5.5 have pass rates of around 45%, with costs per task of approximately $10 and $14, respectively; GPT-6.1 Sol has a pass rate of 30% and a cost of just over $1 per task.
If we combine their respective pass rates and convert them into “how much it actually costs to complete one qualified task”:
Claude Opus 5.5: approximately $22;
Claude Sonnet 5.5: approximately $31;
GPT-6.1 Sol: about $3 more;
Whether $22 or $3 is more, doesn’t depend on your ranking on the list; it depends on “how much a single failure actually costs in your specific business.”
If a single mistake in Webhook code leads to order losses and compliance incidents on the production line, spending nearly $20 more for 15 percentage points of improved accuracy still feels like a steal; but if you’re running high-volume, low‑cost internal scripts where the fallout from errors is minimal, then paying almost ten times as much for the top‑tier model is nothing but a waste of computing power.
The market has already begun to price in this divergence.
Since 2026, Zhipu has raised the average selling price of its APIs by approximately 101%, turning the API business’s gross margin from -0.4% to a positive 24.6%, with no significant customer churn; after launching time-based pricing for DeepSeek, demand remains strong even when peak-hour prices are doubled.
As the benchmark‑scoring hype fades, every enterprise CTO should ask just three questions before paying to integrate an API: Does this leaderboard’s test suite resemble my real‑world business? Is the testing toolchain I’m using the same one that runs in production? And, based on the cost of failure I can afford, how much will each successful use case actually cost me?
For more exciting content, follow TMTPost on WeChat (ID: taimeiti), or download the TMTPost App.