Search headlines, summaries and articles to connect the dots.
Property Finder found that 96.8% of their vision-AI evaluation labels were inherited from the incumbent model, hiding its true accuracy. See how they built unbiased ground truth, applied evolutionary prompt optimization, and selected a more accurate, cost-effective model on Amazon Bedrock, cutting cost by 86% across 11 image-quality attributes.
In this post, we show how Exowatt used Amazon SageMaker AI serverless model customization to build a repeatable AI pipeline that produces task-specific fine-tuned models for two deployment targets, edge and cloud, achieving higher accuracy than frontier models at up to 49x lower cost.
Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA...
Mirendil released Fern 0.1, its first model, pretrained from scratch with a 10M-token context window. It targets agentic AI R&D work, scores 100% on RULER single needle at 10M, but trails frontier models on agentic coding. The post Mirendil Launches Fern 0.1, Its First Model, With 10M-Token Context appeared first on MarkTechPost.
Old benchmarks approach perfect scores, yet new benchmarks consistently fall short in real-world performance. As large-model evaluation itself has become a business, companies are shifting their selection criteria from ranking-based assessments to cost‑benefit analyses.
Baidu has released Unlimited OCR 2, an MIT-licensed 3B OCR model that parses hundred-page documents in one request. Its SR-SWA attention keeps per-step decoding cost nearly flat, scoring 77.1 on MarathonDocBench The post Baidu Open-Sources Unlimited OCR 2 for Hundred-Page Document Parsing appeared first on MarkTechPost.
Data Culture built a 3-step evaluation framework that took one AI analytics deployment from 0% accuracy to 94.7%.
When you’re speaking to a device, you expect text to appear as you talk. That makes streaming automatic speech recognition (ASR) a different challenge from batch transcription, where the model... The post Optimizing Streaming Speech Recognition on Arm® Edge Devices with ExecuTorch appeared first on PyTorch.
Nace AI has released Drex for Search, bringing its Drex decision models to document search for AI agents. Nace reports 92.05% on BrowseComp-Plus with 12.6 search calls, and fewer agent turns on OfficeQA Pro. The post Nace AI Releases Drex for Search for AI Agents: 92.05% on BrowseComp-Plus With 12.6 Search Calls appeared first on MarkTechPost.
Author | Yu Hang Yuan Editor | Jing Yu On the morning of October 23, Kimi officially posted a 24-second video on X and its Video Account. The scene shows a retro‑style vintage computer, with Kimi Code running on the screen and lines of Three.js code being typed into the terminal, generating a complex 3D scene. Keen-eyed netizens quickly spotted the answer in the model bar at the top left: “Model: Kimi K3.1.” Kimi’s official teaser video for K3.1 | Image source: X @Kimi_Moonshot Using an antiquated machine to run some of today’s most cutting‑edge 3D code—this stark contrast itself conveys what Moonshot AI is trying to say. A dozen hours later, just after midnight on October 24, right on Programmer’s Day, Kimi K3.1 Pro officially launched, accompanied by a Highspeed version that’s two to three times faster. The company promises that the full model weights will be released before November 4. More than three months have passed since the previous generation, K3, was unveiled. In those three months, Moonshot AI has experienced more than most companies see in an entire year. 01 “More for the same price”—what’s new in 0.1? First, let’s clarify what K3.1 Pro is. It isn’t an entirely new model. Moonshot AI explicitly states that K3.1 Pro is a post‑training variant of K3; the underlying architecture remains unchanged—still a hybrid expert architecture with 2.8 trillion total parameters and 104 billion activation parameters, still featuring a 1 million‑token context window, and still natively supporting image and video inputs. What has changed is the post‑training phase. According to the company, they’ve invested “significantly increased post‑training compute power,” conducting large‑scale joint reinforcement learning across hundreds of downstream tasks. To support this, they completely reengineered the underlying infrastructure—including the reinforcement learning framework, sandbox environment, and long‑context reasoning capabilities. In essence, the same brain has been retrained from scratch on how to perform its tasks. The results are reflected in benchmark scores. On Terminal‑Bench 4.0, which evaluates coding agents, K3.1 Pro achieved 65.7%, surpassing GPT‑6 Astra’s 58.2% and falling short of Claude Opus’s 66.4% by less than one percentage point. As for the previous generation, K3…
AI startup Recursive has unveiled Flint, an open-source coding model that it claims was developed primarily using its in-house AI research system, Eureka Machine. With just 40 billion parameters and only 2.5 billion activations in practice, the model can run offline on a standard laptop—such as an M4 Pro MacBook—at an inference speed of 131 tokens per second, outperforming most models of comparable size on benchmarks like SWE-Bench.
NVIDIA has launched Kumo Relational, a pre-trained relational foundation model that can perform predictive tasks—such as churn prediction, fraud detection, and demand forecasting—directly on enterprise‑grade relational databases, without the need for task fine-tuning, feature engineering, or table flattening. In 41 benchmark tests spanning 15 databases, it outperformed supervised models like XGBoost across the board, and on the SAP SALT enterprise benchmark, its MRR metric surpassed AutoGluon by 8%. The model supports databases with up to 500 billion rows and enables zero‑code predictive queries via Predictive Query Language (PQL).
IBM Research has introduced the ALTK-Evolve method, which leverages GPT-4.1 to automatically extract structured behavioral guidelines—such as identity verification and refund confirmation—from its own task trajectories. These guidelines are then injected into the ReAct agent system via a purely prompt-engineering approach, significantly improving the accuracy and reliability of customer-service tasks without updating model weights or relying on human annotations. Its retail branch achieved a Pass⁴ score of 67.5%, ranking first on the τ²-bench leaderboard as of October 12, 2026, surpassing all state-of-the-art models and fine-tuned systems.
Learn how OpenAI is improving ChatGPT across low-resource programming languages, with new evaluations for software engineering, SQLite, and COBOL migration.
We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post ReviewBench: An open benchmark for AI code review appeared first on The GitHub Blog.
MentalHealthBench is an expert-informed benchmark for evaluating helpful and safe AI responses across realistic mental health conversations.
How UK AISI and EvalEval Are Making Benchmark Results Reproducible
OpenAI outlines a path to shared global AI standards, calling for coordinated evaluation, reporting, and governance to improve safety.
Piloting the world's first double-blind AI evaluations
