Skip to content
AI NEWS JOURNAL
AI, IN PERSPECTIVE

Search AI news

Search headlines, summaries and articles to connect the dots.

Newest first

Agents are users, so your software needs evals

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These tests run multiple trials, score outcomes, and help identify failures across steps like searching cafes, checking budgets, and placing orders. Using frameworks like Harbor, teams can define tasks with specific instructions, environments, verifiers, and known-good solutions. Metrics such as success rate, Pass@k, and Pass^k reveal how reliably agents perform, while trace analysis uncovers root causes of failure. The process includes setting a goal based on observability, customer feedback, or roadmap needs; defining measurable metrics; designing task sets covering common and edge cases; running evaluations at scale using tools like Vercel Sandbox and AI Gateway; and analyzing traces to iterate improvements. Automated trace analysis enables scaling, and repeated evaluation supports continuous improvement, regression detection, and product optimization.

Vercel Sandbox starts up to 4x faster

Vercel Sandbox startup times have improved significantly outside the Washington, D.C. (iad1) region by leveraging regional caching for reused image data, reducing initialization time by up to 75% in Paris and from 3.5s to 1.05s in Seoul, with smaller gains in regions closer to the origin due to shorter distances.

WTF are evals? How to test agents and the software they use

Agents interact with software through a funnel similar to humans: Discovery, Selection, Access, and Success. To measure whether agents complete user tasks correctly, end-to-end (E2E) evaluations are essential. These evaluations assess the full sequence of steps an agent takes, ensuring outcomes meet user requirements. Using frameworks like Harbor, developers can design tasks with defined environments, verifiers, and known-good solutions, then run trials across variants to measure success rate, pass@k, pass^k, efficiency, time, and cost. Analyzing traces reveals root causes of failures, enabling targeted improvements. Iterative evaluation helps catch regressions, test changes, and guide product development, turning evaluations into ongoing infrastructure for building better agent-driven products.

How Zapier turns ideas into working software in hours with Vercel

Zapier leverages Vercel to rapidly develop interactive partner demos in under two hours, build internal applications like Marketing Hive and Marketing OS in a day each, and securely share sensitive data through SSO-protected deployments. The platform enables non-developers to deploy applications, supports AI orchestration via AI Gateway, and facilitates scalable, secure code execution with 3.3 million sandboxes created monthly. Vercel's speed, security, and ease of use have become central to Zapier’s innovation across engineering, marketing, and sales teams.

Introducing Geist Serif

Vercel has launched Geist Serif, a high-contrast display serif typeface designed as part of the Geist family, which includes Geist Sans, Geist Mono, and Geist Pixel. Built on the same foundational principles as its predecessors, Geist Serif features fine serifs, sharp hairlines, and a variable axis called Element Shape (ELSH) offering four dot variants: Regular, Square, Triangle, and Heart. It maintains consistent x-height and cap height with Geist Sans for seamless integration across layouts. The font supports 352 Latin-script languages, includes 894 glyphs, and is available via npm and Vercel’s font platform. Future updates will include additional weights, an italic variant, Cyrillic support, and availability on Google Fonts. The release marks a shift toward editorial and fashion-inspired design, honoring print traditions while leveraging modern screen capabilities.