All AI updates

Reproducible Evaluation as the Substrate for Safe Prompt Compression in Production Agents

arXiv:2610.31972v1 Announce Type: new Abstract: Evaluating conversational agents that act on live, mutable enterprise data is hard: replayed sessions drift as backend state changes, and a benchmark built today can silently go stale as the data it describes moves on. We present an offline evaluation harness for a production conversational agent that stays reproducible despite the live data it operates on changing, and that can be regenerated on demand once it goes stale. We then use this harness as the fitness function for GEPA-based prompt optimization under a correctness gate that guards against reward hacking. Applied to a production merchant-facing agent, correctness-gated compression shrinks the agent's full prompt surface by 3.5-7.6% without regressing held-out task correctness, consi

SpecMem introduces a real-time tool-call memory management mechanism, enhancing the agent’s response speed.

On October 26, 2026, arXiv published a preprint paper titled “SPECMEM: Accelerating Agentic Tool Calling via Live Memory Management” (ID 2610.31840v1), introducing a new system called SpecMem that significantly reduces tool-call latency by enabling cross-session, online, user-level memory management for tool invocations.

Why it matters：This study addresses the high latency issue in deploying current tool‑calling agents, which arises from autoregressive decoding. It proposes key techniques such as verifiable write‑back, LRU eviction, and side‑effect gating, and validates their effectiveness on real query streams involving multiple users and sessions—achieving up to 1.74× measured speedup and a 27% average increase in accepted tokens. This represents an important advance in the field of AI system engineering.

KVStream: Adaptive Parallel Storage Management for KV Caching in Large-Scale LLM Serving

arXiv:2610.31827v1 Announce Type: new Abstract: Long-context LLM serving for agentic, coding, and document workloads reuses prefixes whose key-value (KV) cache outgrows GPU and host memory, putting storage I/O on the critical path of the inference step for retrieving prefixes. A single node-local NVMe absorbs all retrievals, while the parallel file system or cloud file store that nodes already mount carries almost no inference traffic. We cast prefix retrieval as aggregate-bandwidth scheduling across these tiers and present KVStream, a KV cache engine that batches a step's reads, defers writes behind the forward pass, replicates each chunk on every tier, and lets each tier adaptively claim the next unread chunks once it is free. Our performance model shows this routing attains the aggregat

OpenHome: From Perception to Insight in Smart-Home Camera Videos

arXiv:2610.31986v1 Announce Type: new Abstract: Vision language models (VLMs) have been increasingly adopted in consumer smart-home camera systems to interpret recorded videos and provide insights. However, video clips captured from smart-home cameras are different from curated web videos dominating existing VLM benchmarks. Smart-home video clips often do not contain subtitles or narration and are captured from different viewing angles, under various lighting conditions, and in diverse scenarios. However, the capabilities of the model in this domain, under sparse visual signals and limited context, are not fully assessed by existing VLM benchmarks. To close this gap, we introduce OpenHome, a visual-only, four-option multiple-choice benchmark for evaluating smart-home video understanding. I

When and How Does Muon Precondition? The Role of Structured Gradient Noise

arXiv:2610.31728v1 Announce Type: new Abstract: Muon has become a popular optimizer for large language model training, yet its theoretical foundations remain poorly understood. In this work, we study how gradient noise geometry shapes Muon's preconditioning benefits. We first show, theoretically and empirically, that under the classical isotropic noise assumption, Muon offers no acceleration beyond a rescaling of the learning rate. This motivates us to investigate structured noise arising from mini-batch sampling in associative memory with heterogeneous task frequencies. We establish an exact balancing threshold for two tasks and extend the analysis to multiple tasks, revealing how task coverage and gradient reliability jointly shape learning. Our analysis shows that Muon can soften the de

Beyond Hotness: High-Resolution Memory Characterization of Cloud and LLM Workloads for Tiered Memory with ARM SPE

arXiv:2610.32711v1 Announce Type: new Abstract: Memory demands from data analytics and LLM workloads grow rapidly, driving up memory costs in cloud and data centers. Multi-tier memory has emerged as a promising solution to balance performance and cost. However, effectively leveraging it requires a deep understanding of workload memory behaviors. This work introduces a high-resolution memory characterization methodology for analyzing workload-specific opportunities for multi-tier memory usage. In response to the increasing deployment of ARM-based processors in data centers, we implement our methodology using the recent Statistical Profiling Extension (SPE) for ARM. By characterizing eight memory-intensive workloads from CloudSuite and LLM applications, we identify region-specific access pat

HyScale: Scaling Long-Context Training for Learned Sparse Attention

arXiv:2610.32057v1 Announce Type: new Abstract: We present HyScale, a long-context training system for learned sparse attention. Learned sparse attention is a promising architecture for million-token contexts, widely adopted in production since DeepSeek-V3.2. Selection, however, grows into the training bottleneck. For each query, it scores every previous token whereas sparse attention reads only $k$; worse, current systems materialize a full score table and repeat selection during recomputation. Our key insight is that the output of selection is not an activation, but rather a set of indices. Building on it, HyScale introduces HyTopK and selection replay. HyTopK finds each query's exact top-$k$ on the fly without a score table, while selection replay reuses forward-pass selection in recomp

The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents Bypasses Intended Workflows

arXiv:2610.31985v1 Announce Type: new Abstract: LLM agents are increasingly deployed in collaborative settings where they pursue separate tasks while sharing resources and operating under review or oversight processes. Such settings create safety risks not only when agents are malicious, but also when they are simply trying to help. Prior work has largely studied adversarial or explicitly collusive behavior; we show that benign agents pursuing separate objectives can spontaneously cooperate in ways that bypass intended workflows. We call this the \emph{helpful agent problem}. We study long-horizon shared-resource environments in which agents sometimes depend on resources owned by their peers and cross-owner changes are normally mediated by a reviewer. When review becomes unavailable, agent

Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?

arXiv:2610.32046v1 Announce Type: new Abstract: Recent \emph{test-time training} (TTT) frameworks have enabled open-weight large language models (LLMs) to outperform inference-only frameworks powered by frontier models on scientific discovery tasks. However, TTT introduces additional computational cost for online weight updates during the discovery process, making computational efficiency an important consideration when scaling to larger models. This motivates us to explore an efficient \emph{test-time adaptation} method for scientific discovery. We introduce \emph{Test-Time Memory Evolution} (TTME), a plug-and-play test-time adaptation method, as an alternative to TTT. TTME continually updates the external memory by distilling the experience to guide subsequent discovery without updating

Do AI-for-Science Agents Preserve Model Safety?

arXiv:2610.32045v1 Announce Type: new Abstract: AI-for-Science agents are increasingly being used for formulating scientific hypotheses, running autonomous experiments, and interpreting evidence. These agents are constructed by integrating a large language model (LLM) into a domain-specific agent harness, which in turn can improve the LLM's scientific capability and enable the model to orchestrate scientific tools. In this work, we ask whether model safety is preserved when an LLM is integrated in an AI-for-Science agent harness. We evaluate five agent harnesses on 50 harmful chemistry requests. Specifically, we study how the safety behavior changes under direct prompting, two simple adversarial prompting strategies, and an adaptive attack. Across four recent GPT variants, our results show

Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

arXiv:2610.32229v1 Announce Type: new Abstract: Training large language models (LLMs) in sub-8-bit arithmetic promises substantial gains in memory bandwidth and compute throughput, yet FP4 formats introduce quantization error severe enough to destabilize optimization if left unaddressed. We present ALTO, an open-source training recipe for GPT-OSS-20B, a 20-billion-parameter mixture-of-experts (MoE) model, on the MLPerf Small MoE Training benchmark under the MXFP4 microscaling format. The recipe combines established techniques from Abecassis, Felix et al. [arXiv:2509.25149] (hybrid 1D/2D block quantization, randomized Hadamard transforms, and stochastic rounding) with a simplified weight de-oscillation scheme adapted from TetraJet-v2 [arXiv:2510.27527]. On C4 validation at global batch size

EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation

arXiv:2610.32115v1 Announce Type: new Abstract: Sparse Mixture-of-Experts (MoE) is a powerful paradigm for scaling Neural Machine Translation (NMT) and managing language heterogeneity. However, practical limitations hinder its deployment on resource-constrained devices: the entire expert pool must remain in memory, and the per-token compute budget is statically fixed during training by rigid top-k routing. We introduce EDWARD-NMT (Elastic Dimensions With Adaptive Routing and Dual-consistency for NMT), a novel MoE encoder-decoder architecture designed for a train once, deploy anywhere paradigm that exposes a family of deployable sub-networks at inference time. EDWARD-NMT integrates three key innovations: (i) Matryoshka-style elastic dimensions to flexibly reduce the memory footprint; (ii) a

Query recall determines the retrieval performance of classical Arabic poetry: BM25 and dense encoders show markedly divergent behavior as query abstraction increases.

A recent study published on October 26, 2026, on arXiv (arXiv:2610.31821v1) shows that, in classical Arabic poetry retrieval tasks, the degree to which users retain fragments of poetic lines—such as whether they can recall half a line—plays a more critical role in determining retrieval performance than model capabilities. BM25 outperforms all dense encoders, including state-of-the-art models with 8B parameters, on “tip-of-the-tongue” queries, but falls significantly behind on abstract queries.

Why it matters：This study, through systematic benchmarking using a bilingual corpus, a three-index model, and a dual-evaluation protocol, for the first time quantitatively reveals the dominant influence of query abstraction on different retrieval paradigms (lexical vs. dense), and—after being validated by both human annotators and open-source weight models—the conclusions demonstrate methodological universality and empirical robustness.

Kolibri: A Sovereign European Model on the Pareto Frontier

arXiv:2610.31945v1 Announce Type: new Abstract: This report introduces Kolibri, an English-German mixture-of-experts transformer with 78.1B total parameters and 3.46B active parameters per token, released as open weights under the Apache 2.0 licence. We built Kolibri for sovereign and specialised deployment, with a particular focus on German and on regulated domains such as public administration, industry, and aerospace. To support these settings, we control the full model-development process, including data, architecture, training infrastructure, post-training, and evaluation. We train Kolibri on 24T tokens across pre-training, mid-training, and long-context extension, with German accounting for more than 20% of the data mix by bytes. To reach this share, we complement German datasets wit

In LLM evaluation, relying solely on the overall F1 score to validate automated scorers carries the risk of misleading results: a new study underscores the need to calibrate for prompt-specific differences.

An arXiv paper (2610.31678v1) points out that, in interactive evaluations of large language models, relying solely on the overall accuracy or F1 score of automated scorers can mask prompt-specific errors, leading to misjudgments of prompt effects. Empirical evidence shows that the same scorer exhibits substantial variation in its false-positive rate across different prompts (2.6%–33.3%), and a seemingly significant 9.3-point effect shrinks to 2.3 points—and becomes non-significant—under dual-LLM review.

Why it matters：This study, based on real experimental data (6 models, 180 fictional entity‑based questions, and 6 prompt conditions), reveals a widespread methodological flaw in current LLM evaluations: equating the scorer’s overall performance with its reliability on key benchmark tasks. Its central claim—that each prompt must be audited separately and uncertainty must be incorporated—rests on clear empirical evidence and offers actionable recommendations, making it highly timely (published on 2026‑10‑26) and backed by a credible source (the arXiv preprint platform).

Behaviour analysis for large language model-driven agents

arXiv:2610.32087v1 Announce Type: new Abstract: Understanding language-agent behaviour requires learning how actions are organised across a trajectory and how this organisation relates to context and outcomes. We propose a general framework that connects data-driven behavioural discovery with analysis and experimental investigation. Categories developed from recorded activity define what is measured. Analyses preserve the behaviours' positions in the process and express their relationships as interpretable hypotheses. Statistical assessment and tests in held-out contexts examine the strength and scope of the evidence. Experiments, including prompt steering, then test how behaviour and outcomes change under deliberately varied conditions. A study of iterative self-reflection illustrates how

Democratizing LLM Training with the Edge: Challenges, Systems, and the Road Ahead

arXiv:2610.32215v1 Announce Type: new Abstract: The growing computational power of modern GPU data centers has let a few industrial giants train and deploy large language models (LLMs) on massive amounts of data. Yet the sheer scale of these models and their training data makes centralized development increasingly hard to sustain, and at the same time shuts out smaller institutions that hold relevant private data but lack comparable infrastructure. In this paper, we organize our positions on distributed LLM training around the pretraining and fine-tuning paradigms, covering federated learning, split learning, and decentralized approaches, and we assess their readiness for geo-distributed settings with heterogeneous hardware and unreliable connectivity. We then present a {\em unified} syste

New research reveals serious measurement bias in LLM-based math tutoring assessments: exact-match scoring underestimates effectiveness by nearly 40 percentage points.

The paper “MisconceptDialogue: Evaluation Criteria Shape Measured Success in LLM Math Tutoring,” published on arXiv on October 26, 2026, introduces the MISCONCEPTDIALOGUE automated evaluation framework, which assesses the performance of five open-source small-parameter LLMs (7B–14B) as math tutoring models through over 5,000 rounds of dialogue. The study finds that the mainstream exact-match scoring method achieves only a 42.1% remediation success rate, whereas an expert‑calibrated LLM referee evaluates the same dialogues at 81.3%, a gap of 39.2 percentage points; moreover, this discrepancy is not accidental—four types of LLM referees show high agreement with 200 human‑annotated results, with a maximum kappa coefficient of 0.91. In addition, the models exhibit virtually no differentiation in their responses to diagnostic information (correct or incorrect concepts), indicating their limited ability to effectively leverage such diagnostic inputs.

Why it matters：This study directly addresses the core flaw in current methodologies for evaluating LLM-based educational applications: reliance on a single, rigid automated scoring system can severely distort assessments of actual competence. Its large-scale simulation experiments, multi-rater convergence validation, and empirical analysis of the breakdown in diagnostic–instructional coupling provide critical warnings and a replicable methodological benchmark for standardizing AI-driven educational assessment.

The study proposes the “Schema Consistency Index” (SCI) to assess the coherence of belief representations in language models.

On October 26, 2026, arXiv published the paper arXiv:2610.31683v1, introducing the Schema Coherence Inventory (SCI)—a 96-item forced-choice scale designed to assess whether the beliefs attributed to persona-conditioned language models genuinely structure their cognition, rather than merely eliciting emotional responses.

Why it matters：This study fills the current gap in the lack of a verifiable measurement method for the “degree of belief internalization” in language models; through validation by clinical experts, testing across multiple model backbones, and the design of target-specific criteria, it provides the first assessment framework that focuses on cognitive structures rather than emotional representations, and has open-sourced all its tools.

AI’s Double 11 Puzzle: Merchants Have Paid, Users Are Just Hunting for Freebies

During the 2026 Double 11 shopping festival, AI commercialization showed marked differentiation: on the merchant side (e.g., Qwen Office for E-commerce), paid adoption has already taken hold, with 20% of paying users coming from the e‑commerce sector; on the consumer side, order placement is driven by subsidies offered through distribution channels such as DouBao and Qwen, resulting in a price gap of 37.5%, yet user trust remains low (only 16.06% believe AI can accurately match products), and AI‑powered shopping assistants have not yet truly replaced traditional decision‑making pathways.

Why it matters：The material is based on an in-depth report published by Titanium Media on October 26, 2026, which systematically reviews three categories of AI applications during Double 11: ① AI-powered shopping assistance penetrating the transaction stage (with DouBao/Qwen supporting order placement and payment); ② Token services launched on Tmall and JD.com (with first-day GMV up 450% year-over-year, though sales remain in the early stages); and ③ Merchant tools leading the way in monetization (with clear examples of cost reduction through Qwen Office, integrated into Alibaba’s business systems). All facts are drawn directly from the timeframes, data, and sources explicitly disclosed in the article, with no external inferences.