Search AI news

Make a Robot Arm Feel Weightless with NVIDIA Holoscan 5

Picture guiding a robot arm by hand while its motors supply just enough torque to support its weight and keep the robot upright. This simple interaction, a...

b12202

The llama.cpp v1.220.2 release introduces enhanced BF16 precision support across multiple components, aligning with reference implementations by moving specific operations like GLM4 family projections, MiMoVL vision matmuls, and Qwen3-TTS speaker encoder conv1d to BF16 where appropriate. Backends such as CUDA, Hexagon, OpenCL, WebGPU, and Vulkan now uniformly handle GGML_PREC_BF16 similarly to F32, while CPU, SpaceMIT, and ET backends extend flash attention support to BF16 with F32 accumulation. Flash attention tests now include BF16 cases, and the KQ multiplication in Llama is updated to request BF16 precision. A TODO remains in clip.cpp for future work.

b12182

The latest release of llama.cpp (v1.2182) includes a change to raise the compute view limit for buffers with few tensors in the meta backend, along with reverting the deepseek41 test model back to a 6-layer configuration. The release provides precompiled binaries for multiple platforms including macOS, Linux, Windows, Android, and iOS, supporting various backends such as CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon-specific hardware. Some openEuler builds remain disabled.

LowRAM: Low-Rank Keys and Amortised Values for Composable KV Cache Compression at Test-Time

arXiv:2611.07025v1 Announce Type: new Abstract: Autoregressive decoding relies on the KV cache, whose memory footprint grows linearly with context length and caps batch size and throughput. Existing post-training methods exploit redundancy across tokens, layers, and feature dimensions, or lower numerical precision. We identify a complementary axis: within each attention head, values are largely predictable from their corresponding keys. We exploit this with value cache amortisation, which fits a lightweight per-head predictor at test time and stores residuals only for the tokens it reconstructs worst, giving $4\times$ near-lossless value compression. The predictor can be initialised by meta-learning for substantially reduced prefill overhead, or fitted from scratch in settings that allow a one-off cost, such as multi-query inference. We pair value amortisation with cross-layer low-rank key compression to produce LowRAM, an asymmetric compression pipeline that stays close to lossless across three model families. By reconstructing only a selected subset of KV pairs at each decoding step, LowRAM can keep decoding overhead minimal and achieve up to $5\times$ higher peak decode throughput at $4\times$ compression. LowRAM further composes with quantisation and eviction, reaching $26.9\times$ compression at $94.5\%$ of baseline performance on long contexts. These results show that treating keys and values as structurally distinct enables efficient, aggressive, and composable KV cache compression.

Migrating vLLM to the LibTorch Stable ABI

How vLLM migrated its CUDA build and compiled dependencies to the LibTorch Stable ABI, enabling compatible PyTorch upgrades without rebuilding vLLM.

Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS

One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to...

vLLM Delivers Day-0 Support for Kimi K3.1 Pro

vLLM supports Kimi K3.1 Pro on day 0: sparse attention on a KDA hybrid, a packed KV cache with FP8 or NVFP4 latents and MXFP4 index keys, token-sharded prefill, decode context parallelism, and fused low-latency decode.

b11944

server: expose the reasoning effort levels of the chat template (#31271) * server: expose the reasoning effort levels of the chat template The capability probe renders a user turn for each candidate level (minimal, low, medium, high, xhigh, max) and for an unknown level. A level is honored when it renders without raising and differently from the unknown level, which shows the template fallback. Levels sharing a render are aliases and the highest one is kept. The caps serialize to JSON, and chat_template_caps in /props reports the list as reasoning_effort_levels, from lowest to highest. * jinja: keep the caps free of the json dependency The caps keep reporting their flags as a map, and the effort levels stay a plain member. common_chat_templates_get_caps() builds the JSON the server reports

Tsinghua University and The Chinese University of Hong Kong jointly open-source AKV: an engineering service system for KV caching tailored to LLM agents.

On October 26, 2026, researchers from Tsinghua University and the Chinese University of Hong Kong open-sourced AKV—a KV cache engineering server built on SGLang and specifically designed for large language model (LLM) agents. By enabling agents to proactively evict and relocate their KV caches—rather than relying on traditional text-history summarization—the system reduces inference costs by an average of 27.7% across four agent benchmarks and improves task scores by 1.7 points on average. The code is licensed under the Apache-2.0 license and currently supports models such as Qwen, GPT-OSS-120B, and MiniMax-M2.7.

Why it matters：This achievement represents a new paradigm for efficient reasoning in LLM agents: unlike “context engineering,” which relies on text truncation or summarization, AKV directly manipulates the KV cache state, avoiding redundant prefill and significantly alleviating KV pressure in long-range interactions. Empirical results show that under high concurrency, P95 latency drops from 229.9 seconds to 30.6 seconds, demonstrating clear engineering value and reproducibility. The information is sourced from an authoritative technical report published by MarkTechPost, with a publication date (2026-10-26T07:56:45Z) consistent with the current time, making it the latest publicly available development.

Tool support

Ollama now supports tool calling with popular models such as Llama 3.1. This enables a model to answer a given prompt using tool(s) it knows about, making it possible for models to perform more complex tasks or interact with the outside world.

Llama 3 is not very censored

Compared to Llama 2, Llama 3 feels much less censored. Meta has substantially lowered false refusal rates. Llama 3 will refuse less than 1/3 of the prompts previously refused by Llama 2.