Make a Robot Arm Feel Weightless with NVIDIA Holoscan 5
Picture guiding a robot arm by hand while its motors supply just enough torque to support its weight and keep the robot upright. This simple interaction, a...
AI NEWS JOURNAL
Search AI news
Search headlines, summaries and articles to connect the dots.
ReKV: Overlapping Quantized KV Transfer and Suffix Recomputation
arXiv:2611.08770v1 Announce Type: new Abstract: Prefill--decode disaggregation moves a prompt's key--value (KV) cache between workers before the first token can be generated. Reducing this handoff to either cache compression or recomputation misses a useful choice: which layers to transmit, at what precision, and which layers to execute again on the decode worker. ReKV transfers a quantized prefix KV cache and one BF16 boundary state, then reconstructs a contiguous suffix while the prefix is in flight. In a model-shaped synthetic benchmark on cross-node H100s, the Q8 configuration meeting separately evaluated quality criteria reduces 8K handoff latency by $2.36\times$, $2.84\times$, and $3.65\times$ over BF16 transfer for Qwen2.5-7B, Qwen2.5-3B, and Qwen3-8B. It is also faster than the best measured transfer-only or recompute-only action at those points. A separate real-weight BF16 recomputation experiment reduces prefill-to-token-return time by $1.16$--$1.63\times$. The preferred action changes with prompt length and link bandwidth.
b12191
The latest update to llama.cpp introduces enhanced MoE (Mixture of Experts) cache functionality, enabling computation of cache misses on the CPU with a configurable CPU share. A new tuning script, moe-cache-tune.py, automatically determines optimal cache size and CPU share settings based on system performance. The update also improves scheduling efficiency, removes redundant sync operations, and refactors code for better maintainability, while maintaining compatibility across multiple platforms including macOS, Linux, Windows, and Android.
b12175
A fix was implemented to override the hardware_concurrency value in CI tests, addressing an issue where test-llama-context and test-backend-ops hung due to incorrect thread count estimation on 8-CPU runners. The solution sets default values of 8 for TEST_ARG_HARDWARE_CONCURRENCY and 4 for TEST_ARG_PARALLEL_WORKERS specifically for the cpu-x64-high-perf configuration.
b12180
The release adds the Kimi-K3.1-Pro text model to llama.cpp, replacing full-attention MLA layers in K3 with DSA (Dynamic Sparse Attention) layers featuring a lightning indexer. The indexer uses an additive mask over the cache and reuses sparse flash attention paths, while preserving existing KDA layers and MoE components. The model's architecture includes four indexer tensors per DSA layer, hybrid memory for caching, and maintains quantization integrity by excluding indexer parameters from quant overrides. Verified against HF checkpoint with identical PPL beyond context length, and supports multiple platforms including macOS, Linux, Windows, Android, and iOS.
b12182
The latest release of llama.cpp (v1.2182) includes a change to raise the compute view limit for buffers with few tensors in the meta backend, along with reverting the deepseek41 test model back to a 6-layer configuration. The release provides precompiled binaries for multiple platforms including macOS, Linux, Windows, Android, and iOS, supporting various backends such as CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon-specific hardware. Some openEuler builds remain disabled.
How NVIDIA Open Source Tools Are Helping Researchers Build Next-Generation 6G Networks
A year ago, NVIDIA open sourced its Aerial wireless software, joining Sionna and a broader suite of open source tools for accelerating next-generation 6G network development. Together, they give researchers a foundation for testing ideas, comparing results and building on each other’s work — connecting simulation and AI training with experiments on real over-the-air wireless […]
b12140
The latest update to llama.cpp (v1.2140) adds support for the `truncate=false` parameter in YaRN, enabling more accurate frequency interpolation for models like gpt-oss and certain PLaMo-3 variants by preserving non-integer interval boundaries. The change includes updates to GGML operations and CPU backend handling, with CUDA backend support, new tests, and refactoring of RoPE helpers into internal headers. Default behavior remains `truncate=true` for backward compatibility.
b12172
The update introduces batched tensor functions `get_tensor_n` and `set_tensor_n` for WebGPU, along with fixes for the editor-config checker and removal of an outdated comment.
b12118
The update enhances OpenCL performance for decode operations by optimizing workgroup usage for q4_K gate up and GLU, and adds support for Flash Attention (FA) in various configurations including GQA 2, 6, and 16 with different head sizes, improving efficiency on specific hardware setups.
b12122
The update introduces optimizations for CUDA by fusing NVFP4 gate/up operations and GLU within MMQ, honoring requested precision in fused gate/up MMQ, sharing scale checks for NVFP4, skipping unused FP4 kernels, testing with W4A8 configurations without scales and with batching, and adding checks for gate_scale before loading fused NVFP4 gate scale.
Configure, Certify, and Operate Kubernetes GPU Clusters with NVIDIA DSX OS
One underperforming GPU can slow an entire distributed training job. The node may pass basic health checks and show normal utilization, leaving operators to...
How NVIDIA Uses AI Agents to Streamline AI Factory Operations
As GPU clusters scale, operations span compute, network, hardware, and scheduling. Site reliability engineers (SREs) often have to examine multiple dashboards...
NVIDIA CUDA 13.5 Brings NVIDIA Vera Rubin GA Support
As artificial intelligence and accelerated computing continue to push the boundaries of what's possible, the latest NVIDIA CUDA release arrives to unlock...
Performance Portability of Out-of-Kernel Optimizations on GPUs: A Molecular Docking Case Study
arXiv:2611.06207v1 Announce Type: new Abstract: Achieving performance portability is challenging for irregular scientific applications built as repeated multi-kernel GPU pipelines, because throughput depends on out-of-kernel decisions as much as on kernel efficiency. Using molecular docking as a representative case study, we propose an out-of-kernel tuning methodology that treats batching policy, worker concurrency, and per-worker memory allocation as explicit optimization parameters, replacing backend-dependent occupancy-driven choices with a more portable execution strategy based on explicit resource bounds. We implement this methodology in the muDock mini-app under a manager/worker execution model where multiple host workers share a GPU through independent streams. We then evaluate the methodology on NVIDIA, AMD, and Intel GPUs across CUDA, HIP, and SYCL implementations. Across all evaluated platform/backend pairs, the memory-bounded formulation exceeds the peak throughput of the occupancy-guided baseline. The fully tuned methodology improves the end-to-end throughput on all tested configurations, achieving an average speedup of 1.53x over the baseline.
b12096
The update introduces the TOP_K_MASK operation for Hexagon, simplifying the kernel and eliminating redundant host configuration.
b12098
The llama.cpp release b12098 includes accurate support for Qwen3-VL video prompt format and provides precompiled binaries for multiple platforms including macOS, Linux, Windows, Android, and openEuler, with various backend options such as CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon-specific optimizations. The release also features a standalone UI package and detailed setup guides for specific hardware configurations.
b12110
The llama.cpp project has been updated to version 0.9.0, with new releases available for multiple platforms including macOS, Linux, Windows, Android, and iOS. The update includes support for various backends such as CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon-specific hardware. Some builds are marked as disabled or experimental, and detailed setup guides are provided for specific configurations.
LiquidCP: Decoupling Attention Tasks from Communication and Kernel Boundaries in Context Parallelism
arXiv:2611.04401v1 Announce Type: new Abstract: Context parallelism (CP) partitions long sequences across GPUs for LLM training and inference. For variable-length training and full prefill, attention-balanced placement can leave token loads uneven, while ring execution repeats overhead at each computation and communication step. In stage-serialized decode CP (DCP), long-running tasks delay dependent work and leave substantial GPU hardware idle. We propose LiquidCP, which unifies training and inference through a common model of Transfer, Compute, and Reduction tasks with tile-level dependencies, decoupling attention tasks from communication and kernel boundaries. For training and full prefill, MegaRing incorporates BuddyRing placement to balance attention and token loads, and coalesces ready KV regions during forward attention to amortize overhead. For decoding and mixed batches, MegaDCP selects history-KV splits by jointly modeling the completion of attention and its dependent reductions. A shared persistent GPU-resident engine executes both designs. Compared with the fastest baseline per configuration on eight H100 GPUs, MegaRing achieves geometric-mean core-attention speedups of $1.23\times$ forward and $1.18\times$ backward, and accelerates a Transformer layer by $1.08$--$1.16\times$. MegaDCP achieves geometric-mean core-attention speedups of $1.20\times$ for decode-only and $1.35\times$ for mixed batches. In vLLM serving with KV prefix cache, it achieves an average time-to-first-token speedup of $1.22\times$.
vLLM Delivers Day-0 Support for Kimi K3.1 Pro
vLLM supports Kimi K3.1 Pro on day 0: sparse attention on a KDA hybrid, a packed KV cache with FP8 or NVFP4 latents and MXFP4 index keys, token-sharded prefill, decode context parallelism, and fused low-latency decode.