Search headlines, summaries and articles to connect the dots.
The Metal kernel for ggml_tri has been updated to use a select operation instead of scaling by comparison, aligning its behavior with the CPU kernel. This change ensures that values outside the triangle are set to zero rather than becoming NaN when encountering ±inf. A new test, test_tri_special, validates this fix across all four triangle types, confirming it passes with the update but failed previously.
A new fast path for d_state=64 has been added to the SSM_SCAN kernel in ggml-cuda, improving generation throughput for Zamba2-7B-Instruct from 11 to 15.1 tok/s—approximately a 35% increase—by eliminating CPU fallbacks for this state size. The change was verified with real-world Zamba2 hyperparameters and maintains correctness, though a separate 28% of scan calls still fall back to CPU due to an undiagnosed issue.
The ggml library has been updated to make tied selections in the radix Top-K algorithm deterministic on CUDA and Metal backends by replacing atomic compaction loops with a deterministic scheme using simdgroup scans and coalesced memory accesses. This change ensures byte-identical outputs across runs, improves performance by 20-45% on M5 Ultra hardware, and enables deterministic testing for Metal and ROCm. Redundant CUDA kernel resets were removed, and Vulkan/Metal changes were reduced or skipped.
A fix has been implemented to prevent crashes in the command-line interface when encountering invalid UTF-8 in prompts or files, with assistance from Claude. This update is part of the b12177 release of llama.cpp, which includes various binary distributions for macOS, Linux, Windows, Android, and other platforms.
A fix for shared memory bank conflicts in the lightning indexer has been implemented for CUDA support in llama.cpp version b12185, improving performance and stability on CUDA-enabled devices.
A change has been made to add the 'readonly' attribute to decodeBuf declarations in the Vulkan backend, with potential performance improvements expected in future driver versions.
The server update addresses slot processing order by ensuring parent slots are visited before child slots, preventing delays caused by child tasks landing on lower slot IDs. This change ensures all child tasks join the same batch after being woken by their parent. Additionally, new test tags have been introduced for monitoring: __TEST_TAG_WAIT_PARENT_SLOT__ for child slot waits, __TEST_TAG_PROMPT_CACHE_RESTORE__ for prompt cache restores, and __TEST_TAG_PROMPT_CACHE_FULL_RESTORE__ when a cached prompt is fully contained in a new one based on token counts.
The Vulkan implementation of the Fast Walsh-Hadamard Transform (FWHT) has been enhanced to support block widths up to 8192, using shared-memory shaders for larger blocks, and now includes F16 input capability via shader-float16, with pipelines created only when device limits allow. This update aligns Vulkan with CUDA's capabilities and improves efficiency by eliminating the need for F32-to-F16 conversions.
The llama.cpp project has been updated to version 0.9.0, with new releases available for multiple platforms including macOS, Linux, Windows, Android, and iOS. The update includes support for various backends such as CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon-specific hardware. Some builds are marked as disabled or experimental, and detailed setup guides are provided for specific configurations.
