Search headlines, summaries and articles to connect the dots.
A new fast path for d_state=64 has been added to the SSM_SCAN kernel in ggml-cuda, improving generation throughput for Zamba2-7B-Instruct from 11 to 15.1 tok/s—approximately a 35% increase—by eliminating CPU fallbacks for this state size. The change was verified with real-world Zamba2 hyperparameters and maintains correctness, though a separate 28% of scan calls still fall back to CPU due to an undiagnosed issue.
The ggml library has been updated to make tied selections in the radix Top-K algorithm deterministic on CUDA and Metal backends by replacing atomic compaction loops with a deterministic scheme using simdgroup scans and coalesced memory accesses. This change ensures byte-identical outputs across runs, improves performance by 20-45% on M5 Ultra hardware, and enables deterministic testing for Metal and ROCm. Redundant CUDA kernel resets were removed, and Vulkan/Metal changes were reduced or skipped.
Optimized TOP_K performance on Hexagon backend by skipping bitonic sort for tiles whose maximum cannot surpass the k-th best value, replacing tile sort with top_k_scan() to reduce 256-float blocks using HVX max-reduce, and inserting only surviving elements into a sorted k-entry list. Supports k values from 2 to 64 and row widths over 1024 in both single-row and multi-row paths. NaN treated as -INF, ties preserve first-seen index. Eliminated scalar reads from VTCM in hex-top-k.
The ggml library now checks view bounds based on the actual bytes reached by strides, improving accuracy for strided views. Views are required to start from contiguous elements along dimension 0, ensuring correct data access. Convolution windows in delta_net are created with all rows at once to prevent stride-related issues.
ggml : relax scale to non-contiguous views (#31146) * ggml : relax scale to non-contiguous views SCALE requires src0 to be 1D-contiguous. Relax this to dim-0 contiguity with arbitrary row strides (ggml_is_contiguous_rows), same as the unary ops in the Metal backend. - ggml: relax the assert in ggml_scale_impl - cpu: address src0 rows through their byte strides (f32, bf16) - cuda: stride-aware kernel, fastdiv row decomposition - metal: support bf16 scale (guarded unary instantiations) - sycl: keep the gate at contiguous, the kernel is flat f32 Add view variants to test_scale. Verified on CPU, Metal and CUDA. Assisted-by: dsh:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (RTX 5090 32GB, DDR5 128GB) * cuda : skip the softcap fusion on non-contiguous views The fused softcap kernel reads src and w
