本文へ移動
AI NEWS JOURNAL
llama.cpp releasesAI NEWS / JOURNAL

b12543

The update introduces a new tensor API conv_2d kernel for Metal, enabling split-K optimizations across all chips, including M5. It improves efficiency by utilizing tensor cores, reducing idle GPU time during long reductions. The kernel now splits reductions into partial sums stored in buffers, with a follow-up kernel summing them deterministically. Address math is optimized using 32-bit byte offsets, reducing computational overhead. A fallback to the generic kernel is implemented for images exceeding 4 GiB. Performance benchmarks show significant speed improvements on M5 Max devices, with reduced execution times across various convolution configurations.

翻訳サービスが停止中のため原文を表示しています。

<details open> metal : CONV_2D tensor API for M5, split-K for all chips (#32542) * metal: add a tensor API conv_2d kernel and split long reductions The conv_2d matmul kernel runs on simdgroup matrices only, and gives each threadgroup the whole reduction of its output tile. On M5 this leaves the tensor cores unused, and convolutions with few output positions and a long reduction (IC * KW * KH in the thousands) run a handful of threadgroups on an otherwise idle GPU. Add kernel_conv_2d_mm_tensor for devices with the tensor API: like kernel_mul_mm, it gathers the patches of 64 output positions into threadgroup memory as the left operand and reads the kernel rows from device memory as the right operand. Devices without the tensor API keep the simdgroup kernel. Both kernels can now split the reduction until there are enough threadgroups: each split writes its partial sums to a buffer placed after the output, as flash attention does, and a small kernel adds them in a fixed order, so the result stays deterministic. CONV_2D is marked as an op whose allocation may exceed its output size. test-backend-ops -o CONV_2D passes, with and without GGML_METAL_TENSOR_DISABLE. CONV_2D, F16 kernel, us/run on an M5 Max: [19,19,256,16] k [4,4,256,4096] 19008 -> 5235 [16,16,128,8] k [3,3,128,512] 333 -> 124 [1536,1536,32,1] k [3,3,32,64] 12086 -> 6512 [33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33 [2054,1,768,1] k [7,1,768,768] 3161 -> 699 * metal: walk the conv_2d patches incrementally, size splits by output The patch gather of both conv_2d matmul kernels spent most of its time on address math: 64-bit multiplies, divisions and four bound checks per value. Walk the reduction index with 32-bit byte offsets inside the image, updated by additions, with one unsigned compare per bound; the host checks that an image fits. The tensor kernel no longer gathers the reduction indices past the end of its split. The number of splits now targets about 1M partial sums, with at least 3 reduction steps per split, instead of counting the tiles of one kernel. CONV_2D, F16 kernel, us/run on an M5 Max: [19,19,256,16] k [4,4,256,4096] 4998 -> 4512 [1536,1536,32,1] k [3,3,32,64] 6721 -> 5354 [4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236 [2054,1,768,1] k [7,1,768,768] 1050 -> 871 with GGML_METAL_TENSOR_DISABLE: [4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283 * nit: share the CONV_2D simdgroup count between host and kernel kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel constant and res.nsg, so the dispatch and the kernel cannot drift apart. * metal: fall back to the generic conv_2d kernel for images past 4 GiB The matmul conv_2d kernels walk the patches with 32-bit byte offsets inside an image. Instead of asserting on nb13, compute the largest offset actually read from ne/nb, which also covers strided views of the input, and route images that do not fit to the generic kernel. * cont : comments Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> </details> **Website:** - <https://llama.app> **Attestations:** - <https://github.com/ggml-org/llama.cpp/attestations/69742362> **macOS/iOS:** - [macOS Apple Silicon (arm64)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-macos-arm64.tar.gz) - macOS Apple Silicon (arm64, KleidiAI enabled) [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23780) - [macOS Intel (x64)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-macos-x64.tar.gz) - [iOS XCFramework](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-xcframework.zip) **Linux:** - [Ubuntu x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-x64.tar.gz) - [Ubuntu arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-arm64.tar.gz) - [Ubuntu 24.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-s390x.tar.gz) - [Ubuntu 26.04 s390x (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-26.04-s390x.tar.gz) - [Ubuntu 26.04 s390x (zDNN)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-zdnn-s390x.tar.gz) - [Ubuntu x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-vulkan-x64.tar.gz) - [Ubuntu arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-vulkan-arm64.tar.gz) - [Ubuntu x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-cuda-12.8-x64.tar.gz) - [CUDA 12.8 libraries](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-b12543-bin-ubuntu-cuda-12.8-x64.tar.gz) - [Ubuntu x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-cuda-13.5-x64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-b12543-bin-ubuntu-cuda-13.5-x64.tar.gz) - [Ubuntu arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-cuda-13.5-arm64.tar.gz) - [CUDA 13.5 libraries](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-b12543-bin-ubuntu-cuda-13.5-arm64.tar.gz) - [Ubuntu x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-rocm-10.0-x64.tar.gz) - [Ubuntu x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-openvino-2026.4.1-x64.tar.gz) - [Ubuntu x64 (SYCL FP32)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-sycl-fp32-x64.tar.gz) - [Ubuntu x64 (SYCL FP16)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-sycl-fp16-x64.tar.gz) - [Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-linux-arm64-snapdragon.tar.gz) - [setup guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/linux.md) **Android:** - [Android arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-android-arm64.tar.gz) - [Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-android-arm64-snapdragon.tar.gz) - [setup guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/backend/snapdragon/README.md) **Windows:** - [Windows x64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-cpu-x64.zip) - [Windows arm64 (CPU)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-cpu-arm64.zip) - [Windows arm64 (OpenCL Adreno)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-opencl-adreno-arm64.zip) - [Windows x64 (CUDA 12)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-cuda-12.4-x64.zip) - [CUDA 12.4 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-bin-win-cuda-12.4-x64.zip) - [Windows x64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-cuda-13.5-x64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-bin-win-cuda-13.5-x64.zip) - [Windows arm64 (CUDA 13)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-cuda-13.5-arm64.zip) - [CUDA 13.5 DLLs](https://github.com/ggml-org/llama.cpp/releases/download/b12543/cudart-llama-bin-win-cuda-13.5-arm64.zip) - [Windows x64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-vulkan-x64.zip) - [Windows arm64 (Vulkan)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-vulkan-arm64.zip) - [Windows x64 (OpenVINO)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-openvino-2026.4.1-x64.zip) - [Windows x64 (SYCL)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-sycl-x64.zip) - [Windows x64 (ROCm 10.0)](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-win-rocm-10.0-x64.zip) **openEuler:** - [DISABLED](https://github.com/ggml-org/llama.cpp/pull/23705) - openEuler x86 (310p) - openEuler x86 (910b, ACL Graph) - openEuler aarch64 (310p) - openEuler aarch64 (910b, ACL Graph) **UI:** - [UI](https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-ui.tar.gz)

出典

原文を読む ↗

関連テーマ

← すべてのニュースへ