How FlyDSL MonoKernels keep agents responsive with high-interactivity decode in vLLM on AMD Instinct GPUs, using model-specific fusion, data-ready handoffs, and specialized-decode integration.
PyTorch Foundation Ambassadors bring people together to learn, share technical knowledge and contribute to PyTorch Foundation projects and the wider open-source AI ecosystem. In this November roundup, we highlight some... The post PyTorch Foundation Ambassador Community Highlights November 2026 appeared first on PyTorch.
Over two days at COEX, more than 2,000 attendees converged for AI Day Seoul, bringing together Korea’s industry leaders, startups and developer communities advancing physical AI and AI agents. At the plenary sessions, Deepu Talla, vice president of robotics and edge AI at NVIDIA, said Korea is “punching above its weight,” with Doosan Robotics, Hyundai […]
TL;DR Helion now uses compile-time heuristics that analyze a kernel’s IR and dataflow to pick fast settings without slow autotuning. It classifies the kernel as matmul, reduction, or pointwise and... The post Fast by Default: Compile-Time Heuristics for Helion Kernels appeared first on PyTorch.
Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations...
TL;DR With help from AI coding agents, our two-person team integrated SGLang with IBM Spyre in less than a week, building a plugin of roughly 1,500 lines. Existing Spyre support... The post Integrating SGLang into Spyre in Under a Week with AI appeared first on PyTorch.
TL;DR PyTorch models now run in the web browser on the GPU through ExecuTorch, with no conversion to another framework’s format. You export with torch.export to an ExecuTorch .pte file,... The post ExecuTorch WebGPU: PyTorch Models on the GPU, in the Browser appeared first on PyTorch.
TL;DR We serve Meta’s Muse Spark Open, an 816B-parameter mixture-of-experts model, on a single TPU v7 host of four chips, with SGLang running on the TorchTPU PyTorch backend. At the... The post Optimizing Muse Spark Open on TPU v7 with SGLang-TorchTPU appeared first on PyTorch.
Autonomous vehicle (AV) developers need reasoning models that can operate within the compute and latency constraints of an onboard system. Larger models provide...
Seekable OCI (SOCI) indexing for Amazon SageMaker AI training and processing jobs takes full container image download off the critical path. SageMaker AI lazy-loads your image to start containers in seconds instead of minutes, with no changes to your job definition. See the benchmarks and how to adopt it.
This article is the first in a series of technical blog posts jointly published by Hugging Face and Amazon, systematically introducing how to use TorchNeuron on AWS Trainium AI accelerator chips for PyTorch model inference performance analysis. It covers the principles of Neuron backend integration, methods for configuring torch.profiler, the use of NeuronConfig and NeuronProfiler tools, and visualized tracing with Neuron Explorer. Additionally, using RMSNorm as an example, it compares the device‑time differences among three implementations: plain PyTorch, the built-in torch.rms_norm fusion, and torch.compile’s automatic fusion.
i14 has launched the Laplace 3B model with 3.028 billion parameters, which replaces Softmax attention with Gated Linear Attention (GLA). During inference, its per-sequence recursive state remains constant at 29.5 MB, independent of context length. The model was trained and validated on a single AWS trn2.48xlarge instance (equipped with 16 Trainium 2 chips), achieving a training throughput of 120,000 tokens per second and an inference throughput of 11,544 output tokens per second (with 1,280 concurrent sequences). All performance metrics are based on internal measurements conducted by i14 and the AWS GenAIIC team.
Hugging Face has released TorchTPU, providing native TPU backend support for PyTorch, enabling libraries such as Transformers and Diffusers to run directly on TPUs without relying on torch_xla or migrating to JAX.
Hugging Face, in collaboration with AWS, has launched TorchNeuron—a PyTorch-native Trainium backend—available for public beta starting October 20, 2026. This technology enables mainstream libraries such as Transformers and Diffusers to run directly on Trainium chips without any modifications, significantly enhancing the usability and compatibility of text-generation and image-generation models.