Skip to content

Bloglink

Updates from the IREE team

IREE ecosystem roundup: October 2026

IREE newsletter, October 2026

Welcome to the October roundup of news from across the IREE ecosystem.

Two IREE-based hardware launches led October: Google released Coral NPU compiler v1.0, and Synaptics launched its Astra SR200 processors, which use the IREE compiler and runtime. Maximilian Bartel also became chair of the IREE Technical Steering Committee (TSC).

Below, you'll find these stories alongside LLVM Developers' Meeting and PyTorch Conference talks, the IREE 3.12 release, and highlights from the community.

Google and Synaptics ship edge AI hardware built on IREE

On 21 October, Google released Coral NPU compiler v1.0, which is based on IREE, and Synaptics launched the Astra SR200 series on top of Coral NPU, describing the chips as a new class of multimodal edge AI microcontrollers that also use the IREE compiler and runtime. Together, the two launches put IREE underneath commercial edge AI silicon from Synaptics and Google Research.

You can buy the Coralboard SL development board at DigiKey; SR200 chips are still sampling.

Synaptics partner Enerzai has also published hands-on work with the IREE-based Torq compiler: optimizing the Torq NPU compiler, porting a ternary speech-to-text model to the Torq NPU, and building a low-memory smart-home agent demo.

Maximilian Bartel now chairs the IREE Technical Steering Committee

Maximilian Bartel, Roofline's CTO, is now Chair of the IREE TSC. In the role, he coordinates the TSC's work and presides over its meetings, while technical oversight for IREE stays with the TSC as a whole. Read the TSC chair announcement on the Roofline blog. (The Edge AI Foundation also shared the news on LinkedIn.)

MIPS, GlobalFoundries, and Qualcomm back IREE in LLVM talks

At the LLVM Developers' Meeting, Roofline engineers presented joint work: RISC-V work with MIPS by GlobalFoundries, and Hexagon NPU work with Qualcomm. You can see an overview of the LLVM Developers' Meeting speakers on the Roofline website, and the presentation recordings will be shared once they're available.

Toradex, HiPEAC, and PyTorch Conference

IREE 3.12 brings a rewritten Vulkan driver

IREE 3.12 shipped with a rewritten Vulkan driver and an experimental hardware abstraction layer (HAL) for AMD's Heterogeneous System Architecture (HSA), alongside an updated PyPI package. Read Phoronix's coverage of the release.

IREE community meeting

Tobias Fuchs talked about the new features in the IREE 3.12 release and outlined how the release cycle will work in the future. There was also a presentation from Sebastian Fischer from Ludwig-Maximilians-University in Munich on their project "anvl", which implements the R language and is investigating the use of IREE to compile for different targets. Watch the recording on YouTube.

The IREE community built backends, products, and tutorials

Community work this month ranged from a new POWER backend to automotive and chip-design products. A few highlights:

Building something with IREE? We'd like to hear about it. Share it on the IREE Discord server or the iree-technical-discussion email list.

Running models on RISC-V with IREE

IREE compiles machine learning models to native RISC-V CPU code, with support for the RISC-V Vector extension (RVV), hand-written microkernels, and data-tiling. This post walks through the full flow for a model: importing it from PyTorch, compiling it for a RISC-V target, running it, and benchmarking the result.

All commands below run under qemu-riscv64. The flow on real hardware is identical — the QEMU invocation is simply replaced by running the tools natively on the target.

Virtual Dense MFMAs for Skinny GEMM

When we have a GEMM A * B = C, and it is the situation that A has a small number of rows and many columns, we classify this problem as a skinny GEMM. The decode phase of LLM inference is a common sight of this problem: a small batch of tokens multiplies against a large weight matrix. Skinny GEMMs are less convenient for modern GPU architectures than their non-skinny cousins. One reason is because modern GPUs take advantage of matrix core units which offer instructions that are specifically designed for matrix multiplication and operate on fixed tile sizes, and skinny GEMMs are too small to utilize them to their intended size.

On AMDGPUs and in particular on the MI3XX Instinct (CDNA) series, these instructions are known as MFMA instructions; for example, V_MFMA_F32_16x16x16_F16. One useful part of the name is the MxNxK tile shape consumed, where M is the number of rows of the left hand matrix, N is the number of columns of the right hand matrix, and K is the shared dimension of both.

Data-Tiling Walkthrough

Data-tiling is the modification of data layout of operands of certain operations, such as matrix multiplication, that prefer specific layouts. These layout preferences depend on the operations and the target hardware. For example, matrix multiplications may need to use hardware matrix multiplication instructions that perform optimally with a specific matrix data layout.

Layout changes may also be motivated by memory access performance, as data-tiling can result in improved locality of memory accesses, fewer cache lines being accessed, and generally simpler memory access patterns that are more likely to be handled performantly by the memory system.

These layout changes can be propagated as far as possible across the workload, so the entire workload can use the updated layouts, as opposed to having to perform layout transformations at runtime. This may involve fusions or constant-evaluation that can amortize or remove layout-transformation overheads.

The main conceptual difficulty in modeling this in tensor-level MLIR is that tensors don't have layouts: tensors are higher-level, abstract arrays. This is addressed by the concept of tensor encodings, which this document will explain.

IREE / MLIR / Linalg tutorial

Introduction

This tutorial is simultaneously about IREE, MLIR, and specifically the MLIR Linalg dialect.

What is MLIR?

MLIR is a programming language, but MLIR in itself is almost just an empty shell. What it really provides is a framework allowing to define MLIR dialects which are where the features come from.

The "IR" part of the MLIR name stands for "intermediate representation". It means that MLIR is meant to be primarily for compiler-internal representations of code. But MLIR is actually fairly nice for humans to work with, and it's not hard to hand-author some MLIR programs from scratch. That is exactly the topic of this tutorial.

Exploring CPU microkernels on a matmul example

Basic setup, command lines

Source file: matmul.mlir:

func.func @matmul_dynamic(%lhs: tensor<?x?xf32>, %rhs: tensor<?x?xf32>, %acc: tensor<?x?xf32>) -> tensor<?x?xf32> {
  %result = linalg.matmul ins(%lhs, %rhs: tensor<?x?xf32>, tensor<?x?xf32>) outs(%acc: tensor<?x?xf32>) -> tensor<?x?xf32>
  return %result: tensor<?x?xf32>
}

Basic compilation command line:

$ iree-compile matmul.mlir -o /tmp/matmul.vmfb \
  --iree-hal-target-device=local \
  --iree-hal-local-target-device-backends=llvm-cpu \
  --iree-llvmcpu-target-cpu=znver4 \
  --iree-llvmcpu-enable-ukernels=all

This creates a IREE bytecode module:

$ ls -l /tmp/matmul.vmfb

-rw-rw-r-- 1 2884 Jan 22 10:37 /tmp/matmul.vmfb

CUDA backend

IREE is being designed with re-targetability as a core goal: it should be possible to use IREE to target a broad spectrum of power regimes, from embedded systems to distributed clusters; and it should be possible to extend IREE to target new back-ends without having to reinvent the wheel each time.

To explore this, we recently branched out from our initial focus on low-latency mobile deployments with a goal of using IREE to target data center workloads on Nvidia CUDA. This post describes how we quickly brought up a CUDA back-end for IREE and used it to train BERT, then shares some metrics and next steps.

Matrix Multiplication with MMT4D

Introduction

Matrix multiplication (matmul) is an important operation in ML workloads that poses specific challenges to code generation. For example, matmul makes repeated accesses to the same data, which makes locality of reference a top concern.

Moreover, modern CPUs instruction set architectures (ISAs) offer specialized SIMD instructions that the matmul implementation needs to use to achieve optimal performance, and these instructions expect data to be in a particular layout.

This article is about an in-development MLIR operation, linalg.mmt4d, offering a compilation path for linalg.matmul that is designed from the ground up for these efficiency considerations.

TFLite support via TOSA

IREE can now execute TensorFlow Lite (TFLite) models through the use of TOSA, an open standard of common tensor operations, and a part of MLIR core. TOSA’s high-level representation of tensor operations provides a common front-end for ingesting models from different frameworks. In this case we ingest a TFLite FlatBuffer and compile it to TOSA IR, which IREE takes as an input format to compile to its various backends.