Skip to content

GlobalOptimization

-iree-global-opt-cleanup-numeric-narrowinglink

Cleans up any numeric narrowing ops inserted by iree-global-opt-infer-numeric-narrowing.

-iree-global-opt-convert-conv-to-im2collink

Rewrites convolutions as contractions over an im2col gather

Rewrites every linalg op that implements a convolution, named or generic, as an all-parallel linalg.generic that gathers the convolved input windows, followed by a contraction over the gathered tensor with the original payload. The match is on the input access, a sum of scaled loops that is not a projected permutation, so it also covers what unit dimension folding leaves of a convolution whose filter window has extent one along a strided axis, which is no longer a convolution. Contraction loops are collapsed wherever the operand layouts allow, so standard layouts become plain or batched matmuls. Convolutions without an output channel dimension, i.e. depthwise convolutions with a unit multiplier and pooling ops, are left unchanged: nothing reuses the gathered input, so the rewrite never pays off. The exception is a convolution whose gather is identical to that of another rewritten convolution, such as the input window sum of a quantized convolution with a weight zero point; it reads the shared gather.

The pass expects quantization to be lowered already: quantized convolutions such as linalg.conv_2d_nhwc_hwcf_q are not matched, and dequantize producers are not looked through.

-iree-global-opt-convert-qdq-to-integer-mathlink

Rewrites contractions over dequantized operands into integer contractions.

Matches a contraction or convolution whose inputs are produced by iree_linalg_ext.dequantize_affine and rewrites it into an integer contraction plus a zero point correction epilogue:

C = sAsB(D - zBRA - zARB + NzAzB)

Matches a plain floating-point multiply-add body with a zero init. Integer reduction dimensions must have positive static extents whose products and zero point corrections fit in signed i32. Dimensions along which a quantization parameter varies remain floating-point reductions over partial integer contractions. Unsupported cases retain the original computation.

Scaling and partial-result accumulation use at least f32, preserving wider scale and output types, before conversion to the output type.

Numerical stability is best effort. Reassociation changes floating-point rounding; no fixed error bound or preservation of signed zero, NaNs, infinities, or overflow/underflow behavior is guaranteed. Disable this transformation when reference floating-point semantics are required with --iree-global-opt-enable-qdq-to-integer-math=false.

-iree-global-opt-convert-strided-contraction-to-contractionlink

Factors out an extract_slice from contraction-like ops with strided inputs.

-iree-global-opt-data-layout-propagationlink

Propagate pack/unpack ops across other ops to improve fusion

-iree-global-opt-decompose-concatlink

Decomposes concatenations into a destination and a sequence of slice inserts.

Optionslink

-enable-concat-transposition : Allows transposing concatenations such that they occur on the inner most dims.

-iree-global-opt-demote-contraction-inputslink

Demotes inputs (LHS, RHS) of linalg matmul-like ops from f32 to a narrower type.

Optionslink

-type      : Select the target type for demotion.
-operation : Select the type of contraction ops to demote.

-iree-global-opt-detach-elementwise-from-named-opslink

Detaches elementwise ops from named Linalg ops.

-iree-global-opt-early-data-tilinglink

Materializes data-tiling layouts before dispatch creation for a supported target.

For a module with a single executable target that supports host encoding materialization, normalizes contraction dimensions, assigns data-tiling encodings, propagates them through elementwise producers and consumers, and materializes them into packed layouts such as linalg.pack, linalg.mmt4d and linalg.unpack. Records the target in the iree.encoding.materialized_layout_target module attribute, which keeps dispatch creation from assigning encodings again and from compiling the module for another target. Other modules are left unchanged, so they use dispatch-time data tiling. The options select the operation families and encoding strategy, as for iree-dispatch-creation-assign-data-tiling-encodings. Only the default encoding strategy materializes into packed layouts, so modules are left unchanged for other strategies.

Optionslink

-data-tiling-op-types : Op families eligible for data tiling. Defaults to {matmul, scaled_matmul}.
-encoding-option      : Select the type of encoding options to add.

-iree-global-opt-erase-unused-linalg-operandslink

Erases unused linalg operand and remove dead code.

-iree-global-opt-expand-tensor-shapeslink

Expands tensor shape dimensions into SSA values across the program.

-iree-global-opt-fold-broadcast-into-batch-matmullink

Folds broadcasts of contraction inputs into its indexing maps.

Rewrites batch_matmul(A, broadcast(B)), where the weight is shared across the batch, into a batch_matmul over A and B whose RHS indexing map drops the batch dim. Broadcasts on either input, as linalg.broadcast or as a broadcasting generic, are folded by composing their map into the contraction's map. When the named op cannot express the folded map (its verifier accepts only a few broadcast forms, e.g. a batch-broadcast RHS must be [K, N]), the contraction is carried by a linalg.generic with the same body instead of moving data. The contraction then consumes its producers directly, so rewrites that match on them (such as the dequantization folding) still apply, and no broadcast is materialized.

A broadcast under an iree_linalg_ext.dequantize_affine that only feeds the contraction folds too, when the scale and zero point do not vary along the broadcast dims: the dequantize is rebuilt at the rank of the broadcast's source and the broadcast is folded into the contraction map in the same step, so the weights are dequantized once rather than once per batch element.

A contraction that is already a linalg.generic, such as a batch_matmul generalized earlier in the pipeline, is recognized through the contraction interface and folded the same way; it stays a generic.

-iree-global-opt-fuse-dequantization-matmullink

Fuses dequantization and matmul linalg.generic ops.

-iree-global-opt-generalize-linalg-named-opslink

Convert some Linalg named ops into linalg.generics.

Optionslink

-enable-generalize-matmul : Convert linalg named matmul ops to generic ops.

-iree-global-opt-infer-numeric-narrowinglink

Infers and inserts util.numeric.optional_narrow ops at points that may be beneficial.

-iree-global-opt-loop-invariant-code-motionlink

Hoist loop invariants out of loops with zero-trip-check.

-iree-global-opt-materialize-homogeneous-encodingslink

Materializes logical encodings to physical encodings if there is a single device target.

-iree-global-opt-optimize-numericslink

Optimizes numerics given annotations added via iree-global-opt-infer-numeric-narrowing.

-iree-global-opt-pack-sub-byte-quantized-constantslink

Packs constant quantized values with sub-byte ranges into i2 or i4 storage.

Packs the constant i8 input of an iree_linalg_ext.dequantize_affine into i2 or i4 storage, which the dequantize then reads through an iree_tensor_ext.bitcast. With w-bit storage, element k * i + j of the innermost dimension goes to bits [j * w, (j + 1) * w) of byte i, where k = 8 / w.

A dequantize can use w-bit storage if its quant_min/quant_max range and its zero point fit w bits. The zero point must be a constant, and the input must have a static, non-empty shape whose innermost dimension is a multiple of k. Each dequantize needs the narrowest such type. An input value that several dequantizes read is packed once, into the widest type they need. Every load of a global is packed into the same type, and the identical packings are merged later.

An input is packed only when constant evaluation folds the packing and no i8 reader remains. The input must be computed from inline constants or immutable globals with an inline initial value, at the top level of a function, where hoisting reaches. Every read of the input value must be the input of a packed dequantize, in the same shape. This includes reads through reshapes and through other loads of the same global, which must be private, since code outside the module can read a public one. Each packing must also store no more bytes than leaving its input unpacked. Unpacked, hoisting stores a computed input no larger than the constants and globals it reads in a global of its own, which packing shrinks. Reshapes and slices are read in place instead, so their packing, like that of any larger input, must take no more bytes than the constants and globals it frees, which only the computation of packed inputs reads. Either way, hoisting accepts the packing at any size threshold. Other inputs, such as runtime values and parameters, stay i8.

-iree-global-opt-propagate-affine-quantizationlink

Propagates affine quantization through adjacent tensor operations.

Moves iree_linalg_ext.quantize_affine before a producer-side tensor.collapse_shape, and moves a consumer-side tensor.expand_shape, or a zero tensor.pad whose padded dimensions do not index the scale and whose zero point is per-tensor, before iree_linalg_ext.dequantize_affine. These shape operations then run on quantized values. A consumer-side linalg.transpose is fused into the dequantize's output map instead. A tensor.collapse_shape of a dequantized value is absorbed by a consuming linalg.transpose, or by a contraction whose inputs are all dequantized, by expanding the consumer's iteration space; a blockwise matmul over collapsed [M, G, L] and [N, G, L] operands thus iterates over (m, n, g, l). When a contraction consumes the result, the dequantize becomes its direct producer so the integer contraction rewrite can match it. A pad is not moved onto storage that is not byte-aligned, such as i4, where it could shift values off byte boundaries.

-iree-global-opt-propagate-linalg-transposelink

Propagates linalg.transpose through a restricted set of ops.

Optionslink

-enable-aggressive-propagation              : Enable aggressive propagation to named ops.
-test-sinking-only                          : Flag used for lit-testing sinking patterns only. Not for general usage
-test-bubbling-only                         : Flag used for lit-testing bubbling patterns only. Not for general usage
-enable-attention-v-transpose               : Enable transposition of attention v operand
-enable-aggressive-propagation-through-conv : enable propagation through convolutions
-enable-edge-reshape-propagation            : Enable propagation of reshapes on the edges of the program
-enable-sink-transpose-through-pad          : Enable sinking transpose through pad operations

-iree-global-opt-quantized-conv-to-convlink

Lower quantized_conv to conv

-iree-global-opt-quantized-matmul-to-matmullink

Lower quantized_matmul to matmul

-iree-global-opt-raise-special-opslink

Raises special ops like softmax to the high level linalg.ext representation.

-iree-global-opt-remove-zero-extent-tensorslink

Removes tensors that have 0-extents.

-iree-global-opt-simplify-pack-unpacklink

Simplifies tensor pack and unpack ops.

-iree-global-opt-split-large-constantslink

Splits constants larger than the max allocation size into shards.

A constant global whose storage exceeds the max allocation size of its resource config (--iree-stream-resource-max-allocation-size unless the target sets one) cannot be packed into a resource within that limit. This pass splits such a private, immutable global with an inline dense value or a named parameter into shards along its outermost non-unit dimension, so each shard is a contiguous byte range of the original that fits into one allocation. A shard of a named parameter refers to its range of the same parameter through the offset and length config entries. The pass then rewrites the uses of the global's loads to use the shards:

  • an input of a structured op whose split dimension is indexed by a single loop: the op is split along that loop, one op per shard. The results of a parallel loop are inserted into the destination, the ops of a reduction loop accumulate into the result of the previous one. This covers e.g. matmuls with a large weight such as a tied LM head.
  • a value read with tensor.extract in the body of an elementwise linalg.generic, e.g. an embedding lookup: one op per shard reads its shard at the rebased, clamped index and keeps the previous op's element unless the index lies in its shard.

A constant with any other use is left unchanged with a warning.

-iree-global-opt-warn-on-uninitialized-valueslink

Catch some uses of uninitialized values