GlobalOptimization
-iree-global-opt-cleanup-numeric-narrowinglink
Cleans up any numeric narrowing ops inserted by iree-global-opt-infer-numeric-narrowing.
-iree-global-opt-convert-conv-to-im2collink
Rewrites convolutions as contractions over an im2col gather
Rewrites every linalg op that implements a convolution, named or generic,
as an all-parallel linalg.generic that gathers the convolved input
windows, followed by a contraction over the gathered tensor with the
original payload. The match is on the input access, a sum of scaled loops
that is not a projected permutation, so it also covers what unit dimension
folding leaves of a convolution whose filter window has extent one along a
strided axis, which is no longer a convolution. Contraction loops are collapsed wherever the operand
layouts allow, so standard layouts become plain or batched matmuls.
Convolutions without an output channel dimension, i.e. depthwise
convolutions with a unit multiplier and pooling ops, are left unchanged:
nothing reuses the gathered input, so the rewrite never pays off. The
exception is a convolution whose gather is identical to that of another
rewritten convolution, such as the input window sum of a quantized
convolution with a weight zero point; it reads the shared gather.
The pass expects quantization to be lowered already: quantized
convolutions such as linalg.conv_2d_nhwc_hwcf_q are not matched, and
dequantize producers are not looked through.
-iree-global-opt-convert-qdq-to-integer-mathlink
Rewrites contractions over dequantized operands into integer contractions.
Matches a contraction or convolution whose inputs are produced by
iree_linalg_ext.dequantize_affine and rewrites it into an integer
contraction plus a zero point correction epilogue:
C = sAsB(D - zBRA - zARB + NzAzB)
Matches a plain floating-point multiply-add body with a zero init. Integer reduction dimensions must have positive static extents whose products and zero point corrections fit in signed i32. Dimensions along which a quantization parameter varies remain floating-point reductions over partial integer contractions. Unsupported cases retain the original computation.
Scaling and partial-result accumulation use at least f32, preserving wider scale and output types, before conversion to the output type.
Numerical stability is best effort. Reassociation changes floating-point
rounding; no fixed error bound or preservation of signed zero, NaNs,
infinities, or overflow/underflow behavior is guaranteed. Disable this
transformation when reference floating-point semantics are required with
--iree-global-opt-enable-qdq-to-integer-math=false.
-iree-global-opt-convert-strided-contraction-to-contractionlink
Factors out an extract_slice from contraction-like ops with strided inputs.
-iree-global-opt-data-layout-propagationlink
Propagate pack/unpack ops across other ops to improve fusion
-iree-global-opt-decompose-concatlink
Decomposes concatenations into a destination and a sequence of slice inserts.
Optionslink
-enable-concat-transposition : Allows transposing concatenations such that they occur on the inner most dims.
-iree-global-opt-demote-contraction-inputslink
Demotes inputs (LHS, RHS) of linalg matmul-like ops from f32 to a narrower type.
Optionslink
-type : Select the target type for demotion.
-operation : Select the type of contraction ops to demote.
-iree-global-opt-detach-elementwise-from-named-opslink
Detaches elementwise ops from named Linalg ops.
-iree-global-opt-early-data-tilinglink
Materializes data-tiling layouts before dispatch creation for a supported target.
For a module with a single executable target that supports host encoding
materialization, normalizes contraction dimensions, assigns data-tiling
encodings, propagates them through elementwise producers and consumers,
and materializes them into packed layouts such as linalg.pack,
linalg.mmt4d and linalg.unpack. Records the target in the
iree.encoding.materialized_layout_target module attribute, which keeps
dispatch creation from assigning encodings again and from compiling the
module for another target. Other modules are left unchanged, so they use
dispatch-time data tiling. The options select the operation families and
encoding strategy, as for iree-dispatch-creation-assign-data-tiling-encodings.
Only the default encoding strategy materializes into packed layouts, so
modules are left unchanged for other strategies.
Optionslink
-data-tiling-op-types : Op families eligible for data tiling. Defaults to {matmul, scaled_matmul}.
-encoding-option : Select the type of encoding options to add.
-iree-global-opt-erase-unused-linalg-operandslink
Erases unused linalg operand and remove dead code.
-iree-global-opt-expand-tensor-shapeslink
Expands tensor shape dimensions into SSA values across the program.
-iree-global-opt-fold-broadcast-into-batch-matmullink
Folds broadcasts of contraction inputs into its indexing maps.
Rewrites batch_matmul(A, broadcast(B)), where the weight is shared across
the batch, into a batch_matmul over A and B whose RHS indexing map
drops the batch dim. Broadcasts on either input, as linalg.broadcast or
as a broadcasting generic, are folded by composing their map into the
contraction's map. When the named op cannot express the folded map (its
verifier accepts only a few broadcast forms, e.g. a batch-broadcast RHS
must be [K, N]), the contraction is carried by a linalg.generic with the
same body instead of moving data. The contraction then consumes its
producers directly, so rewrites that match on them (such as the
dequantization folding) still apply, and no broadcast is materialized.
A broadcast under an iree_linalg_ext.dequantize_affine that only feeds
the contraction folds too, when the scale and zero point do not vary along
the broadcast dims: the dequantize is rebuilt at the rank of the
broadcast's source and the broadcast is folded into the contraction map in
the same step, so the weights are dequantized once rather than once per
batch element.
A contraction that is already a linalg.generic, such as a batch_matmul
generalized earlier in the pipeline, is recognized through the contraction
interface and folded the same way; it stays a generic.
-iree-global-opt-fuse-dequantization-matmullink
Fuses dequantization and matmul linalg.generic ops.
-iree-global-opt-generalize-linalg-named-opslink
Convert some Linalg named ops into linalg.generics.
Optionslink
-enable-generalize-matmul : Convert linalg named matmul ops to generic ops.
-iree-global-opt-infer-numeric-narrowinglink
Infers and inserts util.numeric.optional_narrow ops at points that may be beneficial.
-iree-global-opt-loop-invariant-code-motionlink
Hoist loop invariants out of loops with zero-trip-check.
-iree-global-opt-materialize-homogeneous-encodingslink
Materializes logical encodings to physical encodings if there is a single device target.
-iree-global-opt-optimize-numericslink
Optimizes numerics given annotations added via iree-global-opt-infer-numeric-narrowing.
-iree-global-opt-pack-sub-byte-quantized-constantslink
Packs constant quantized values with sub-byte ranges into i2 or i4 storage.
Packs the constant i8 input of an iree_linalg_ext.dequantize_affine
into i2 or i4 storage, which the dequantize then reads through an
iree_tensor_ext.bitcast. With w-bit storage, element k * i + j of the
innermost dimension goes to bits [j * w, (j + 1) * w) of byte i, where
k = 8 / w.
A dequantize can use w-bit storage if its quant_min/quant_max range
and its zero point fit w bits. The zero point must be a constant, and the
input must have a static, non-empty shape whose innermost dimension is a
multiple of k. Each dequantize needs the narrowest such type. An input
value that several dequantizes read is packed once, into the widest type
they need. Every load of a global is packed into the same type, and the
identical packings are merged later.
An input is packed only when constant evaluation folds the packing and no
i8 reader remains. The input must be computed from inline constants or
immutable globals with an inline initial value, at the top level of a
function, where hoisting reaches. Every read of the input value must be the
input of a packed dequantize, in the same shape. This includes reads
through reshapes and through other loads of the same global, which must be
private, since code outside the module can read a public one.
Each packing must also store no more bytes than leaving its input
unpacked. Unpacked, hoisting stores a computed input no larger than the
constants and globals it reads in a global of its own, which packing
shrinks. Reshapes and slices are read in place instead, so their packing,
like that of any larger input, must take no more bytes than the constants
and globals it frees, which only the computation of packed inputs reads.
Either way, hoisting accepts the packing at any size threshold. Other
inputs, such as runtime values and parameters, stay i8.
-iree-global-opt-propagate-affine-quantizationlink
Propagates affine quantization through adjacent tensor operations.
Moves iree_linalg_ext.quantize_affine before a producer-side
tensor.collapse_shape, and moves a consumer-side tensor.expand_shape,
or a zero tensor.pad whose padded dimensions do not index the scale and
whose zero point is per-tensor, before iree_linalg_ext.dequantize_affine.
These shape operations then run on quantized values. A consumer-side
linalg.transpose is fused into the dequantize's output map instead. A
tensor.collapse_shape of a dequantized value is absorbed by a consuming
linalg.transpose, or by a contraction whose inputs are all dequantized,
by expanding the consumer's iteration space; a blockwise matmul over
collapsed [M, G, L] and [N, G, L] operands thus iterates over
(m, n, g, l). When a contraction consumes the result, the dequantize
becomes its direct producer so the integer contraction rewrite can match
it. A pad is not moved onto storage that is not byte-aligned, such as i4,
where it could shift values off byte boundaries.
-iree-global-opt-propagate-linalg-transposelink
Propagates linalg.transpose through a restricted set of ops.
Optionslink
-enable-aggressive-propagation : Enable aggressive propagation to named ops.
-test-sinking-only : Flag used for lit-testing sinking patterns only. Not for general usage
-test-bubbling-only : Flag used for lit-testing bubbling patterns only. Not for general usage
-enable-attention-v-transpose : Enable transposition of attention v operand
-enable-aggressive-propagation-through-conv : enable propagation through convolutions
-enable-edge-reshape-propagation : Enable propagation of reshapes on the edges of the program
-enable-sink-transpose-through-pad : Enable sinking transpose through pad operations
-iree-global-opt-quantized-conv-to-convlink
Lower quantized_conv to conv
-iree-global-opt-quantized-matmul-to-matmullink
Lower quantized_matmul to matmul
-iree-global-opt-raise-special-opslink
Raises special ops like softmax to the high level linalg.ext representation.
-iree-global-opt-remove-zero-extent-tensorslink
Removes tensors that have 0-extents.
-iree-global-opt-simplify-pack-unpacklink
Simplifies tensor pack and unpack ops.
-iree-global-opt-split-large-constantslink
Splits constants larger than the max allocation size into shards.
A constant global whose storage exceeds the max allocation size of its
resource config (--iree-stream-resource-max-allocation-size unless the
target sets one) cannot be packed into a resource within that limit. This
pass splits such a private, immutable global with an inline dense value or
a named parameter into shards along its outermost non-unit dimension, so
each shard is a contiguous byte range of the original that fits into one
allocation. A shard of a named parameter refers to its range of the same
parameter through the offset and length config entries. The pass then
rewrites the uses of the global's loads to use the shards:
- an input of a structured op whose split dimension is indexed by a single loop: the op is split along that loop, one op per shard. The results of a parallel loop are inserted into the destination, the ops of a reduction loop accumulate into the result of the previous one. This covers e.g. matmuls with a large weight such as a tied LM head.
- a value read with
tensor.extractin the body of an elementwiselinalg.generic, e.g. an embedding lookup: one op per shard reads its shard at the rebased, clamped index and keeps the previous op's element unless the index lies in its shard.
A constant with any other use is left unchanged with a warning.
-iree-global-opt-warn-on-uninitialized-valueslink
Catch some uses of uninitialized values