'stream' Dialectlink

A dialect designed to model execution partitioning and scheduling.

The stream dialect is designed to take tensor programs and convert them to explicitly scheduled asynchronous programs. This includes placing ops on specific targets, partitioning the work between the targets, scheduling the work for concurrency, and encoding tensors into target-specific resources.

+--------+    +----------+    +-------+
| flow.* | -> | stream.* | -> | hal.* |
+--------+    +----------+    +-------+

This sits in-between the flow and hal dialects.

flow models tensor programs by separating work into dispatchable functions in order to isolate the main host program data flow and the dense tensor compute operations.
stream models explicitly scheduled asynchronous programs by partitioning the dispatchable work, specifying target affinities, encoding tensors into target-specific forms, and scheduling the work to run concurrently.
hal models a low-level hardware abstraction layer used to manage buffers and issue asynchronous work across a variety of device types. The dialect is largely 1:1 with the IREE HAL C API.

Transforms in the dialect lower tensor values into opaque resources with the goal of ensuring no tensors survive in the IR. At entry stream.tensor.* ops are used to capture the source tensor encoding information (data type, shapes, etc) and then lowered into stream.async.* ops that model the asynchronous workloads on the opaque resources. The asynchronous operations are then partitioned, allocated, and scheduled for execution using the stream.cmd.* ops.

It's intended that after transformation through the stream dialect the program is ready for execution on an abstract machine. At this level of representation buffers have still not been allocated and devices are not yet resolved, however the information captured in the stream IR allows such operations to be done trivially. To this end all ops carry the symbolic size of the resources on which they operate as well as the lifetime of the resources they are acting upon. This manifests in the usage of the !stream.resource type:

// Unresolved lifetime (resolved during the iree-stream-refine-usage pass):
!stream.resource<*>
// An externally managed value (passed in via the program API).
!stream.resource<external>
// A staging buffer for uploads/downloads.
!stream.resource<staging>
// A short-lived value that is used across streams.
!stream.resource<transient>
// A long-lived value that persists across streams in globals.
!stream.resource<variable>
// An immutable value that persists for the duration of the program.
!stream.resource<constant>

Operations using resources carry the size of all operand result resources:

// %update (40 bytes) is being inserted into %target (296 bytes).
// Can be dynamic values such as those originating from dynamic dimensions.
%13 = stream.async.update %update, %target[%c256 to %c296] :
    !stream.resource<transient>{%c40} ->
    %target as !stream.resource<transient>{%c296}

Once all stream.async.* work is moved into executable regions (such as stream.async.execute) !stream.timepoint values are used to sequence the execution. These timepoints represent some point in time where all execution up to that timepoint has completed and any results that were produced by the execution are available for use. Attempting to use the resources before their corresponding timepoint has been reached will lead to undefined behavior. The benefit of this is that after timepoints are established in the IR it's possible to induce aliasing of resources without breaking execution correctness.

'stream' Dialect

Operationslink

Async control flow opslink

`stream.async.call` (Stream::AsyncCallOp)link

Calls a streamable external host function

Syntax:

operation ::= `stream.async.call` (`on` `(` $affinity^ `)`)?
              $callee ``
              custom<DispatchOperands>($resource_operands,
              $resource_operand_offsets,
              $resource_operand_ends,
              $resource_operand_lengths) attr-dict `:`
              custom<ShapedFunctionType>(ref($resource_operands),
              type($resource_operands), $resource_operand_sizes,
              type($results), $result_sizes,
              $tied_operands)

Calls a function taking/returning resource values with stream semantics. Asynchronous calls must have no side-effects.

Note that returned resources must have their sizes declared prior to the call as this is what allows the call to be made on the stream. If external host logic is required to compute the size (avoid at all costs!) a separate func.call can be used outside of the stream to do so. If sizes are unknownable until the operation is performed it should be made as a normal asynchronous host call with 'coarse-fences' instead.

Traits: AttrSizedOperandSegments, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, CallOpInterface, Stream_AffinityOp, Stream_StreamableOp, SymbolUserOpInterface, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`callee`	::mlir::FlatSymbolRefAttr	flat symbol reference attribute
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or index or integer or floating-point or complex-type or any type
`resource_operand_sizes`	variadic of index
`resource_operand_offsets`	variadic of index
`resource_operand_ends`	variadic of index
`resource_operand_lengths`	variadic of index
`result_sizes`	variadic of index

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource or index or integer or floating-point or complex-type

`stream.async.concurrent` (Stream::AsyncConcurrentOp)link

Executes all ops concurrently

Syntax:

operation ::= `stream.async.concurrent` (`on` `(` $affinity^ `)`)?
              `with` ``
              custom<ResourceRegion>($resource_operands,
              type($resource_operands), $resource_operand_sizes,
              type($results), $result_sizes,
              $tied_operands, $body)
              attr-dict-with-keyword

Represents a wave of work scheduled concurrently (each op executing at the same time). All resource inputs must be captured explicitly. All results are only ready once all nested ops complete execution.

Waves can be nested to create a DAG. For example, take the following graph:

                  |
        v---------+---------v
+-------|-------+   +-------|-------+
|    v--+--v    |   |    v--+--v    |
| +----+ +----+ |   | +----+ +----+ |
| | %a | | %b | |   | | %c | | %d | |
| +----+ +----+ |   | +----+ +----+ |
|    +--v--+    |   |    +--v--+    |
+-------|-------+   +-------|-------+
        +---------v---------+
                  |

Represented with nested waves:

  %0 = stream.async.concurrent with(%arg) -> ... {
    %1 = stream.async.concurrent with(%arg as %arg0) -> ... {
      %a = ...
      %b = ...
      stream.yield %a, %b
    }
    %2 = stream.async.concurrent with(%arg as %arg1) -> ... {
      %c = ...
      %d = ...
      stream.yield %c, %d
    }
    stream.yield %1, %2
  }

Traits: AttrSizedOperandSegments, HasParent<IREE::Stream::AsyncExecuteOp, IREE::Stream::AsyncConcurrentOp>, RecursiveMemoryEffects, SingleBlockImplicitTerminator<IREE::Stream::YieldOp>, SingleBlock, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, ClosureOpInterface, RegionBranchOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_operand_sizes`	variadic of index
`result_sizes`	variadic of index

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource

`stream.async.execute` (Stream::AsyncExecuteOp)link

Executes a dependency-aware sequence of streamable ops

Syntax:

operation ::= `stream.async.execute` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              `with` ``
              custom<ResourceRegion>($resource_operands,
              type($resource_operands), $resource_operand_sizes,
              type($results), $result_sizes,
              $tied_operands, $body)
              `=` `` `>` type($result_timepoint)
              attr-dict-with-keyword

Evaluates the operations within the region by dependency order while obeying ties when present. Nested ops execute serially in block order and nested stream.async.concurrent ops can be used to run multiple ops concurrently within the stream. All resource inputs must be captured explicitly. All results are only ready once all nested ops complete execution and the returned timepoint is reached. Zero or more timepoints may be provided to block execution until they are all reached; zero timepoints indicates that execution may begin immediately.

Traits: AttrSizedOperandSegments, RecursiveMemoryEffects, SingleBlockImplicitTerminator<IREE::Stream::YieldOp>, SingleBlock, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, ClosureOpInterface, RegionBranchOpInterface, Stream_AffinityOp, Stream_TimelineOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_operand_sizes`	variadic of index
`result_sizes`	variadic of index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`result_timepoint`	a timepoint indicating execution availability

`stream.async.func` (Stream::AsyncFuncOp)link

Streamable function declaration

Syntax:

operation ::= `stream.async.func` custom<SymbolVisibility>($sym_visibility)
              $sym_name
              ``
              custom<ShapedFunctionSignature>($function_type,
              $tied_operands,
              $arg_attrs,
              $res_attrs)
              attr-dict-with-keyword
              ($body^)?

Declares a function that can be called as an asynchronous streaming operation via stream.async.call. Today only external functions are allowed.

Traits: IsolatedFromAbove, Stream_AsyncPhaseOp

Interfaces: CallableOpInterface, FunctionOpInterface, Symbol

Attributes:link

Attribute	MLIR Type	Description
`sym_name`	::mlir::StringAttr	string attribute
`function_type`	::mlir::TypeAttr	type attribute of function type
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`sym_visibility`	::mlir::StringAttr	string attribute
`arg_attrs`	::mlir::ArrayAttr	Array of dictionary attributes
`res_attrs`	::mlir::ArrayAttr	Array of dictionary attributes

Async opslink

`stream.async.alloca` (Stream::AsyncAllocaOp)link

Allocates a transient value with undefined contents

Syntax:

operation ::= `stream.async.alloca` (`on` `(` $affinity^ `)`)?
              attr-dict `:` type($result) `{` $storage_size `}`

Allocates a transient value (one that is short-lived and local to the current computation) with undefined contents. Consumers of the allocated result must assume nothing of the contents and use discard access.

Traits: AlwaysSpeculatableImplTrait, Stream_AsyncPhaseOp

Interfaces: ConditionallySpeculatable, MemoryEffectOpInterface (MemoryEffectOpInterface), Stream_AffinityOp, StreamableOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Allocate on ::mlir::SideEffects::DefaultResource}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`storage_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.clone` (Stream::AsyncCloneOp)link

Clones the contents of a value

Syntax:

operation ::= `stream.async.clone` (`on` `(` $affinity^ `)`)?
              $source `:`
              type($source) `` `{` $source_size `}` `->`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Clones the contents of a value at a snapshot in time. Future changes to the cloned value will not effect the result. Acts as a copy-on-write operation.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, Stream_AffinityOp, StreamableOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.collective` (Stream::AsyncCollectiveOp)link

Performs a collective operation

Syntax:

operation ::= `stream.async.collective` `` $op `` `[` $element_count `]`
              (`on` `(` $affinity^ `)`)?
              `channel` `(` $channel `)`
              custom<CollectiveParam>(ref($op), $param) ``
              $source `[` $source_offset `to` $source_end `for` $source_length `]` `,`
              $target `[` $target_offset `to` $target_end `for` $target_length `]` `:`
              type($source) `` `{` $source_size `}` `->`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

TODO: document different usage. For now this should be considered a prototype and that modeling of collective operations may change in the future to better ensure in-place operations (where send/recv is a subset of recv/send). We may have dedicated operations for the send and recv verbs as they have sequencing implications - or we could add optional sequencing to this base op.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`op`	::mlir::iree_compiler::IREE::Stream::CollectiveAttr	collective operation and specification
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_end`	index
`target_length`	index
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offset`	index
`source_end`	index
`source_length`	index
`element_count`	index
`channel`	a collective communication channel
`param`	32-bit signless integer

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.constant` (Stream::AsyncConstantOp)link

Defines a constant resource

Syntax:

operation ::= `stream.async.constant` (`on` `(` $affinity^ `)`)?
              `:`
              type($result) `` `{` $result_size `}`
              `=`
              $value
              attr-dict-with-keyword

Returns a new resource with the given constant value.

Traits: AlwaysSpeculatableImplTrait, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp, StreamableOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`value`	::mlir::Attribute	any attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.copy` (Stream::AsyncCopyOp)link

Copies a subview of a stream resource to another

Syntax:

operation ::= `stream.async.copy` (`on` `(` $affinity^ `)`)?
              $source `[` $source_offset `to` $source_end `]` `,`
              $target `[` $target_offset `to` $target_end `]` `,`
              $length `:`
              type($source) `` `{` $source_size `}` `->`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Copies a subview of a resource into a subview of another. As with memcpy this does not support overlapping updates into the same resource. Unlike stream.async.update copy sources cannot be allocated in-place.

Equivalent to a stream.async.slice + stream.async.update.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_end`	index
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offset`	index
`source_end`	index
`length`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.dispatch` (Stream::AsyncDispatchOp)link

Dispatches a parallelized grid of work

Syntax:

operation ::= `stream.async.dispatch` (`on` `(` $affinity^ `)`)?
              custom<DispatchEntryPoints>($entry_points)
              (`[` $workload^ `]`)? ``
              custom<DispatchOperands>($resource_operands,
              $resource_operand_offsets,
              $resource_operand_ends,
              $resource_operand_lengths) attr-dict `:`
              custom<ShapedFunctionType>(ref($resource_operands),
              type($resource_operands), $resource_operand_sizes,
              type($results), $result_sizes,
              $tied_operands)

Calls the specified entry point function once for each element in the specified workgroup count. Each workgroup has access to the same operands and results and is able to load/store at will.

Traits: AttrSizedOperandSegments, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, Stream_AffinityOp, Stream_StreamableOp, SymbolUserOpInterface, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`entry_points`	::mlir::ArrayAttr	symbol ref array attribute
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`workload`	variadic of index
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or index or integer or floating-point or complex-type
`resource_operand_sizes`	variadic of index
`resource_operand_offsets`	variadic of index
`resource_operand_ends`	variadic of index
`resource_operand_lengths`	variadic of index
`result_sizes`	variadic of index

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource

`stream.async.fill` (Stream::AsyncFillOp)link

Fills a subview of a stream resource with a value

Syntax:

operation ::= `stream.async.fill` (`on` `(` $affinity^ `)`)?
              $value `,`
              $target `[` $target_offset `to` $target_end `for` $target_length `]` `:`
              type($value) `->`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Splats a value into a subview of the given stream resource and returns the resource with the update applied.

Equivalent to a stream.async.splat + stream.async.update.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_end`	index
`target_length`	index
`value`	8-bit signless integer or 16-bit signless integer or 32-bit signless integer or 64-bit signless integer

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.load` (Stream::AsyncLoadOp)link

Loads a value from a resource

Syntax:

operation ::= `stream.async.load` $source `[` $source_offset `]` `:`
              type($source) `` `{` $source_size `}`
              `->`
              type($result)
              attr-dict-with-keyword

Returns the element at the given location from within the resource.

Traits: AlwaysSpeculatableImplTrait, Stream_AsyncPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`source`	staging resource
`source_size`	index
`source_offset`	index

Results:link

Result	Description
`result`	index or integer or floating-point or complex-type or vector of any type values

`stream.async.slice` (Stream::AsyncSliceOp)link

Slices out a cloned subview of a value

Syntax:

operation ::= `stream.async.slice` (`on` `(` $affinity^ `)`)?
              $source `[` $source_offset `to` $source_end `]` `:`
              type($source) `` `{` $source_size `}` `->`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Slices a subrange of a stream resource based on a byte range. Acts as a copy-on-write operation.

Traits: AlwaysSpeculatableImplTrait, Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_StreamableOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offset`	index
`source_end`	index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.splat` (Stream::AsyncSplatOp)link

Splats a value into a resource

Syntax:

operation ::= `stream.async.splat` (`on` `(` $affinity^ `)`)?
              $value `:` type($value) `->` type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Returns a new resource with the given primitive value splatted out to fill the entire contents.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, Stream_AffinityOp, StreamableOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`value`	8-bit signless integer or 16-bit signless integer or 32-bit signless integer or 64-bit signless integer
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.async.store` (Stream::AsyncStoreOp)link

Stores a value into a resource

Syntax:

operation ::= `stream.async.store` $value `,`
              $target `[` $target_offset `]` `:`
              type($value)
              `->`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Returns a resource with the element at the given offset set to the given value.

Traits: AlwaysSpeculatableImplTrait, Stream_AsyncPhaseOp

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), TiedOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`target`	staging resource
`target_size`	index
`target_offset`	index
`value`	index or integer or floating-point or complex-type or vector of any type values

Results:link

Result	Description
`result`	staging resource

`stream.async.transfer` (Stream::AsyncTransferOp)link

Transfers a resource from one location/state to another

Syntax:

operation ::= `stream.async.transfer` $source `:` type($source)
              `` `{` $source_size `}`
              (`from` `(` $source_affinity^ `)`)?
              `->`
              (`to` `(` $result_affinity^ `)`)?
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Transfers a resource between different states (such as a staging lifetime to a local lifetime) or different affinities. This is roughly equivalent to a cast but may have special semantics when later lowered to one or more devices with discrete memory spaces or pools.

Traits: Stream_AsyncPhaseOp

Interfaces: AffinityOpInterface, AsyncAccessOpInterface, Stream_StreamableOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`source_affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity
`result_affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource or staging resource
`source_size`	index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource or staging resource

`stream.async.update` (Stream::AsyncUpdateOp)link

Updates a slice of a subview of a resource in-place

Syntax:

operation ::= `stream.async.update` (`on` `(` $affinity^ `)`)?
              $update `,`
              $target `[` $target_offset `to` $target_end `]` `:`
              type($update) `` `{` $update_size `}` `->`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Copies a value into a resource based on a byte range. The returned value is the entire updated target value. Updates can be turned into placement allocations and avoid copies.

Traits: Stream_AsyncPhaseOp

Interfaces: AsyncAccessOpInterface, InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_end`	index
`update`	resource or external resource or transient resource or variable resource or constant resource
`update_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

Channel opslink

`stream.channel.count` (Stream::ChannelCountOp)link

Returns the total number of participants in the group

Syntax:

operation ::= `stream.channel.count` $channel `:` type($result)
              attr-dict-with-keyword

Returns the total participant count in the collective communicator group.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`channel`	a collective communication channel

Results:link

Result	Description
`result`	index

`stream.channel.create` (Stream::ChannelCreateOp)link

Creates a new channel for collective communication

Syntax:

operation ::= `stream.channel.create` (`on` `(` $affinity^ `)`)?
              (`id` `(` $id^ `)`)?
              (`group` `(` $group^ `)`)?
              (`rank` `(` $rank^ `)`)?
              (`count` `(` $count^ `)`)?
              `:` type($result)
              attr-dict-with-keyword

Returns a new channel with the given rank associated with the specified affinity. Collective operations using this channel must only be submitted on compatible affinities.

The group and ID are optional and may be null. The rank and count can be omitted to indicate a default inherited from the environment or device configuration at runtime.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`group`	::mlir::StringAttr	string attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`id`	a reference counted byte buffer
`rank`	index
`count`	index

Results:link

Result	Description
`result`	a collective communication channel

`stream.channel.rank` (Stream::ChannelRankOp)link

Returns the rank of the local participant in the group

Syntax:

operation ::= `stream.channel.rank` $channel `:` type($result)
              attr-dict-with-keyword

Returns the rank the channel represents as a participant in a collective group in [0, count).

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`channel`	a collective communication channel

Results:link

Result	Description
`result`	index

`stream.channel.split` (Stream::ChannelSplitOp)link

Splits a collective communication channel

Syntax:

operation ::= `stream.channel.split` $channel `,` $color `,` $key
              `:` type($channel) `->` type($result)
              attr-dict-with-keyword

Partitions the group associated with the given channel into disjoint subgroups for each unique value of color. Each new subgroup contains all participants of the same color and within each subgroup the key argument is used to define the rank order. When multiple participants in a group use the same key the tie will be broken using their rank in the parent group. A color of -1 indicates that the rank does not participate in any subgroup and will return a null channel.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`channel`	a collective communication channel
`color`	index
`key`	index

Results:link

Result	Description
`result`	a collective communication channel

Executable opslink

`stream.binding.subspan` (Stream::BindingSubspanOp)link

Returns an alias to a subspan of interface binding data

Syntax:

operation ::= `stream.binding.subspan` $binding `` `[` $byte_offset `]`
              attr-dict `:` type($binding) `->` type($result) (`{` $dynamic_dims^ `}`)?

Returns a subview to a tensor or memref-like type from a binding. The same binding may have multiple subviews at different byte offsets.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Util_ShapeAwareOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`binding`	a managed resource binding into an executable scope
`byte_offset`	index
`dynamic_dims`	variadic of index

Results:link

Result	Description
`result`	any type

`stream.dispatch.workgroup.count` (Stream::DispatchWorkgroupCountOp)link

Returns the total workgroup count of the grid

Syntax:

operation ::= `stream.dispatch.workgroup.count` `[` $dimension `]` attr-dict `:` type($result)

The total number of workgroups along each dimension in the dispatch grid.

Represented as a 3D grid classically written as XYZ. Corresponds to the NumWorkgroups SPIR-V built-in and the gridDim CUDA built-in variable.

%x = stream.dispatch.workgroup.count[0] : index
%y = stream.dispatch.workgroup.count[1] : index
%z = stream.dispatch.workgroup.count[2] : index

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`dimension`	::mlir::IntegerAttr	index attribute

Results:link

Result	Description
`result`	index

`stream.dispatch.workgroup.id` (Stream::DispatchWorkgroupIDOp)link

Returns the index of the current workgroup in the grid

Syntax:

operation ::= `stream.dispatch.workgroup.id` `[` $dimension `]` attr-dict `:` type($result)

The global workgroup ID of the current workgroup in the range of [0, stream.dispatch.workgroup.count) along each dimension.

Represented as a 3D grid classically written as XYZ. Corresponds to the WorkgroupId SPIR-V built-in and the blockIdx CUDA built-in variable.

%x = stream.dispatch.workgroup.id[0] : index
%y = stream.dispatch.workgroup.id[1] : index
%z = stream.dispatch.workgroup.id[2] : index

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`dimension`	::mlir::IntegerAttr	index attribute

Results:link

Result	Description
`result`	index

`stream.dispatch.workgroup.size` (Stream::DispatchWorkgroupSizeOp)link

Returns the size of each workgroup in invocations

Syntax:

operation ::= `stream.dispatch.workgroup.size` `[` $dimension `]` attr-dict `:` type($result)

The number of local invocations within the current workgroup along each dimension. Depending on backend this may map to the SIMT thread count or inner loop nest parameters.

Workgroup sizes are not determined at the stream dialect level as they are dependent on the target backend determined when lowering into the HAL. It's still possible to use the symbolic workgroup size inside of dispatch executables as a placeholder for the resolved value once in the HAL.

Represented as a 3D grid classically written as XYZ. Corresponds to the WorkgroupSize SPIR-V built-in and the blockDim CUDA built-in variable.

%x = stream.dispatch.workgroup.size[0] : index
%y = stream.dispatch.workgroup.size[1] : index
%z = stream.dispatch.workgroup.size[2] : index

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`dimension`	::mlir::IntegerAttr	index attribute

Results:link

Result	Description
`result`	index

`stream.executable.end` (Stream::ExecutableEndOp)link

Terminator pseudo-op for the executable op

Syntax:

operation ::= `stream.executable.end` attr-dict

Traits: HasParent<IREE::Stream::ExecutableOp>, Terminator

`stream.executable.export` (Stream::ExecutableExportOp)link

Defines an executable entry point for dispatch operations

Syntax:

operation ::= `stream.executable.export` custom<SymbolVisibility>($sym_visibility)
              custom<SymbolAlias>($sym_name, $function_ref)
              custom<WorkgroupCountRegion>($workgroup_count)
              attr-dict-with-keyword

Specifies an exported function with an externally-visible alias. Multiple exports can reference the same internal function.

Each entry point can have a unique workgroup count calculation region. This region takes the workload parameters passed to each flow.dispatch and produces an XYZ workgroup count for the 3D grid dispatch.

Traits: HasParent<IREE::Stream::ExecutableOp>, IsolatedFromAbove

Interfaces: Symbol

Attributes:link

Attribute	MLIR Type	Description
`sym_visibility`	::mlir::StringAttr	string attribute
`sym_name`	::mlir::StringAttr	string attribute
`function_ref`	::mlir::FlatSymbolRefAttr	flat symbol reference attribute

`stream.executable` (Stream::ExecutableOp)link

Generic executable module

Syntax:

operation ::= `stream.executable` custom<SymbolVisibility>($sym_visibility)
              $sym_name
              attr-dict-with-keyword
              regions

An executable module containing one or more public functions. The contents of the functions are safe to dispatch and can be lowered further to target-specific backend IR representations.

Traits: IsolatedFromAbove, SingleBlockImplicitTerminator<IREE::Stream::ExecutableEndOp>, SingleBlock, SymbolTable, Util_ObjectLike

Interfaces: Symbol

Attributes:link

Attribute	MLIR Type	Description
`sym_visibility`	::mlir::StringAttr	string attribute
`sym_name`	::mlir::StringAttr	string attribute

Execution context opslink

Operations for interacting with the execution context that stream operations execute within.

`stream.context.resolve` (Stream::ContextResolveOp)link

Resolves low-level context resources based on type

Syntax:

operation ::= `stream.context.resolve` (`on` `(` $affinity^ `)`)?
              attr-dict `:` type($results)

WIP; allows for accessing the implementation details of lower-level dialects such as the HAL. This will likely be reworked in the future to either live inside other dialects, use some op interface instead of having a dedicated op here, or remove the op entirely and make resolution happen explicitly.

Examples:

// Returns a HAL device.
= stream.context.resolve on(#something) : !hal.device
// Returns a HAL device and (optional) queue affinity.
= stream.context.resolve on(#something) : !hal.device, i64
// Returns a HAL allocator and (optional) queue affinity.
= stream.context.resolve on(#something) : !hal.allocator, i64

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Results:link

Result	Description
`results`	variadic of any type

Explicit command opslink

`stream.cmd.call` (Stream::CmdCallOp)link

Calls a streamable external host function

Syntax:

operation ::= `stream.cmd.call` $callee ``
              custom<CmdCallOperands>($resource_operands,
              $resource_operand_offsets,
              $resource_operand_lengths,
              $resource_operand_accesses) attr-dict `:`
              custom<ShapedFunctionType>(ref($resource_operands),
              type($resource_operands),
              $resource_operand_sizes,
              type($results),
              $result_sizes,
              $tied_operands)

Calls a function operating on resource values with stream semantics. Asynchronous calls must have no side-effects.

Traits: AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: CallOpInterface, Stream_StreamableOp, Stream_SubviewEffectOp, SymbolUserOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`callee`	::mlir::FlatSymbolRefAttr	flat symbol reference attribute
`tied_operands`	::mlir::ArrayAttr	64-bit integer array attribute
`resource_operand_accesses`	::mlir::ArrayAttr	access array attribute

Operands:link

Operand	Description
`resource_operands`	variadic of index or integer or floating-point or complex-type or resource or external resource or transient resource or variable resource or constant resource or any type
`resource_operand_sizes`	variadic of index
`resource_operand_offsets`	variadic of index
`resource_operand_lengths`	variadic of index
`result_sizes`	variadic of index

Results:link

Result	Description
`results`	variadic of index or integer or floating-point or complex-type

`stream.cmd.collective` (Stream::CmdCollectiveOp)link

Dispatches a collective operation

Syntax:

operation ::= `stream.cmd.collective` `` $op `` `[` $element_count `]`
              `channel` `(` $channel `)`
              (`param` `(` $param^ `:` type($param) `)`)? `{`
              custom<DispatchResources>($resources, type($resources), $resource_sizes,
              $resource_offsets, $resource_lengths,
              $resource_accesses)
              `\n` `}`
              attr-dict-with-keyword

Dispatches a collective operation specified against the device. If grouped with other collectives in a stream.cmd.concurrent region the collective operations may fuse and execute more efficiently.

Traits: AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`op`	::mlir::iree_compiler::IREE::Stream::CollectiveAttr	collective operation and specification
`resource_accesses`	::mlir::ArrayAttr	access array attribute

Operands:link

Operand	Description
`channel`	a collective communication channel
`element_count`	index
`param`	32-bit signless integer
`resources`	variadic of resource or external resource or transient resource or variable resource or constant resource
`resource_sizes`	variadic of index
`resource_offsets`	variadic of index
`resource_lengths`	variadic of index

`stream.cmd.concurrent` (Stream::CmdConcurrentOp)link

Executes all ops concurrently

Syntax:

operation ::= `stream.cmd.concurrent` $body
              attr-dict-with-keyword

Represents a wave of work scheduled concurrently (each op executing at the same time).

Waves can be nested to create a DAG. For example, take the following graph:

                  |
        v---------+---------v
+-------|-------+   +-------|-------+
|    v--+--v    |   |    v--+--v    |
| +----+ +----+ |   | +----+ +----+ |
| | @a | | @b | |   | | @c | | @d | |
| +----+ +----+ |   | +----+ +----+ |
|    +--v--+    |   |    +--v--+    |
+-------|-------+   +-------|-------+
        +---------v---------+
                  |

Represented with nested waves:

  stream.cmd.concurrent {
    stream.cmd.concurrent {
      stream.cmd.dispatch @a
      stream.cmd.dispatch @b
    }
    stream.cmd.concurrent {
      stream.cmd.dispatch @c
      stream.cmd.dispatch @d
    }
  }

Traits: HasParent<IREE::Stream::CmdExecuteOp, IREE::Stream::CmdSerialOp, IREE::Stream::CmdConcurrentOp>, RecursiveMemoryEffects, SingleBlockImplicitTerminator<IREE::Stream::YieldOp>, SingleBlock, Stream_CmdPhaseOp

Interfaces: RegionBranchOpInterface, Stream_StreamableOp

`stream.cmd.copy` (Stream::CmdCopyOp)link

Copies a subview of a stream resource to another

Syntax:

operation ::= `stream.cmd.copy` $source `[` $source_offset `]` `,`
              $target `[` $target_offset `]` `,`
              $length `:`
              type($source) `` `{` $source_size `}` `->`
              type($target) `` `{` $target_size `}`
              attr-dict-with-keyword

Copies a subview of a resource into a subview of another. As with memcpy this does not support overlapping updates into the same resource.

Traits: Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Operands:link

Operand	Description
`source`	any stream-compatible type
`source_size`	index
`source_offset`	index
`target`	any stream-compatible type
`target_size`	index
`target_offset`	index
`length`	index

`stream.cmd.discard` (Stream::CmdDiscardOp)link

Discards a subview of a resource

Syntax:

operation ::= `stream.cmd.discard` $target `[` $target_offset `for` $target_length `]` `:`
              type($target) `` `{` $target_size `}`
              attr-dict-with-keyword

Discards a subview of a resource, indicating that after this command the specified contents are no longer needed. This can be used to trim memory or invalidate caches.

Traits: Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Operands:link

Operand	Description
`target`	any stream-compatible type
`target_size`	index
`target_offset`	index
`target_length`	index

`stream.cmd.dispatch` (Stream::CmdDispatchOp)link

Dispatches a parallelized grid of work

Syntax:

operation ::= `stream.cmd.dispatch` custom<DispatchEntryPoints>($entry_points)
              (`[` $workload^ `]`)? ``
              (`(` $uniform_operands^ `:` type($uniform_operands) `)`)? `{`
              custom<DispatchResources>($resources, type($resources), $resource_sizes,
              $resource_offsets, $resource_lengths,
              $resource_accesses)
              `\n` `}`
              attr-dict-with-keyword

Calls the specified entry point function once for each element in the specified workgroup count. Each workgroup has access to the same operands and results and is able to load/store at will.

Traits: AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, SymbolUserOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`entry_points`	::mlir::ArrayAttr	symbol ref array attribute
`resource_accesses`	::mlir::ArrayAttr	access array attribute

Operands:link

Operand	Description
`workload`	variadic of index
`uniform_operands`	variadic of index or integer or floating-point or complex-type
`resources`	variadic of resource or external resource or transient resource or variable resource or constant resource
`resource_sizes`	variadic of index
`resource_offsets`	variadic of index
`resource_lengths`	variadic of index

`stream.cmd.execute` (Stream::CmdExecuteOp)link

Executes a dependency-aware sequence of streamable ops

Syntax:

operation ::= `stream.cmd.execute` (`once` $once^)?
              (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              `with` ``
              custom<ExplicitResourceRegion>($resource_operands,
              type($resource_operands), $resource_operand_sizes,
              $body)
              `=` `` `>` type($result_timepoint)
              attr-dict-with-keyword

Evaluates the operations within the region by dependency order while obeying ties when present. Nested ops execute serially in block order and nested stream.cmd.concurrent ops can be used to run multiple ops concurrently within the stream. All resource inputs must be captured explicitly. All results are only ready once all nested ops complete execution and the returned timepoint is reached. Zero or more timepoints may be provided to block execution until they are all reached; zero timepoints indicates that execution may begin immediately.

Traits: AttrSizedOperandSegments, RecursiveMemoryEffects, SingleBlockImplicitTerminator<IREE::Stream::YieldOp>, SingleBlock, Stream_CmdPhaseOp

Interfaces: ClosureOpInterface, InferTypeOpInterface, RegionBranchOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`once`	::mlir::UnitAttr	unit attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_operand_sizes`	variadic of index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.cmd.fill` (Stream::CmdFillOp)link

Fills a subview of a stream resource with a value

Syntax:

operation ::= `stream.cmd.fill` $value `,`
              $target `[` $target_offset `for` $target_length `]` `:`
              type($value) `->`
              type($target) `` `{` $target_size `}`
              attr-dict-with-keyword

Splats a value into a subview of the given stream resource and returns the resource with the update applied.

Traits: Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_length`	index
`value`	8-bit signless integer or 16-bit signless integer or 32-bit signless integer

`stream.cmd.flush` (Stream::CmdFlushOp)link

Flushes a subview of a resource

Syntax:

operation ::= `stream.cmd.flush` (`to` `(` $source_affinity^ `)`)?
              $target `[` $target_offset `for` $target_length `]` `:`
              type($target) `` `{` $target_size `}`
              attr-dict-with-keyword

Transfers a resource to an external target. The resource memory is made available to the target and can be made visible there using stream.cmd.invalidate.

Traits: Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`source_affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	any stream-compatible type
`target_size`	index
`target_offset`	index
`target_length`	index

`stream.cmd.func` (Stream::CmdFuncOp)link

Streamable function declaration

Syntax:

operation ::= `stream.cmd.func` custom<SymbolVisibility>($sym_visibility)
              $sym_name ``
              custom<DispatchFunctionSignature>($function_type,
              $arg_attrs,
              $res_attrs)
              attr-dict-with-keyword
              ($body^)?

Declares a function that can be called as an asynchronous streaming operation via stream.cmd.call. Today only external functions are allowed.

Traits: IsolatedFromAbove, Stream_CmdPhaseOp

Interfaces: CallableOpInterface, FunctionOpInterface, Symbol

Attributes:link

Attribute	MLIR Type	Description
`sym_name`	::mlir::StringAttr	string attribute
`function_type`	::mlir::TypeAttr	type attribute of function type
`sym_visibility`	::mlir::StringAttr	string attribute
`arg_attrs`	::mlir::ArrayAttr	Array of dictionary attributes
`res_attrs`	::mlir::ArrayAttr	Array of dictionary attributes

`stream.cmd.invalidate` (Stream::CmdInvalidateOp)link

Invalidates a subview of a resource

Syntax:

operation ::= `stream.cmd.invalidate` (`from` `(` $source_affinity^ `)`)?
              $target `[` $target_offset `for` $target_length `]` `:`
              type($target) `` `{` $target_size `}`
              attr-dict-with-keyword

Transfers a resource from an external source into the current target. The resource memory is assumed to have been made available at the source via stream.cmd.flush.

Traits: Stream_CmdPhaseOp

Interfaces: Stream_StreamableOp, Stream_SubviewEffectOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`source_affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	any stream-compatible type
`target_size`	index
`target_offset`	index
`target_length`	index

`stream.cmd.serial` (Stream::CmdSerialOp)link

Executes all ops serially (in-order)

Syntax:

operation ::= `stream.cmd.serial` $body
              attr-dict-with-keyword

Represents a sequence of work scheduled serially (each op executing one after the other).

Regions can be nested to create a DAG. For example, take the following graph:

                  |
        v---------+-----v
+-------|-------+   +---|----+
|    v--+--v    |   |   v    |
| +----+ +----+ |   | +----+ |
| | @a | | @b | |   | | @c | |
| +----+ +----+ |   | +----+ |
|    |     |    |   |   |    |
|    |     |    |   | +-v--+ |
|    |     |    |   | | @d | |
|    |     |    |   | +----+ |
|    +--v--+    |   |   |    |
+-------|-------+   +---|----+
        +---------v-----+
                  |

Represented with nested regions:

  stream.cmd.concurrent {
    stream.cmd.concurrent {
      stream.cmd.dispatch @a
      stream.cmd.dispatch @b
    }
    stream.cmd.serial {
      stream.cmd.dispatch @c
      stream.cmd.dispatch @d
    }
  }

Traits: HasParent<IREE::Stream::CmdExecuteOp, IREE::Stream::CmdSerialOp, IREE::Stream::CmdConcurrentOp>, RecursiveMemoryEffects, SingleBlockImplicitTerminator<IREE::Stream::YieldOp>, SingleBlock, Stream_CmdPhaseOp

Interfaces: RegionBranchOpInterface, Stream_StreamableOp

File opslink

File ops.

`stream.file.constant` (Stream::FileConstantOp)link

Creates a file backed by the provided constant host memory

Syntax:

operation ::= `stream.file.constant` (`on` `(` $affinity^ `)`)?
              $source `[` $source_offset `for` $source_length `]` `:`
              type($source) `` `{` $source_size `}`
              `->`
              type($result)
              attr-dict-with-keyword

Synchronously wraps a host heap buffer into a stream-accessible file handle. Changing the source buffer after definition has undefined behavior.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, MemoryEffectOpInterface (MemoryEffectOpInterface), NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp, SubrangeOperandOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Allocate on ::mlir::SideEffects::DefaultResource}, MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	a reference counted byte buffer
`source_size`	index
`source_offset`	index
`source_length`	index

Results:link

Result	Description
`result`	a file handle used for I/O operations

`stream.file.read` (Stream::FileReadOp)link

Reads a segment of a file into a resource

Syntax:

operation ::= `stream.file.read` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`):(`:`)?
              $source `[` $source_offset `]` `,`
              $target `[` $target_offset `]` `,`
              $length `:`
              type($source) `->`
              type($target) `` `{` $target_size `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously reads a segment of a file into a resource.

Some implementations can stream directly from the source file into device-local memory and file ops should be preferred to manually staging memory through host buffers.

Traits: Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	a file handle used for I/O operations
`source_offset`	64-bit signless integer
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`length`	index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.file.write` (Stream::FileWriteOp)link

Writes a segment of a file from a resource

Syntax:

operation ::= `stream.file.write` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`):(`:`)?
              $source `[` $source_offset `]` `,`
              $target `[` $target_offset `]` `,`
              $length `:`
              type($source) `` `{` $source_size `}` `->`
              type($target)
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously writes a segment of a resource into a file. The file range must be valid within the file as this operation cannot grow the underlying file storage.

Some implementations can stream directly from device-local memory into the target file and file ops should be preferred to manually staging memory through host buffers.

Traits: Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offset`	index
`target`	a file handle used for I/O operations
`target_offset`	64-bit signless integer
`length`	index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

Miscellaneous opslink

`stream.return` (Stream::ReturnOp)link

Returns results from a region

Syntax:

operation ::= `stream.return` attr-dict
              $operands `:` type($operands)

The values returned are copied by-value.

Traits: AlwaysSpeculatableImplTrait, HasParent<IREE::Stream::ExecutableExportOp>, ReturnLike, Terminator

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), RegionBranchTerminatorOpInterface

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`operands`	variadic of any type

`stream.yield` (Stream::YieldOp)link

Yields stream values from an execution region

Syntax:

operation ::= `stream.yield` attr-dict
              ($resource_operands^ `:`
              custom<ShapedTypeList>(type($resource_operands),
              $resource_operand_sizes))?

The values returned represent the asynchronous value at the point in time the SSA value is defined (or tied).

Traits: AlwaysSpeculatableImplTrait, HasParent<IREE::Stream::AsyncExecuteOp, IREE::Stream::AsyncConcurrentOp, IREE::Stream::CmdExecuteOp, IREE::Stream::CmdSerialOp, IREE::Stream::CmdConcurrentOp>, SameVariadicOperandSize, Terminator

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), RegionBranchTerminatorOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_operand_sizes`	variadic of index

Pseudo Opslink

Pseudo ops for conversion support.

`stream.tensor.export` (Stream::TensorExportOp)link

Conversion placeholder for stream->other type conversion

Syntax:

operation ::= `stream.tensor.export` (`on` `(` $affinity^ `)`)?
              $source `:`
              $source_encoding (`` `{` $source_encoding_dims^ `}`)?
              `in`
              type($source) `` `{` $source_size `}`
              `->`
              type($result)
              attr-dict-with-keyword

Defines a conversion to a higher-level dialect type such as tensor that is resolved during lowering into the stream dialect. This can be used to interoperate between levels of the stack that require specifying stream types and those that prior to lowering do not handle them.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, TiedOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`source_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource or staging resource
`source_encoding_dims`	variadic of index
`source_size`	index

Results:link

Result	Description
`result`	any type

`stream.tensor.import` (Stream::TensorImportOp)link

Conversion placeholder for other->stream type conversion

Syntax:

operation ::= `stream.tensor.import` (`on` `(` $affinity^ `)`)?
              $source `:`
              type($source)
              `->`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result) `{` $result_size `}`
              attr-dict-with-keyword

Defines a conversion from a higher-level dialect type such as tensor that is resolved during lowering into the stream dialect. This can be used to interoperate between levels of the stack that require specifying stream types and those that prior to lowering do not handle them.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, TiedOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	any type
`result_encoding_dims`	variadic of index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource or staging resource

Resource opslink

Generic resource ops.

`stream.resource.alloc` (Stream::ResourceAllocOp)link

Allocates a persistent resource

Syntax:

operation ::= `stream.resource.alloc` (`uninitialized` $uninitialized^)?
              (`on` `(` $affinity^ `)`)?
              attr-dict `:`
              type($result) `{` $storage_size `}`

Allocates a persistent value (one that is long-lived and possibly external to the program) with undefined contents. Consumers of the allocated result must assume nothing of the contents and use discard access.

Uninitialized allocations will have undefined contents and must only be used when all bytes are discarded prior to any reads. Runtimes decide what "undefined contents" means and here it only indicates that execution will be correct even if the memory starts with non-zero values.

If multiple values are allocated from the same operation it implies that they have matching lifetimes. When lowering to execution environments the separate allocations may be fused into one or more slab allocations in order to reduce overheads. How many allocations can be fused is based on the size of the individual resources and the target constraints (how large any single buffer may be, etc).

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, MemoryEffectOpInterface (MemoryEffectOpInterface), Stream_AffinityOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Allocate on ::mlir::SideEffects::DefaultResource}

Attributes:link

Attribute	MLIR Type	Description
`uninitialized`	::mlir::UnitAttr	unit attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`storage_size`	index

Results:link

Result	Description
`result`	any stream-compatible type

`stream.resource.alloca` (Stream::ResourceAllocaOp)link

Allocates a transient value with undefined contents

Syntax:

operation ::= `stream.resource.alloca` `uninitialized`
              (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`):(`:`)?
              attr-dict
              type($result) `{` $storage_size `}`
              `=` `` `>`
              type($result_timepoint)

Allocates a transient value (one that is short-lived and local to the current computation) with undefined contents. Consumers of the allocated result must assume nothing of the contents and use discard access.

The resource returned is not valid for use until the timepoint is reached; execution using this resource must await on the timepoint.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, MemoryEffectOpInterface (MemoryEffectOpInterface), Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Allocate on ::mlir::SideEffects::DefaultResource}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`storage_size`	index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result`	any stream-compatible type
`result_timepoint`	a timepoint indicating execution availability

`stream.resource.constants` (Stream::ResourceConstantsOp)link

Asynchronously uploads or maps constant values

Syntax:

operation ::= `stream.resource.constants` (`on` `(` $affinity^ `)`)?
              attr-dict `:`
              custom<ConstantValueList>(type($results),
              $result_sizes,
              $values)
              `\n` ` ` ` ` `=` `` `>` type($result_timepoint)

Represents an upload of constant resources that may be packed, suballocated, and mapped depending on the final lowering target.

In runtime environments where memory is shared between host and device this turns into a mapping operation that avoids additional memory allocation and copies. When memory cannot be shared an asynchronous stream will be created to allocate and copy all of the constant values.

Though this op returns a unique resource for each constant value it's expected that almost all end up aliasing into the same storage. The exact packing and number of storage resources that are needed are not known until lowering to a particular backend, though, so they are separate here for proper usage tracking.

Both constant and variable resources can be produced; a constant is immutable while a variable will be treated as a constant-value initializer for a mutable resource. By modeling these together it's not required that variable initializers first be allocated, copied to the target, and then copied into the variable storage if the target is capable of doing a direct upload or mapping.

Traits: AlwaysSpeculatableImplTrait, SameVariadicResultSize

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`values`	::mlir::ArrayAttr	constant value array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`result_sizes`	variadic of index

Results:link

Result	Description
`results`	variadic of constant resource or variable resource
`result_timepoint`	a timepoint indicating execution availability

`stream.resource.dealloca` (Stream::ResourceDeallocaOp)link

Frees a transient value when available

Syntax:

operation ::= `stream.resource.dealloca` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              $operand `:` type($operand) `{` $operand_size `}`
              `=` `` `>` type($result_timepoint)
              attr-dict

Deallocates a transient value (one that is short-lived and local to the current computation) previously allocated using stream.resource.alloca.

The resource is considered live and valid until the provided timepoint is reached and the memory is only made available for future requests after the result timepoint is reached.

Interfaces: InferTypeOpInterface, MemoryEffectOpInterface (MemoryEffectOpInterface), Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Free on ::mlir::SideEffects::DefaultResource}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`operand`	any stream-compatible type
`operand_size`	index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.resource.load` (Stream::ResourceLoadOp)link

Loads a value from a staging resource

Syntax:

operation ::= `stream.resource.load` $source `[` $source_offset `]` `:`
              type($source) `` `{` $source_size `}`
              `->`
              type($result)
              attr-dict-with-keyword

Returns the element(s) at the given offset in the staging resource. The operation will complete synchronously against the resource though it may introduce a yield point if the staging resource needs to be transferred.

Interfaces: Util_SizeAwareOp

Operands:link

Operand	Description
`source`	staging resource
`source_size`	index
`source_offset`	index

Results:link

Result	Description
`result`	index or integer or floating-point or complex-type or vector of any type values

`stream.resource.pack` (Stream::ResourcePackOp)link

Packs variable-sized slices into a single slab

Syntax:

operation ::= `stream.resource.pack` (`on` `(` $affinity^ `)`)?
              (`offset` `(` $offset^ `)`)?
              `slices` `(` `{`
              custom<PackSliceRanges>($lifetime_intervals,
              $dynamic_slice_sizes,
              type($packed_offsets))
              `}` `)`
              `:` type($total_length)
              attr-dict-with-keyword

Performs a greedy packing of one or more sized slices with specified lifetimes and returns their relative offsets in an aliased linear space.

Slices are [start, end] = %slice_byte_size, where the start and end values define an inclusive lifetime range and the size is the total number of bytes required to be live for that range.

// Computes the total length required for the packed values and the offsets
// of the 3 slices requested relative to the base of the packed memory:
%total_length, %offset_0, %offset_1, %offset_2 =
    stream.resource.pack
        // Each slice gets one result offset:
        slices({
          // 3 slices where A and B overlap and will get unique offsets
          // while B and C do not overlap and are allowed to alias.
          [0, 10] = %size_0,  // A => %offset_0
          [3,  8] = %size_1,  // B => %offset_1
          [9, 10] = %size_2,  // C => %offset_2
          ...
        }) : index

The lifetime start and end points (inclusive) are only used for relative comparisons and may originate with any meaning (op order in block, epoch, phase of the moon, etc). The packing algorithm uses the intervals to determine slice liveness and when aliasing is safe.

The size of each slice may either be a constant or runtime-computed dynamic value. Constant slices can achieve more dense packing than the dynamic values and CSE/canonicalization should be applied to ensure that as many of the dynamic values are equivalent if possible.

The total length required to pack all slices is returned and can be used to acquire storage. The individual slice offsets are 0-based and as such if are directly used as buffer offsets may need additional offsetting. This can either be applied via the optional offset operand or slicing of the underlying allocation buffer.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`lifetime_intervals`	::mlir::ArrayAttr	index array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`offset`	index
`dynamic_slice_sizes`	variadic of index

Results:link

Result	Description
`total_length`	index
`packed_offsets`	variadic of index

`stream.resource.size` (Stream::ResourceSizeOp)link

Returns the size of the resource storage in bytes

Syntax:

operation ::= `stream.resource.size` $operand
              attr-dict `:` type($operand)

Returns a possibly runtime-dynamic byte size of the resource backing storage. This may differ from the logical storage size of a value based on the alignment requirements of the target as well as encoding of higher level values such as sparse tensor formats.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`operand`	any stream-compatible type

Results:link

Result	Description
`result`	index

`stream.resource.store` (Stream::ResourceStoreOp)link

Stores a value into a staging resource

Syntax:

operation ::= `stream.resource.store` $value `,`
              $target `[` $target_offset `]` `:`
              type($value)
              `->`
              type($target) `{` $target_size `}`
              attr-dict-with-keyword

The operation will complete synchronously against the resource though it may introduce a yield point if the staging resource needs to be acquired.

Interfaces: MemoryEffectOpInterface (MemoryEffectOpInterface), Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Write on ::mlir::SideEffects::DefaultResource}

Operands:link

Operand	Description
`target`	staging resource
`target_size`	index
`target_offset`	index
`value`	index or integer or floating-point or complex-type or vector of any type values

`stream.resource.subview` (Stream::ResourceSubviewOp)link

Slices out a cloned subview of a value

Syntax:

operation ::= `stream.resource.subview` $source `[` $source_offset `]` `:`
              type($source) `` `{` $source_size `}` `->`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Aliases a byte subrange of a resource.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), StreamableOpInterface, TiedOpInterface, Util_SizeAwareOp, Util_SubrangeOp, ViewLikeOpInterface

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`source`	any stream-compatible type
`source_size`	index
`source_offset`	index
`result_size`	index

Results:link

Result	Description
`result`	any stream-compatible type

`stream.resource.try_map` (Stream::ResourceTryMapOp)link

Maps read-only memory into a resource

Syntax:

operation ::= `stream.resource.try_map` (`on` `(` $affinity^ `)`)?
              $source `[` $source_offset `]` `:`
              type($source)
              `->`
              type($did_map) `,` type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Synchronously maps a host heap buffer into a stream-accessible resource with the requested lifetime. If the given source cannot be mapped the did_map result will be 0 and users must find another route into memory (such as file I/O). The resulting resource is not coherent with the source and behavior is undefined if the underlying contents change.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, MemoryEffectOpInterface (MemoryEffectOpInterface), NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{MemoryEffects::Allocate on ::mlir::SideEffects::DefaultResource}, MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	a reference counted byte buffer
`source_offset`	index
`result_size`	index

Results:link

Result	Description
`did_map`	1-bit signless integer
`result`	resource or external resource or transient resource or variable resource or constant resource

Resource parameter I/O opslink

Resource parameter I/O ops.

`stream.parameter.gather` (Stream::ParameterGatherOp)link

Gathers multiple resources from a parameter scope

Syntax:

operation ::= `stream.parameter.gather` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              `{`
              custom<ParameterGatherOperations>(
              $source_scope, $source_keys, $source_offsets,
              $target, type($target), $target_size, $target_offsets, $target_lengths)
              `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously gathers one or more resources into a single target stream resource. This is equivalent to one stream.parameter.read per parameter but allows implementations that can batch operations to do so without additional timeline overhead.

Traits: AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`source_scope`	::mlir::StringAttr	string attribute
`source_keys`	::mlir::ArrayAttr	string array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source_offsets`	variadic of 64-bit signless integer
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offsets`	variadic of index
`target_lengths`	variadic of index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.parameter.load` (Stream::ParameterLoadOp)link

Reads one or more resources from a parameter scope

Syntax:

operation ::= `stream.parameter.load` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              `{`
              custom<ParameterLoadOperations>(
              $source_scope, $source_keys, $source_offsets,
              type($results), $result_sizes)
              `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously reads one or more resources from an external parameter provider and returns the resulting stream resources. Depending on the resource type this may alias existing cached storage or be directly mapped to the parameter origin or result in a copy as if stream.resource.alloca and stream.parameter.read had been used per parameter.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`source_scope`	::mlir::StringAttr	string attribute
`source_keys`	::mlir::ArrayAttr	string array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source_offsets`	variadic of 64-bit signless integer
`result_sizes`	variadic of index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource
`result_timepoint`	a timepoint indicating execution availability

`stream.parameter.read` (Stream::ParameterReadOp)link

Reads a resource from a parameter scope

Syntax:

operation ::= `stream.parameter.read` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              custom<ParameterReference>($source_scope, $source_key)
              `` `[` $source_offset `]` `->`
              $target `[` $target_offset `for` $target_length `]` `:`
              type($target) `` `{` $target_size `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously reads a resource from an external parameter provider into the provided target resource range.

Traits: Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`source_scope`	::mlir::StringAttr	string attribute
`source_key`	::mlir::StringAttr	string attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source_offset`	64-bit signless integer
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_size`	index
`target_offset`	index
`target_length`	index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.parameter.scatter` (Stream::ParameterScatterOp)link

Scatters multiple resources to a parameter scope

Syntax:

operation ::= `stream.parameter.scatter` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              `{`
              custom<ParameterScatterOperations>(
              $source, type($source), $source_size, $source_offsets, $source_lengths,
              $target_scope, $target_keys, $target_offsets)
              `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously scatters one or more resources from a single source resource into one or more parameters. This is equivalent to one stream.parameter.write per parameter but allows implementations that can batch operations to do so without additional overhead.

Traits: AttrSizedOperandSegments, Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`target_scope`	::mlir::StringAttr	string attribute
`target_keys`	::mlir::ArrayAttr	string array attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offsets`	variadic of index
`source_lengths`	variadic of index
`target_offsets`	variadic of 64-bit signless integer
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.parameter.write` (Stream::ParameterWriteOp)link

Writes a resource to a parameter scope

Syntax:

operation ::= `stream.parameter.write` (`on` `(` $affinity^ `)`)?
              (`await` `(` $await_timepoint^ `)` `=` `` `>`)?
              $source `[` $source_offset `for` $source_length `]` `:`
              type($source) `` `{` $source_size `}` `->`
              custom<ParameterReference>($target_scope, $target_key)
              `` `[` $target_offset `]`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Asynchronously writes a resource to an external parameter provider from the provided source resource range.

Traits: Stream_CmdPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_TimelineOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`target_scope`	::mlir::StringAttr	string attribute
`target_key`	::mlir::StringAttr	string attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_size`	index
`source_offset`	index
`source_length`	index
`target_offset`	64-bit signless integer
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

Synchronization opslink

`stream.timepoint.await` (Stream::TimepointAwaitOp)link

Awaits a timepoint before returning a set of resources

Syntax:

operation ::= `stream.timepoint.await` $await_timepoint `=` `` `>`
              $resource_operands `:`
              custom<ShapedTypeList>(type($resource_operands),
              type($results), $resource_operand_sizes)
              attr-dict-with-keyword

After asynchronous execution scheduling resources may exist in different states at different points in the execution timeline. This op enables resolving the version of a resource after a particular point in the timeline. As timepoints transitively chain the timepoint must only cover the resource availability but not be limited to its original production timepoint.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_TimelineOp, TiedOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`resource_operands`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_operand_sizes`	variadic of index
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`results`	variadic of resource or external resource or transient resource or variable resource or constant resource or staging resource

`stream.timepoint.barrier` (Stream::TimepointBarrierOp)link

Returns a timepoint indicating when a resource is available

Syntax:

operation ::= `stream.timepoint.barrier` (`on` `(` $affinity^ `)`)?
              $resource `:` type($resource) `` `{` $resource_size `}`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

After asynchronous execution scheduling resources may exist in different states at different points in the execution timeline. This op enables identifying when the version of a resource after a particular point in the timeline is available. As timepoints transitively chain the timepoint must only cover the resource availability but not be limited to its original production timepoint.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_TimelineOp, TiedOpInterface, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`resource`	resource or external resource or transient resource or variable resource or constant resource or staging resource
`resource_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource or staging resource
`result_timepoint`	a timepoint indicating execution availability

`stream.timepoint.chain_external` (Stream::TimepointChainExternalOp)link

Exports a timepoint to an external dialect type

Syntax:

operation ::= `stream.timepoint.chain_external` (`on` `(` $affinity^ `)`)?
              $await_timepoint
              `=` `` `>`
              `(` $external_values `:` type($external_values) `)`
              attr-dict-with-keyword

Defines a conversion to an external dialect type such as hal.fence that is resolved during lowering into the stream dialect. This can be used to interoperate between levels of the stack that require specifying stream types and those that prior to lowering do not handle them.

Interfaces: Stream_AffinityOp

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`await_timepoint`	a timepoint indicating execution availability
`external_values`	variadic of any type

`stream.timepoint.export` (Stream::TimepointExportOp)link

Exports a timepoint to an external dialect type

Syntax:

operation ::= `stream.timepoint.export` (`on` `(` $affinity^ `)`)?
              $await_timepoint
              `=` `` `>`
              `(` type($results) `)`
              attr-dict-with-keyword

Defines a conversion to an external dialect type such as hal.fence that is resolved during lowering into the stream dialect. This can be used to interoperate between levels of the stack that require specifying stream types and those that prior to lowering do not handle them.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`await_timepoint`	a timepoint indicating execution availability

Results:link

Result	Description
`results`	variadic of any type

`stream.timepoint.immediate` (Stream::TimepointImmediateOp)link

Results an immediately-available timepoint

Syntax:

operation ::= `stream.timepoint.immediate` attr-dict
              `=` `` `>` type($result_timepoint)

Timepoints indicate a point in the execution timeline and this op can be used to get a placeholder representing the start of the timeline. Any waits on the returned timepoint will resolve immediately. This generally folds away but can be useful if needing to initialize globals or branch args.

Traits: AlwaysSpeculatableImplTrait, ConstantLike

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_TimelineOp

Effects: MemoryEffects::Effect{}

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.timepoint.import` (Stream::TimepointImportOp)link

Imports a timepoint from an external dialect type

Syntax:

operation ::= `stream.timepoint.import` (`on` `(` $affinity^ `)`)?
              $operands `:` `(` type($operands) `)`
              `=` `` `>`
              type($result_timepoint)
              attr-dict-with-keyword

Defines a conversion from an external dialect type such as hal.semaphore that is resolved during lowering into the stream dialect. This can be used to interoperate between levels of the stack that require specifying stream types and those that prior to lowering do not handle them.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`operands`	variadic of any type

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

`stream.timepoint.join` (Stream::TimepointJoinOp)link

Joins one or more timepoints into the max of all of them

Syntax:

operation ::= `stream.timepoint.join` `max` `(` $await_timepoints `)` `=` `` `>` type($result_timepoint)
              attr-dict-with-keyword

Returns a timepoint that indicates that all of the input timepoints have been reached.

Traits: AlwaysSpeculatableImplTrait

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_TimelineOp

Effects: MemoryEffects::Effect{}

Operands:link

Operand	Description
`await_timepoints`	variadic of a timepoint indicating execution availability

Results:link

Result	Description
`result_timepoint`	a timepoint indicating execution availability

Tensor opslink

`stream.tensor.clone` (Stream::TensorCloneOp)link

Clones the contents of a value

Syntax:

operation ::= `stream.tensor.clone` (`on` `(` $affinity^ `)`)?
              $source `:`
              $source_encoding (`` `{` $source_encoding_dims^ `}`)?
              `in`
              type($source) `` `{` $source_size `}`
              `->`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Clones the contents of a value at a snapshot in time. Future changes to the cloned value will not effect the result. Acts as a copy-on-write operation.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_StreamableOp, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`source_encoding`	::mlir::TypeAttr	any type attribute
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_encoding_dims`	variadic of index
`source_size`	index
`result_encoding_dims`	variadic of index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.constant` (Stream::TensorConstantOp)link

Defines a constant tensor value

Syntax:

operation ::= `stream.tensor.constant` (`on` `(` $affinity^ `)`)?
              `:`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result)
              `=`
              $value
              attr-dict-with-keyword

Returns a typed resource initialized to the given constant value.

Traits: AlwaysSpeculatableImplTrait, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp, Stream_StreamableOp, Util_ShapeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`value`	::mlir::TypedAttr	TypedAttr instance
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`result_encoding_dims`	variadic of index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.empty` (Stream::TensorEmptyOp)link

Defines an empty tensor value

Syntax:

operation ::= `stream.tensor.empty` (`on` `(` $affinity^ `)`)?
              `:`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Returns a typed resource initialized with no contents. This still carries shape metadata and may encode to a non-empty resource such as in cases where the empty representation still has data (e.g. sparse tensors). Subsequent writes must populate any ranges of the tensor that are later read.

Traits: AlwaysSpeculatableImplTrait, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), OpAsmOpInterface, Stream_AffinityOp, StreamableOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`result_encoding_dims`	variadic of index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.fill` (Stream::TensorFillOp)link

Fills a subview of a stream resource with a value

Syntax:

operation ::= `stream.tensor.fill` (`on` `(` $affinity^ `)`)?
              $value `,` $target (`[` $start_indices `for` $lengths^ `]`)? `:`
              type($value)
              `->`
              $target_encoding (`` `{` $target_encoding_dims^ `}`)?
              `in`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Splats a value into a subview of the given stream resource and returns the resource with the update applied.

Equivalent to a stream.tensor.splat + stream.tensor.update.

Traits: AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`target_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_encoding_dims`	variadic of index
`target_size`	index
`start_indices`	variadic of index
`lengths`	variadic of index
`value`	index or integer or floating-point or complex-type

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.load` (Stream::TensorLoadOp)link

Loads a value from a tensor element

Syntax:

operation ::= `stream.tensor.load` $source (`[` $indices^ `]`)? `:`
              $source_encoding (`` `{` $source_encoding_dims^ `}`)?
              `in`
              type($source) `` `{` $source_size `}`
              `->`
              type($result)
              attr-dict-with-keyword

Returns the element at the given location from within the tensor.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`source_encoding`	::mlir::TypeAttr	any type attribute

Operands:link

Operand	Description
`source`	staging resource
`source_encoding_dims`	variadic of index
`source_size`	index
`indices`	variadic of index

Results:link

Result	Description
`result`	index or integer or floating-point or complex-type or vector of any type values

`stream.tensor.sizeof` (Stream::TensorSizeOfOp)link

Calculates the storage size of a given high-level type

Syntax:

operation ::= `stream.tensor.sizeof` (`on` `(` $affinity^ `)`)?
              $encoding (`{` $encoding_dims^ `}`)?
              attr-dict `:` type($storage_size)

Target-dependent storage size calculation using a high-level annotated type. While within the stream dialect the storage size of a value is left as a placeholder using this op. The requisite target-specific parameters for expanding the size calculation are only available after affinities have been assigned.

Traits: AlwaysSpeculatableImplTrait, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`encoding_dims`	variadic of index

Results:link

Result	Description
`storage_size`	index

`stream.tensor.slice` (Stream::TensorSliceOp)link

Slices out a cloned subview of a value

Syntax:

operation ::= `stream.tensor.slice` (`on` `(` $affinity^ `)`)?
              $source `[` $start_indices `for` $lengths `]` `:`
              $source_encoding (`` `{` $source_encoding_dims^ `}`)?
              `in`
              type($source) `` `{` $source_size `}`
              `->`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Slices a subrange of a stream resource based on a tensor encoding. Acts as a copy-on-write operation.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, Stream_StreamableOp, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`source_encoding`	::mlir::TypeAttr	any type attribute
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`source`	resource or external resource or transient resource or variable resource or constant resource
`source_encoding_dims`	variadic of index
`source_size`	index
`start_indices`	variadic of index
`lengths`	variadic of index
`result_encoding_dims`	variadic of index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.splat` (Stream::TensorSplatOp)link

Splats a value into a shaped tensor

Syntax:

operation ::= `stream.tensor.splat` (`on` `(` $affinity^ `)`)?
              $value
              `:` type($value)
              `->`
              $result_encoding (`` `{` $result_encoding_dims^ `}`)?
              `in`
              type($result) `` `{` $result_size `}`
              attr-dict-with-keyword

Returns a typed resource initialized to the given primitive value.

Traits: AlwaysSpeculatableImplTrait, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, NoMemoryEffect (MemoryEffectOpInterface), Stream_AffinityOp, StreamableOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`result_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`value`	index or integer or floating-point or complex-type
`result_encoding_dims`	variadic of index
`result_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

`stream.tensor.store` (Stream::TensorStoreOp)link

Stores a value into a tensor element

Syntax:

operation ::= `stream.tensor.store` $value `,`
              $target (`[` $indices^ `]`)? `:`
              type($value)
              `->`
              $target_encoding (`` `{` $target_encoding_dims^ `}`)?
              `in`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Returns a tensor with the element at the given index set to the given value.

Traits: AlwaysSpeculatableImplTrait, AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: ConditionallySpeculatable, InferTypeOpInterface, NoMemoryEffect (MemoryEffectOpInterface), TiedOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Effects: MemoryEffects::Effect{}

Attributes:link

Attribute	MLIR Type	Description
`target_encoding`	::mlir::TypeAttr	any type attribute

Operands:link

Operand	Description
`target`	staging resource
`target_encoding_dims`	variadic of index
`target_size`	index
`indices`	variadic of index
`value`	index or integer or floating-point or complex-type or vector of any type values

Results:link

Result	Description
`result`	staging resource

`stream.tensor.trace` (Stream::TensorTraceOp)link

Traces one or more tensor values at runtime

Syntax:

operation ::= `stream.tensor.trace` $key `=` `[`
              custom<EncodedResourceOperands>(
              $resources, type($resources), $resource_sizes,
              $resource_encodings, $resource_encoding_dims)
              `]` attr-dict-with-keyword

Traces out to a runtime trace sink (console, log file, etc) the given tensors. The key is arbitrary and can be used for identifying the set of values being traced.

Traits: AttrSizedOperandSegments

Interfaces: ShapeAwareOpInterface, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`key`	::mlir::StringAttr	string attribute
`resource_encodings`	::mlir::ArrayAttr	type array attribute

Operands:link

Operand	Description
`resources`	variadic of staging resource
`resource_sizes`	variadic of index
`resource_encoding_dims`	variadic of index

`stream.tensor.update` (Stream::TensorUpdateOp)link

Updates a slice of a subview of a resource in-place

Syntax:

operation ::= `stream.tensor.update` (`on` `(` $affinity^ `)`)?
              $update `,` $target `[` $start_indices `]` `:`
              $update_encoding (`` `{` $update_encoding_dims^ `}`)?
              `in`
              type($update) `` `{` $update_size `}`
              `->`
              $target_encoding (`` `{` $target_encoding_dims^ `}`)?
              `in`
              custom<ShapedTiedResult>(type($target), $target_size)
              attr-dict-with-keyword

Copies a value into a resource based on tensor encodings. The returned value is the entire updated target value.

Traits: AttrSizedOperandSegments, Stream_TensorPhaseOp

Interfaces: InferTypeOpInterface, Stream_AffinityOp, Stream_StreamableOp, TiedOpInterface, Util_ShapeAwareOp, Util_SizeAwareOp

Attributes:link

Attribute	MLIR Type	Description
`target_encoding`	::mlir::TypeAttr	any type attribute
`update_encoding`	::mlir::TypeAttr	any type attribute
`affinity`	::mlir::iree_compiler::IREE::Stream::AffinityAttr	defines execution context affinity

Operands:link

Operand	Description
`target`	resource or external resource or transient resource or variable resource or constant resource
`target_encoding_dims`	variadic of index
`target_size`	index
`start_indices`	variadic of index
`update`	resource or external resource or transient resource or variable resource or constant resource
`update_encoding_dims`	variadic of index
`update_size`	index

Results:link

Result	Description
`result`	resource or external resource or transient resource or variable resource or constant resource

Attributeslink

CollectiveAttrlink

collective operation and specification

Syntax:

#stream.collective<
  CollectiveKind,   # kind
  std::optional<CollectiveReductionOp>,   # reduction
  CollectiveElementType   # element_type
>

Specifies the collective operation to perform and any mode bits required.

Parameters:link

Parameter	C++ type	Description
kind	`CollectiveKind`
reduction	`std::optional<CollectiveReductionOp>`
element_type	`CollectiveElementType`

NamedParameterAttrlink

named parameter referenced an optional scope and key

Syntax:

#stream.parameter.named<
  ::mlir::Type,   # type
  StringAttr,   # scope
  StringAttr,   # key
  DictionaryAttr   # config
>

Species an externally-defined parameter that can be referenced by an optional scope defining a set of parameters and a key uniquely identifying the parameter within its scope.

Parameters:link

Parameter	C++ type	Description
type	`::mlir::Type`
scope	`StringAttr`
key	`StringAttr`
config	`DictionaryAttr`

PartitioningConfigAttrlink

defines partitioning configuration

Configures the partitioning algorithm to use and its configuration. Partitioning is useful to adjust when scheduling behavior of targets is radically different - such as single-threaded vs. multi-threaded CPUs or bespoke ML accelerators vs. general purpose GPUs. This mechanism controls the amount of concurrency, parallelism, memory consumption, and latency.

Parameters:link

Parameter	C++ type	Description
favor	`IREE::Stream::FavorAttr`

ResourceConfigAttrlink

defines resource constraints configuration

Defines resource storage constraints. These allow for packing and layout algorithms to ensure they are producing usable results on target devices.

Parameters:link

Parameter	C++ type	Description
maxAllocationSize	`int64_t`
minBufferOffsetAlignment	`int64_t`
maxBufferRange	`int64_t`
minBufferRangeAlignment	`int64_t`
indexBits	`int64_t`
aliasMutableBindings	`bool`
memoryModel	`IREE::Stream::MemoryModel`

TimepointAttrlink

an immediately-resolved timepoint

Parameters:link

Parameter	C++ type	Description
type	`::mlir::Type`

Type constraintslink

constant resourcelink

Stream constants are immutable values that are available for the lifetime of the program once initialized.

external resourcelink

Stream external values represent asynchronously-available and sequenced values that are owned and managed by external code - such as those passed in or out of the program entry points. Though external values are managed during an invocation the same as other stream values the visibility into them does not extend outside of the invocation they are provided to.

Stream values are not usable directly outside of a stream execution or transfer operation. If the contents of the value are needed they must first be transferred via stream.transfer - which may incur a copy.

staging resourcelink

Stream upload/download staging resource. These are used outside of streams and then transferred to other stream resources such as variables or transients for use inside of streams. Dispatches and several other operations cannot directly operate on these resources.

transient resourcelink

Stream transients represent asynchronously-available and sequenced values that have a short lifetime - often only passed between stream executions. It is expected that transient values are not stored in global state and have minimal lifetime as they may be heavily pooled or suballocated.

Stream values are not usable directly outside of a stream execution or transfer operation. If the contents of the value are needed they must first be transferred via stream.transfer - which may incur a copy.

resourcelink

A stream resource that has not yet had its lifetime calculated.

variable resourcelink

Stream variables represent asynchronously-available and sequenced values that have a long lifetime relative to the work being performed on them. These variables are often stored in global state and may live for the entire duration of the program.

Stream values are not usable directly outside of a stream execution or transfer operation. If the contents of the value are needed they must first be transferred via stream.transfer - which may incur a copy.

Typeslink

BindingTypelink

a managed resource binding into an executable scope

Syntax: !stream.binding

A resource binding available within an executable dispatch function. The bindings map 1:1 with the resources bound during dispatch operations.

ChannelTypelink

a collective communication channel

Syntax: !stream.channel

Represents a single participant in a collective clique. Multiple channels may exist within the same program to allow for partial operations or hierarchical operations.

In programs that model SPMD behavior internally channels can be created or provided by hosting applications. For example, the program could expose a @set_channels(!util.list<!stream.channel>) method that stores the channels in globals for use throughout the program allowing for application-controlled channel configuration.

FileTypelink

a file handle used for I/O operations

Syntax: !stream.file

A file handle that can be asynchronously read and written into/from stream resources.

ResourceTypelink

a managed resource

Stream external values represent asynchronously-available and sequenced values that are owned and managed by external code - such as those passed in or out of the program entry points. Though external values are managed during an invocation the same as other stream values the visibility into them does not extend outside of the invocation they are provided to.

Stream values are not usable directly outside of a stream execution or transfer operation. If the contents of the value are needed they must first be transferred via stream.transfer - which may incur a copy.

Parameters:link

Parameter	C++ type	Description
lifetime	`IREE::Stream::Lifetime`

TimepointTypelink

a timepoint indicating execution availability

Syntax: !stream.timepoint

Represents a point in the execution timeline that when resolved indicates that all of the execution prior to this timepoint has completed and the results of the execution are available for use. This includes transitive dependencies as well; if timepoint B is dependent on timepoint A then when B is available so too must be A.

Enumslink

CollectiveElementTypelink

valid CollectiveElementType

Cases:link

Symbol	Value	String
Sint8	`0`	si8
Uint8	`1`	ui8
Sint16	`2`	si16
Uint16	`3`	ui16
Sint32	`4`	si32
Uint32	`5`	ui32
Sint64	`6`	si64
Uint64	`7`	ui64
Float16	`8`	f16
Float32	`9`	f32
Float64	`10`	f64
BFloat16	`11`	bf16

CollectiveKindlink

valid CollectiveKind

Cases:link

Symbol	Value	String
AllGather	`0`	all_gather
AllReduce	`1`	all_reduce
AllToAll	`2`	all_to_all
Broadcast	`3`	broadcast
Reduce	`4`	reduce
ReduceScatter	`5`	reduce_scatter
Send	`6`	send
Recv	`7`	recv
SendRecv	`8`	send_recv

CollectiveReductionOplink

valid CollectiveReductionOp

Cases:link

Symbol	Value	String
None	`0`	none
ReductionSum	`1`	sum
ReductionProduct	`2`	product
ReductionMinimum	`3`	minimum
ReductionMaximum	`4`	maximum
ReductionAverage	`5`	average

Favorlink

IREE partitioning bias

Cases:link

Symbol	Value	String
Debug	`0`	debug
MinPeakMemory	`1`	min-peak-memory
MaxConcurrency	`2`	max-concurrency

Lifetimelink

IREE Stream value lifetime

Cases:link

Symbol	Value	String
Unknown	`0`	*
External	`1`	external
Staging	`2`	staging
Transient	`3`	transient
Variable	`4`	variable
Constant	`5`	constant

MemoryModellink

stream resource memory model

Cases:link

Symbol	Value	String
Unified	`0`	Unified
Discrete	`1`	Discrete

ResourceAccessBitfieldlink

valid ResourceAccess

Cases:link

Symbol	Value	String
None	`0`	None
Read	`1`	Read
Write	`2`	Write

'stream' Dialectlink

Operationslink

Async control flow opslink

stream.async.call (Stream::AsyncCallOp)link

Attributes:link

Operands:link

Results:link

stream.async.concurrent (Stream::AsyncConcurrentOp)link

Attributes:link

Operands:link

Results:link

stream.async.execute (Stream::AsyncExecuteOp)link

Attributes:link

Operands:link

Results:link

stream.async.func (Stream::AsyncFuncOp)link

Attributes:link

Async opslink

stream.async.alloca (Stream::AsyncAllocaOp)link

Attributes:link

Operands:link

Results:link

stream.async.clone (Stream::AsyncCloneOp)link

Attributes:link

Operands:link

Results:link

stream.async.collective (Stream::AsyncCollectiveOp)link

Attributes:link

Operands:link

Results:link

stream.async.constant (Stream::AsyncConstantOp)link

Attributes:link

Operands:link

Results:link

stream.async.copy (Stream::AsyncCopyOp)link

Attributes:link

Operands:link

Results:link

stream.async.dispatch (Stream::AsyncDispatchOp)link

Attributes:link

Operands:link

Results:link

stream.async.fill (Stream::AsyncFillOp)link

Attributes:link

Operands:link

Results:link

stream.async.load (Stream::AsyncLoadOp)link

Operands:link

Results:link

stream.async.slice (Stream::AsyncSliceOp)link

Attributes:link

Operands:link

Results:link

stream.async.splat (Stream::AsyncSplatOp)link

Attributes:link

Operands:link

Results:link

stream.async.store (Stream::AsyncStoreOp)link

Operands:link

Results:link

stream.async.transfer (Stream::AsyncTransferOp)link

Attributes:link

Operands:link

Results:link

stream.async.update (Stream::AsyncUpdateOp)link

Attributes:link

Operands:link

Results:link

Channel opslink

stream.channel.count (Stream::ChannelCountOp)link

Operands:link

Results:link

stream.channel.create (Stream::ChannelCreateOp)link

Attributes:link

Operands:link

Results:link

stream.channel.rank (Stream::ChannelRankOp)link

Operands:link

Results:link

stream.channel.split (Stream::ChannelSplitOp)link

`stream.async.call` (Stream::AsyncCallOp)link

`stream.async.concurrent` (Stream::AsyncConcurrentOp)link

`stream.async.execute` (Stream::AsyncExecuteOp)link

`stream.async.func` (Stream::AsyncFuncOp)link

`stream.async.alloca` (Stream::AsyncAllocaOp)link

`stream.async.clone` (Stream::AsyncCloneOp)link

`stream.async.collective` (Stream::AsyncCollectiveOp)link

`stream.async.constant` (Stream::AsyncConstantOp)link

`stream.async.copy` (Stream::AsyncCopyOp)link

`stream.async.dispatch` (Stream::AsyncDispatchOp)link

`stream.async.fill` (Stream::AsyncFillOp)link

`stream.async.load` (Stream::AsyncLoadOp)link

`stream.async.slice` (Stream::AsyncSliceOp)link

`stream.async.splat` (Stream::AsyncSplatOp)link

`stream.async.store` (Stream::AsyncStoreOp)link

`stream.async.transfer` (Stream::AsyncTransferOp)link

`stream.async.update` (Stream::AsyncUpdateOp)link

`stream.channel.count` (Stream::ChannelCountOp)link

`stream.channel.create` (Stream::ChannelCreateOp)link

`stream.channel.rank` (Stream::ChannelRankOp)link

`stream.channel.split` (Stream::ChannelSplitOp)link

`stream.binding.subspan` (Stream::BindingSubspanOp)link

`stream.dispatch.workgroup.count` (Stream::DispatchWorkgroupCountOp)link

`stream.dispatch.workgroup.id` (Stream::DispatchWorkgroupIDOp)link

`stream.dispatch.workgroup.size` (Stream::DispatchWorkgroupSizeOp)link

`stream.executable.end` (Stream::ExecutableEndOp)link

`stream.executable.export` (Stream::ExecutableExportOp)link

`stream.executable` (Stream::ExecutableOp)link

`stream.context.resolve` (Stream::ContextResolveOp)link

`stream.cmd.call` (Stream::CmdCallOp)link

`stream.cmd.collective` (Stream::CmdCollectiveOp)link

`stream.cmd.concurrent` (Stream::CmdConcurrentOp)link

`stream.cmd.copy` (Stream::CmdCopyOp)link

`stream.cmd.discard` (Stream::CmdDiscardOp)link

`stream.cmd.dispatch` (Stream::CmdDispatchOp)link

`stream.cmd.execute` (Stream::CmdExecuteOp)link

`stream.cmd.fill` (Stream::CmdFillOp)link

`stream.cmd.flush` (Stream::CmdFlushOp)link

`stream.cmd.func` (Stream::CmdFuncOp)link

`stream.cmd.invalidate` (Stream::CmdInvalidateOp)link

`stream.cmd.serial` (Stream::CmdSerialOp)link

`stream.file.constant` (Stream::FileConstantOp)link

`stream.file.read` (Stream::FileReadOp)link

`stream.file.write` (Stream::FileWriteOp)link

`stream.return` (Stream::ReturnOp)link