Skip to content

Integration test debugginglink

This document includes tips for triaging integration test correctness issues. Feel free to reach out to @hanhanW or ask questions on Discord for more help.

General tipslink

Narrow down reproducerslink

  • Models themselves can be large, and IREE breaks models into dispatches/kernels and then launches those individually. Program outputs could diverge starting from any individual launch. To get a smaller reproducer, you can use --iree-flow-trace-dispatch-tensors.
  • You can compare the logs between builds/backends to get an idea about which dispatch results in wrong outputs. The dumped inputs can be reused in a flagfile.

Once a suspicious dispatch is identified, we can create a test case based on the dispatch function. The dispatch function can be derived after the OutlineDispatchRegions pass. The function signatures have to be modified manually. You'll have to put flow.dispatch.tensor.load variables to function arguments, and replace with return op.

Note: This only works when dispatch formation logics are identical between runs.

iree-experimental repository testslink

Follow README to run the model. The MLIR files will be generated. You'll find the saved file from log. E.g.,

[ RUN      ] MobilenetV2Int8Test.test_compile_tflite
I0401 17:27:04.084272 140182373025024] Setting up for IREE
I0401 17:27:04.085064 140182373025024] Invoke IREE Pipeline:

Unfortunately, the artifacts are not dumped in the runs. There is an issue for tracking this. A workaround can be found in the issue.

TensorFlow integration testslink

These are steps to reproduce/address failures in TF/TFLite integration tests. These instructions are most stable on Linux, though they may work with a few tweaks on Windows and macOS.

All steps here assume starting from the IREE root directory.

  1. First create a Python virtual environment to install packages into:

    python -m venv iree-tf.venv
    source iree-tf.venv/bin/activate
    # Install test requirements
    python -m pip install -r ./integrations/tensorflow/test/requirements.txt
  2. Install IREE's tools and Python bindings or build them from source

    Install distributed packages

    # Install packages from nightly releases
    # This should work for most cases, as the importers change infrequently
    python -m pip install \
      iree-compiler iree-runtime iree-tools-tf iree-tools-tflite \

    OR build from source

    # Build Python bindings from source
    cmake -G Ninja -B ../iree-build/ -DIREE_BUILD_PYTHON_BINDINGS=ON .
    cmake --build ../iree-build/
    # Add IREE built-from-source Python packages to PYTHONPATH
    source .env
    # Install IREE TF/TFLite Python packages
    python -m pip install integrations/tensorflow/python_projects/iree_tf
    python -m pip install integrations/tensorflow/python_projects/iree_tflite
  3. Run the python test command line

    The command can be obtained from the run file. For example, if iree_tfl_tests/ failed,

    cd integrations/tensorflow/test/
    cat iree_tfl_tests/
    # REQUIRES: llvmcpu
    # RUN: %PYTHON -m iree_tfl_tests.posenet_i8_test --target_backend=llvmcpu --artifacts_dir=%t
    cd python/
    python -m iree_tfl_tests.posenet_i8_test --target_backend=llvmcpu --artifacts_dir=/tmp/posenet_i8_failure

    Note that the command can only be run under integrations/tensorflow/test/python directory.

  4. Extract intermediate files and use with native tools

    The test will create an iree_input.mlir in the temp directory specified. Those can then be fed into iree-compile (built locally to reproduce the error)

    iree-compile \
      --iree-hal-target-backends=llvm-cpu \
      --iree-input-type=stablehlo \