Skip to content

GitHub Actionslink

Overviewlink

We use GitHub Actions for continuous automation (CI) and continuous delivery (CD) workflows:

  • Code formating and linting.
  • Building from source and running tests.
  • Building packages for testing and releases.
  • Testing packages across a variety of platforms.
  • Publishing the https://iree.dev website.
  • Updating dependencies.

Workflows are defined directly in the repository at .github/workflows/. We use a mix of GitHub-hosted runners and self-hosted runners to get automated build and test coverage across a variety of platforms and hardware accelerators.

Terminology primerlink

(Read more on https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions)

  • Workflows are configurable automated processes that run one or more jobs.
  • Jobs are a set of steps in a workflow that are executed on the same runner.
  • Steps are lists of commands or meta actions to run in a shell environment.
  • Runners are servers (physical or virtual machines) that run workflows when triggered.
  • Events are specific activities in a repository that trigger a workflow run.
graph
  accTitle: Example workflow run diagram
  accDescr {
    An event runs two jobs - job 1 on runner 1 and job 2 on runner.
    Job 1 runs four steps, each either an action or script.
    Job 2 runs three other steps.
  }

  event("Event")

  event --> runner_1
  event --> runner_2

  subgraph runner_1["Runner 1"]
    job_1("Job 1
    • Step 1: Run action
    • Step 2: Run script
    • Step 3: Run script
    • Step 4: Run action
    ")
  end

  subgraph runner_2["Runner 2"]
    job_2("Job 2
    • Step 1: Run action
    • Step 2: Run script
    • Step 3: Run script
    ")
  end

Workflow descriptions and statuslink

Package testslink

These workflows build packages from source then run test suites using them.

graph LR
  accTitle: Package tests
  accDescr {
    Package tests start with a build_package step.
    After build_package, individual jobs are run in parallel for NVIDIA t4
    tests, AMD mi300 tests, etc.
  }

  build_packages --> test_nvidia_t4
  build_packages --> test_amd_mi300
  build_packages --> test_etc
  • Treat test workflows as code that downstream users would write - the packages used in workflows should be interchangeable with packages installed directly from PyPI or GitHub Releases.
  • Test workflows can build the IREE runtime from source (possibly cross-compiling) but they should use iree-compile and any other host tools from the built packages.
  • Test workflows can install other packages (e.g. tensorflow, torch) and fetch from model repositories like Hugging Face as needed to run test suites.
Workflow file Build status Event triggers
Package tests
pkgci.yml PkgCI pull_request, push

Platform buildslink

These workflows build the full project from source using standard options then run basic tests.

  • To keep these workflows focused, they should not need any special hardware (e.g. GPUs).
Workflow file Build status Event triggers
ci_linux_x64_clang.yml CI - Linux x64 clang pull_request, push
ci_linux_arm64_clang.yml CI - Linux arm64 clang schedule
ci_macos_x64_clang.yml CI - macOS x64 clang schedule
ci_windows_x64_msvc.yml CI - Windows x64 MSVC schedule

Other build configurationslink

These workflows build the full project from source using optional settings then run basic tests.

  • Workflows in this category can use sanitizers, debug builds, alternate compilers, and other features that maintainers want automated coverage for.
Workflow file Build status Event triggers
ci_linux_x64_clang_asan.yml CI - Linux x64 clang ASan pull_request, push
ci_linux_x64_clang_tsan.yml CI - Linux x64 clang TSan schedule
ci_linux_x64_clang_debug.yml CI - Linux x64 clang debug schedule
ci_linux_x64_gcc.yml CI - Linux x64 gcc schedule

Other workflowslink

Workflow file Build status Event triggers
ci.yml cI pull_request, push
build_package.yml Build Release Packages schedule
publish_website.yml publish_website push
samples.yml Samples schedule

Writing and editing workflowslink

Docker and dependencieslink

Workflow files typically require some external dependencies in the form of software packages, environment settings, and sometimes even system/hardware drivers. One way to manage these dependencies is to bundle them into a container using a tool like Docker.

Tip

We recommend only using Docker containers within workflow files in specific cicumstances and with moderation.

IREE contains a cross-compiler and minimal runtime, both of which are designed to run on a wide range of systems. Using carefully constructed containers for basic development risks the project only working within such containers.

These sorts of dependencies may be a good fit for using Docker containers:

  • Infrequently changing large dependencies like compiler toolchains.
  • Dependencies with complicated installs (e.g. building from source, moving files to specific paths).
  • System dependencies like GPU drivers.
  • Environment settings like disk partitions.

Here are alternative ways to fetch and configure workflow/job dependencies:

Workflow triggerslink

Of the events that trigger workflows, we most commonly use:

  • pull_request
    • Jobs most representative of core developer workflows should aim to run here. Jobs can be marked required or opt-in on presubmit on a case-by-case basis.
  • push
    • Jobs running here should be a superset of jobs running on pull_request.
  • schedule
    • Jobs designed to run nightly (e.g. nightly releases), jobs for non-core configurations (like certain sanitizers/fuzzers), and jobs using self-hosted runners in low supply can run on a schedule instead of on every commit.
  • workflow_dispatch
    • This trigger is mostly used for manual workflow debugging.
    • Where possible, jobs should allow this trigger so maintainers can test workflows without needing to send pull requests.

"Presubmit" and "postsubmit"

We use the terminology "presubmit" and "postsubmit" to differentiate between stages when checks run:

  • "Presubmit" checks run on code that has not yet been reviewed/approved/merged with either of the pull_request or workflow_dispatch triggers.
  • "Postsubmit" checks run on code that has been merged to a common branch like main with either of the push or schedule triggers.

In an ideal world every check would run on presubmit, but some operating system or hardware runners are in short supply and some workflows are slow even with sufficient resources (e.g. benchmark suites). We try to strike a balance between utility and economics.

Example workflow triggers

  • ci_linux_x64_clang_asan.yml runs on presubmit (pull_request trigger) and postsubmit (push trigger). Even though this workflow builds the compiler and needs to use large build machines because, it is generally useful for all C/C++ compiler and runtime changes.
  • ci_linux_x64_clang_tsan.yml is similar to the ASan build but it runs on the schedule event because it is only situationally useful and we want to limit use of large build machines. It would run on GitHub-hosted runners if they could handle it without running out of disk space.
  • ci_linux_arm64_clang.yml uses the schedule event since GitHub does not offer free Linux arm64 runners.

Required and optional checkslink

Any workflow that runs on the pull_request event can be either optional (the default) or required.

  • All required checks must be passing for a pull request to be merged.
  • Pull requests can be merged with optional checks pending or even failing.
  • The auto-merge feature will wait for required reviews to be met and required status checks to pass.

Note

Required checks must use only either standard GitHub-hosted runners or runners from the CPU builder pool.

Opt-in for presubmit jobslink

GitHub supports paths and paths-ignore filters for push and pull_request events that can be used to configure which workflows run based on paths modified. This mechanism is simple but somewhat limited in what it can express, so we have a custom mechanism for marking certain jobs as conditionally enabled:

  • Always run on push events, after pull requests are merged (postsubmit).

  • Jobs may be marked as opt-in for pull_request events (presubmit) by editing build_tools/github_actions/configure_ci.py. That script runs as part of the setup.yml action, which jobs can depend on like so:

    jobs:
  setup:
    uses: ./.github/workflows/setup.yml

  test:
    needs: [setup]
    if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test')
    steps:
      - ...

  • Opt-in jobs can also be set up to run automatically if specific file paths are modified, much like GitHub's paths and paths-ignore filters.

To bypass the computed configuration of workflows to run, see the CI behavior manipulation section of our contributing guide.

Using GitHub-hosted and self-hosted runnerslink

We group runners into categories:

  1. GitHub-hosted runners: standard (free)
    • Workflow jobs should use these GitHub-hosted runners when at all possible.
  2. GitHub-hosted runners: large (paid)
    • We aren't currently using these, but they have been useful where self-hosting is difficult.
  3. Self-hosted runners: CPU builders
    • The core project maintainers sponsor a pool of powerful CPU build machines used to build the core project and packages. To limit the load on this pool, we restrict jobs using these runners to only the most essential.
  4. Self-hosted runners: special hardware
    • Project contributors can offer their own hardware as self-hosted runners, allowing tests to be run on that hardware at whatever frequency they have the capacity to support.
    • Due to how difficult it can be to keep self-hosted runners operating reliably, and how access is limited to the group that maintains the hardware, any jobs using these self-hosted runners must be optional and easy to disable.
    • Self-hosted runners can either be ephemeral (one job per runner, compatible with autoscaling), or persistent. Persistent runners can retain local build and artifact caches to improve workflow time substantially.

Contributing self-hosted runners

Want to run tests on your own hardware as part of IREE's upstream CI? Get in touch with us on one of our communication channels and we'd be happy to discuss the options available.

Maintenance tipslink

Self-hosted runner maintenancelink

Configuration scripting for runners hosted on Google Cloud Platform (GCP) is stored at ./build_tools/github_actions/runner/.

Useful resourceslink

Official:

Community: