GitHub Actionslink
Overviewlink
We use GitHub Actions for continuous automation (CI) and continuous delivery (CD) workflows:
- Code formating and linting.
- Building from source and running tests.
- Building packages for testing and releases.
- Testing packages across a variety of platforms.
- Publishing the https://iree.dev website.
- Updating dependencies.
Workflows are defined directly in the repository at
.github/workflows/
.
We use a mix of GitHub-hosted runners and self-hosted runners to get automated
build and test coverage across a variety of platforms and hardware accelerators.
Terminology primerlink
(Read more on https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions)
- Workflows are configurable automated processes that run one or more jobs.
- Jobs are a set of steps in a workflow that are executed on the same runner.
- Steps are lists of commands or meta actions to run in a shell environment.
- Runners are servers (physical or virtual machines) that run workflows when triggered.
- Events are specific activities in a repository that trigger a workflow run.
graph
accTitle: Example workflow run diagram
accDescr {
An event runs two jobs - job 1 on runner 1 and job 2 on runner.
Job 1 runs four steps, each either an action or script.
Job 2 runs three other steps.
}
event("Event")
event --> runner_1
event --> runner_2
subgraph runner_1["Runner 1"]
job_1("Job 1
• Step 1: Run action
• Step 2: Run script
• Step 3: Run script
• Step 4: Run action
")
end
subgraph runner_2["Runner 2"]
job_2("Job 2
• Step 1: Run action
• Step 2: Run script
• Step 3: Run script
")
end
Workflow descriptions and statuslink
Package testslink
These workflows build packages from source then run test suites using them.
graph LR
accTitle: Package tests
accDescr {
Package tests start with a build_package step.
After build_package, individual jobs are run in parallel for NVIDIA t4
tests, AMD mi300 tests, etc.
}
build_packages --> test_nvidia_t4
build_packages --> test_amd_mi300
build_packages --> test_etc
- Treat test workflows as code that downstream users would write - the packages used in workflows should be interchangeable with packages installed directly from PyPI or GitHub Releases.
- Test workflows can build the IREE runtime from source (possibly
cross-compiling) but they should use
iree-compile
and any other host tools from the built packages. - Test workflows can install other packages (e.g.
tensorflow
,torch
) and fetch from model repositories like Hugging Face as needed to run test suites.
Workflow file | Build status | Event triggers |
---|---|---|
Package tests | ||
pkgci.yml |
pull_request , push |
Platform buildslink
These workflows build the full project from source using standard options then run basic tests.
- To keep these workflows focused, they should not need any special hardware (e.g. GPUs).
Workflow file | Build status | Event triggers |
---|---|---|
ci_linux_x64_clang.yml |
pull_request , push |
|
ci_linux_arm64_clang.yml |
schedule |
|
ci_macos_x64_clang.yml |
schedule |
|
ci_windows_x64_msvc.yml |
schedule |
Other build configurationslink
These workflows build the full project from source using optional settings then run basic tests.
- Workflows in this category can use sanitizers, debug builds, alternate compilers, and other features that maintainers want automated coverage for.
Workflow file | Build status | Event triggers |
---|---|---|
ci_linux_x64_clang_asan.yml |
pull_request , push |
|
ci_linux_x64_clang_tsan.yml |
schedule |
|
ci_linux_x64_clang_debug.yml |
schedule |
|
ci_linux_x64_gcc.yml |
schedule |
|
ci_linux_x64_clang_byollvm.yml |
schedule |
|
ci_linux_x64_bazel.yml |
pull_request , push |
Other workflowslink
Workflow file | Build status | Event triggers |
---|---|---|
ci.yml |
pull_request , push |
|
build_package.yml |
schedule |
|
publish_website.yml |
push |
|
samples.yml |
schedule |
Writing and editing workflowslink
Docker and dependencieslink
Workflow files typically require some external dependencies in the form of software packages, environment settings, and sometimes even system/hardware drivers. One way to manage these dependencies is to bundle them into a container using a tool like Docker.
Tip
We recommend only using Docker containers within workflow files in specific cicumstances and with moderation.
IREE contains a cross-compiler and minimal runtime, both of which are designed to run on a wide range of systems. Using carefully constructed containers for basic development risks the project only working within such containers.
These sorts of dependencies may be a good fit for using Docker containers:
- Infrequently changing large dependencies like compiler toolchains.
- Dependencies with complicated installs (e.g. building from source, moving files to specific paths).
- System dependencies like GPU drivers.
- Environment settings like disk partitions.
Here are alternative ways to fetch and configure workflow/job dependencies:
- Install from a package manager like
pip
orapt
. - Use an action like
actions/setup-python to install
packages and add them to
PATH
. - Use GitHub-hosted runners and their installed software.
Workflow triggerslink
Of the events that trigger workflows, we most commonly use:
pull_request
- Jobs most representative of core developer workflows should aim to run here. Jobs can be marked required or opt-in on presubmit on a case-by-case basis.
push
- Jobs running here should be a superset of jobs running on
pull_request
.
- Jobs running here should be a superset of jobs running on
schedule
- Jobs designed to run nightly (e.g. nightly releases), jobs for non-core configurations (like certain sanitizers/fuzzers), and jobs using self-hosted runners in low supply can run on a schedule instead of on every commit.
workflow_dispatch
- This trigger is mostly used for manual workflow debugging.
- Where possible, jobs should allow this trigger so maintainers can test workflows without needing to send pull requests.
"Presubmit" and "postsubmit"
We use the terminology "presubmit" and "postsubmit" to differentiate between stages when checks run:
- "Presubmit" checks run on code that has not yet been
reviewed/approved/merged with either of the
pull_request
orworkflow_dispatch
triggers. - "Postsubmit" checks run on code that has been merged to a common branch
like
main
with either of thepush
orschedule
triggers.
In an ideal world every check would run on presubmit, but some operating system or hardware runners are in short supply and some workflows are slow even with sufficient resources (e.g. benchmark suites). We try to strike a balance between utility and economics.
Example workflow triggers
ci_linux_x64_clang_asan.yml
runs on presubmit (pull_request
trigger) and postsubmit (push
trigger). Even though this workflow builds the compiler and needs to use large build machines because, it is generally useful for all C/C++ compiler and runtime changes.ci_linux_x64_clang_tsan.yml
is similar to the ASan build but it runs on theschedule
event because it is only situationally useful and we want to limit use of large build machines. It would run on GitHub-hosted runners if they could handle it without running out of disk space.ci_linux_arm64_clang.yml
uses theschedule
event since GitHub does not offer free Linux arm64 runners.
Required and optional checkslink
Any workflow that runs on the pull_request
event can be either optional
(the default) or
required.
- All required checks must be passing for a pull request to be merged.
- Pull requests can be merged with optional checks pending or even failing.
- The auto-merge feature will wait for required reviews to be met and required status checks to pass.
Note
Required checks must use only either standard GitHub-hosted runners or runners from the CPU builder pool.
Opt-in for presubmit jobslink
GitHub supports
paths
and paths-ignore
filters
for push
and pull_request
events that can be used to configure which
workflows run based on paths modified. This mechanism is simple but somewhat
limited in what it can express, so we have a custom mechanism for marking
certain jobs as conditionally enabled:
- Always run on
push
events, after pull requests are merged (postsubmit). -
Jobs may be marked as opt-in for
pull_request
events (presubmit) by editingbuild_tools/github_actions/configure_ci.py
. That script runs as part of thesetup.yml
action, which jobs can depend on like so:jobs: setup: uses: ./.github/workflows/setup.yml test: needs: [setup] if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test') steps: - ...
-
Opt-in jobs can also be set up to run automatically if specific file paths are modified, much like GitHub's
paths
andpaths-ignore
filters.
To bypass the computed configuration of workflows to run, see the CI behavior manipulation section of our contributing guide.
Using GitHub-hosted and self-hosted runnerslink
We group runners into categories:
- GitHub-hosted runners: standard (free)
- Workflow jobs should use these GitHub-hosted runners when at all possible.
- GitHub-hosted runners: large (paid)
- We aren't currently using these, but they have been useful where self-hosting is difficult.
- Self-hosted runners: CPU builders
- The core project maintainers sponsor a pool of powerful CPU build machines used to build the core project and packages. To limit the load on this pool, we restrict jobs using these runners to only the most essential.
- Self-hosted runners: special hardware
- Project contributors can offer their own hardware as self-hosted runners, allowing tests to be run on that hardware at whatever frequency they have the capacity to support.
- Due to how difficult it can be to keep self-hosted runners operating reliably, and how access is limited to the group that maintains the hardware, any jobs using these self-hosted runners must be optional and easy to disable.
- Self-hosted runners can either be ephemeral (one job per runner, compatible with autoscaling), or persistent. Persistent runners can retain local build and artifact caches to improve workflow time substantially.
Contributing self-hosted runners
Want to run tests on your own hardware as part of IREE's upstream CI? Get in touch with us on one of our communication channels and we'd be happy to discuss the options available.
Maintenance tipslink
- Certain workflow failures are posted in the
#github-ci
channel in IREE's Discord server. - GitHub supports notifications for workflow runs but only for the user that last modified a workflow.
- GitHub sometimes experiences incidents outside of our control. Monitor https://www.githubstatus.com/ to see if issues are widespread.
- Repository admins can monitor repository runners at https://github.com/iree-org/iree/settings/actions/runners.
- Organization admins can monitor organization runners at https://github.com/organizations/iree-org/settings/actions/runners.
Useful resourceslink
Official:
- Guides for GitHub Actions: https://docs.github.com/en/actions/guides.
- Events that trigger workflows: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows.
- About GitHub-hosted runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners.
- About large runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners.
- GitHub Actions Runner application: https://github.com/actions/runner.
- GitHub Actions Runner Images with included software: https://github.com/actions/runner-images.
Community:
- A curated list of awesome things related to GitHub Actions: https://github.com/sdras/awesome-actions.