nomad-onload is tooling to integrate Nomad and OpenOnload.

It provides a Nomad Device Plugin that exposes OpenOnload capabilites to Nomad via virtual devices. This enables kernel-bypass of the networking stack of any Docker-driver Nomad Job. In addition to TCP and UDP acceleration, facilities like epoll and pipes are brought to userspace as well.

When installed, this plugin will discover an Onload installation and make plugin-based devices available to Nomad Clients. The plugin publishes the following "device types"; which are available depend on whether the host OS has installed Onload and TCPDirect.

When a one of those device types, such as onload is specified in a Nomad Job's resource stanza, then the plugin will install Onload binaries and libraries and device files into the Task, and optionally LD_PRELOAD Onload. Onload performance tuning may be applied via its environment variables:

task {
  env {
    EF_TCP_SERVER_LOOPBACK = "2"
    EF_TCP_CLIENT_LOOPBACK = "4"
  } 
  resources {
    device "onload" {}
  }
}

You may also be interested in Neomantra's docker-onload tooling for Onload-enabled Docker images. Running high-performance kernel-bypass workloads is a vast topic; this High Performance Redis gist includes an introduction.

Binaries for multiple platforms are released on GitHub through GitHub Actions.

To install the Nomad Onload Device Plugin on your Nomad Client instance, copy the nomad-onload-device binary to the host's Nomad plugin_dir (e.g. /opt/nomad/data/plugins).

Then add a plugin config stanza to your Nomad configuration:

# onload.hcl
plugin "nomad-onload-device" {
  config {
    # Mount Onload into images
    mount_onload = true
  }
}

The following settings are available to configure the plugin behavior, per above:

You can run onload_stackdump inside the container, but you must remove LD_PRELOAD first:

$ docker exec -it -e LD_PRELOAD= redis-d3e926be-72c4-1940-953b-7e1bbb7a75dd  /usr/bin/onload_stackdump lots | head
============================================================
ci_netif_dump_to_logger: stack=6 name=
  cplane_pid=142738
  namespace=net:[4026531840]
  Onload 8.1.2.26 uid=999 pid=1 ns_flags=80
...

Building is performed with Taskfile, creating the following binaries:

• onload-probe (simple test tool)
• nomad-onload-device-plugin (the plugin)

$ task
task: [tidy] go mod tidy
task: [tidy] go mod tidy
task: [tidy] go mod tidy
task: [install-deps] go build -o ./bin/launcher github.com/hashicorp/nomad/plugins/shared/cmd/launcher
task: [build-onload-probe] go build -o ./bin/onload-probe cmd/onload-probe/*.go
task: [build-plugin] go build -o ./bin/nomad-device-plugin-onload cmd/nomad-device-onload/*.go

• Mount profiles
• PCIBusID
• HealthDesc
• Device Statistics
• Device Attributes
• Redis example
• XDP example

Thanks to the Nomad and Onload teams and the organizations that have supported their open collaboration!

Much of the code has been reviewed and adapted from:

• Nomad Skeleton Device Plugin
• Nvidia Device Plugin
• Kubernetes Onload

Copyright (c) 2024 Neomantra BV. Authored by Evan Wies.

Released under the MIT License, see LICENSE.txt.