[Devices] Offer support for hardware-accelerated inference in Firecracker

[raduweiss]

Doing hardware-accelerated inference in a serverless environment is compelling use case.

However, adding straight up GPU passthrough means that microVM can't oversubscribe memory, and we need to add PCI emulation to Firecracker, which comes with a lot of extra complexity/attack surface.

The first step here will be to research the options and alternatives (e.g., GPU passthrough, or something else), and figure out the path forward.

Related issues: #849, #776.

[nlauchande]

I am very interested on this usecase.

[richardliaw]

+1, very interested in this use case. Any update on this? (I understand it's still in the research phase)

[zaharidichev]

@raduweiss is this something that anyone is working on atm? Is it still on the roadmap?

[ananos]

Hi @zaharidichev,

we have some thoughts on this [1], shared them earlier this year in the slack wοrkspace [2], but a chat is still pending I'm afraid. We have a rough proof-of-concept implementation on firecracker, based on the design principles of [1], which exhibits negligible overhead for image inference (jetson-inference backend, using tensorRT, tested on an NVIDIA jetson nano & a generic x86_64 machine with an RTX 2060 SUPER & another machine with a T4). We should be able to open-source the whole stack pretty soon. Feel free to drop us a line if you're interested in our early PoC.

Essentially, the idea is that we abstract away the hardware-specific operations via a slim runtime library/system, that supports any kind of backend (ranging from a simple CUDA/OpenCL function to a TensorFlow operation/app). Combined with a simple virtio frontend/backend implementation we are able to forward operations from a guest to the host/monitor, which in turn executes the actual "acceleratable" function on the hardware accelerator.

Another option (if latency is not critical to you) could be to use rCUDA, which we plan to try but haven't had the time yet...

BTW, @raduweiss we should plan to have that chat [2] at some point -- give us a shout when you are available!

cheers,
Tassos

[1] https://blog.cloudkernels.net/posts/vaccel/
[2] https://firecracker-microvm.slack.com/archives/CDL3FUR8B/p1591093992140800

[raduweiss]

@ananos , yeah our bad, we totally dropped the ball here. Our apologies! I'll reply directly so we can talk.

[ananos]

Hi @zaharidichev, all

just wanted to share our blog post about our approach on the above: https://blog.cloudkernels.net/posts/vaccel_v2/

using nvidia-container-runtime & a docker image we've put together, you are able to run the jetson-inference image classification example from a Firecracker VM. You can find more info in the above post or @ https://vaccel.org. Of course, you can ping us, we will be more than happy to share how to try out vAccel on Firecracker.

cheers,
Tassos

[amrragab8080]

Any update on the GPU support in Firecracker?