Deploying VITA-1.5 Multimodal Model with ExecuTorch

[jordanqi]

🚀 The feature, motivation and pitch

I’m trying to deploy a VITA-1.5 multimodal model (supports audio, vision, and text) using ExecuTorch.

The tokenizer is in Hugging Face tokenizer.json format, and I’d like to ask:

1. Is there any suggested way to convert the model into .pte format for ExecuTorch?
2. Since this is a new architecture, is there any guidance or examples for adding custom models?
3. Can I still use the LlamaDemo Android app with this multimodal？

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

[Jack-Khuu]

cc: @larryliu0820 for MM
@kirklandsign for Android

[Jack-Khuu]

Hi @jordanqi, if you haven't joined our discord channel, we would love to have you on there :)

[jordanqi]

Hi @jordanqi, if you haven't joined our discord channel, we would love to have you on there :)

I haven't joined the discord, please send the channel link to me thanks

[kirklandsign]

https://discordapp.com/channels/1334270993966825602/1334274225586049044

[kirklandsign]

Is there any suggested way to convert the model into .pte format for ExecuTorch?
Since this is a new architecture, is there any guidance or examples for adding custom models?

Is this model from HF? @guangy10 maybe know

Can I still use the LlamaDemo Android app with this multimodal？

If it can run with desktop llama_runner out of box then it can run with LlamaDemo Android app, but not sure about image processing and prompt format part

[kirklandsign]

We support Hugging Face tokenizer.json format right now

[larryliu0820]

Hi @jordanqi thanks for your interest. A lot of the features you are asking for are under development, so there might not be convenient ways to do them, but I believe with a bit of work you can make these working!

Is there any suggested way to convert the model into .pte format for ExecuTorch?

Yes I would suggest you to take a look at the Llava example for multimodal LLMs: https://github.com/pytorch/executorch/blob/main/examples/models/llava/export_llava.py

We are actively working on a more general API that works on models with the same architecture

Since this is a new architecture, is there any guidance or examples for adding custom models?

Again please refer to the Llava example. We are open to take a new model under example directory, as long as it works.

Can I still use the LlamaDemo Android app with this multimodal？

Most likely you can use the demo app for image prefill, but to support audio we need a lot of work. Also keep in mind that right now the model and the runner are coupled so it's really a case by case situation.