Add VLM support

[mkopcins]

Explore options to add vlm support inside runner for llms. There is a demo app in ET docs

[mergennachin]

Just fyi, we've recently added MultimodalRunner API in https://github.com/pytorch/executorch/tree/main/extension/llm/runner

And moving the Llava apps to MultimodalRunner directly https://github.com/pytorch/executorch/blob/main/examples/models/llava/main.cpp

[wassgha]

Any progress here, @msluszniak ? Need help with anything?

[msluszniak]

Hi @wassgha, the team is currently on a break and will be back on Jan 7th. As far as I know, VLMs are still in the backlog and will be picked up soon. I’d advise waiting until someone from the SWM AI team reaches out to you with more information. That said, any contribution is very welcome! :)

[msluszniak]

cc: @chmjkb @IgorSwat

[chmjkb]

Hey @wassgha, we will be assessing existing Executorch-compatible VLMs soon to decide on timing for this feature and will follow up once we have more clarity.

If you’d like to contribute in the meantime, checking which VLMs are exportable to Executorch and how they perform would be very helpful.

One more question while we’re here: what would be the best way for you to pass video data to our API? That feedback would help us a lot too 🙂