feat: implementation of multimodal runner#892

Open

NorbertKlockiewicz wants to merge 46 commits intomainfrom

Commits on Mar 2, 2026

feat: initial implementation of multimodal runner with lfm vlm
NorbertKlockiewicz
committed
feat: unified LLM runner for text-only and multimodal PTEs

NorbertKlockiewicz
and
claude
committed
feat: add conversational VLM demo with multimodal/text-only support and fix token generation bugs

NorbertKlockiewicz
and
claude
committed
fix: default UnifiedRunner temperature to 0.8 and topp to 0.9
NorbertKlockiewicz
committed
feat: add NativeMessage struct and JSI conversion for message history
NorbertKlockiewicz
committed
feat: declare generateMultimodal on LLM and register JSI binding
NorbertKlockiewicz
committed
fix: remove redundant unordered_map and vector includes from LLM.h
NorbertKlockiewicz
committed
feat: implement generateMultimodal with per-turn chat template and image cache

NorbertKlockiewicz
and
claude
committed
feat: add mediaPath to Message, remove sendMessageWithImage from LLMType
NorbertKlockiewicz
committed
feat: replace sendMessageWithImage with sendMessage(msg, mediaPath?) using generateMultimodal

NorbertKlockiewicz
and
claude
committed
fix: use updatedHistory for multimodal routing, remove redundant reset before generateMultimodal
NorbertKlockiewicz
committed
fix: skip system messages in generateMultimodal, clear imageUri after send
NorbertKlockiewicz
committed
feat: show image thumbnail in user message bubble when mediaPath is set
NorbertKlockiewicz
committed
fix: use resizeMode contain so full image is always visible in message bubble
NorbertKlockiewicz
committed
refactor: derive isMultimodal from load param, unify load branches, remove tokenizerConfig guard

NorbertKlockiewicz
and
claude
committed
refactor: remove isMultimodal flag, inline generateMultimodal into sendMessage

NorbertKlockiewicz
and
claude
committed
fix: make tokenizerConfigSource required throughout load pipeline

NorbertKlockiewicz
and
claude
committed
fix: prepend system prompt to multimodal history before generateMultimodal

NorbertKlockiewicz
and
claude
committed
refactor: unify generate — Jinja renders prompt+<image> tokens in JS, C++ splits on placeholder

NorbertKlockiewicz
and
claude
committed
fix: collect imagePaths from messageHistoryWithPrompt, not full history

NorbertKlockiewicz
and
claude
committed
fix: typing
NorbertKlockiewicz
committed
feat: correctly calculate image tokens
NorbertKlockiewicz
committed
fix: add missing import
NorbertKlockiewicz
committed
fix: fall back to max_seq_len when model doesn't export max_context_len
NorbertKlockiewicz
committed
fix: address code review — error on image/placeholder mismatch, remove double reset, fix max_context_len fallback, require tokenizerConfigSource, pass tools in multimodal branch, capture callback b…
NorbertKlockiewicz
committed
feat: dynamic sendMessage type based on flag
NorbertKlockiewicz
committed

Commits on Mar 3, 2026

Commits on Mar 5, 2026