Mistral Small 24b

[phly95]

Is it possible to use this to get Mistral Small 24b running on an 8GB GPU like the RTX 3060Ti?

[dreaming-panda]

We plan to support the Mistral model series by next week. Thank you!

[dreaming-panda]

Thank you for your interest. We finally got this done. Check config/chat_config_mistral_8gb.json. I got 6-7 tokens per second on my computer within 8GB VRAM.

Looking for your feedback! @phly95