100B LLM model supported?

[Yiming-xie-2021]

An article posted on wechat social media announced this framework supports 100B LLM, and outputs tokens at relatively high speed, is this true? i didn't find 100B llm in your frontpage list. thanks.

[adxdits]

Hello, the 100B in Llama3-8B-1.58-100B-tokens refers to training tokens, not model parameters, it's actually an 8B-parameter model and I think no real 100B-parameter model is currently available, though the framework claims architectural support for that scale.

[PawelNPL]

The biggest real Bitnet model is Llama3-8B-1.58-100B-tokens.
But it was trained on very small amount of data, so for now the best Bitnet model is microsoft/bitnet-b1.58-2B-4T.
And other(better) bitnet models are unlikely to appear anytime soon. Training a Bitnet is more difficult than standard LLMs.

Bitnet will shine at 70B+ parameters.... in the future.