(Performance) Optimized x86 and generic q1_0(_g128) dot#10
Open
pl752 wants to merge 7 commits intoPrismML-Eng:prismfrom
Open
(Performance) Optimized x86 and generic q1_0(_g128) dot#10pl752 wants to merge 7 commits intoPrismML-Eng:prismfrom
pl752 wants to merge 7 commits intoPrismML-Eng:prismfrom
Conversation
The Q1_0_g128 vec_dot kernel had a bug where `sumi` was declared as `int` but accumulated `float` partial products (`d1 * sumi_block`), causing float-to-int truncation that destroyed dot product results and produced gibberish output on CPU. Additionally, the x86 kernel was purely scalar (one bit at a time). This adds an AVX-512BW path that processes 32 elements per iteration using mask_sub + madd + fma, with a single horizontal reduction at the end. Benchmarks (Bonsai-8B, CPU-only, AVX-512): Before: 0.73 t/s prompt, 0.65 t/s generation (gibberish output) After: 23.2 t/s prompt, 13.5 t/s generation (coherent output) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello
This is yet another PR about the
fix of the truncation andoptimization of the cpu inference.In this case I have:
Note that this PR is built on top of the #3 by @jordankzf, who implemented AVX-512 workflow
Benchmarks were performed with:
Bonsai-1.7B.gguf(Q1_0_g128)6pp 512t/stg 128t/sSSSE3AVXAVX2+FMAAVX512BW*extrapolated frompp 32/tg 16:1.659 t/spp and0.862 t/stg, as I was impatient.**new SIMD instruction kinds improve performance even on AMD Zen4 implementation of AVX-512, which uses 256 bit pipeline twice instead of implementing full 512 bit oneI would appreciate your feedback