ggml-cpu: extend RVV repack GEMM and GEMV to other VLENs

[taimur-10x]

Summary

This PR extends existing repacking and GEMM/GEMV kernels for quantization types to other RVV VLENs 128 to 1024) and adds new kernels for Q5_K and MXFP4.

Key Changes

• Added repacking RVV GEMM and GEMV kernels for:

  • Q5_K
  • MXFP4

• Extended the following implementations to VLEN=128 to 1024.

  • Q4_0
  • Q8_0
  • Q2_K
  • Q4_K
  • IQ4_NL

Tile Sizes

VLEN	Tiling	LHS	RHS	OUT
128	4, 8, 1	4x1	8x1	4x8
256	4, 16, 1	4x1	16x1	4x16
512	4, 32, 1	4x1	32x1	4x32
1024	4, 64, 1	4x1	64x1	4x64

Testing

Kernels were functionally tested on QEMU for VLENs (128-bit to 1024-bit) for a range of input sizes.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256) using llama-bench.

Q5_K

Prompt Processing

Model	Prompt Size	Repack GEMM 4x16x1 (Tok/s)	Vec Dot (Tok/ s)
Tinyllama Q5_K 1.1B	32	25.62	6.94
Tinyllama Q5_K 1.1B	64	24.74	6.84
Tinyllama Q5_K 1.1B	128	24.71	6.53
Tinyllama Q5_K 1.1B	256	24.75	6.72
Tinyllama Q5_K 1.1B	512	23.71	6.99

Token Generation

Model	Tokens Generated	Repack GEMV 1x16x1 (Tok/s)	Vec Dot (Tok/s)
Tinyllama Q5_K 1.1B	10	6.82	5.98
Tinyllama Q5_K 1.1B	16	6.48	5.29
Tinyllama Q5_K 1.1B	32	6.78	5.75
Tinyllama Q5_K 1.1B	64	6.54	5.15
Tinyllama Q5_K 1.1B	100	6.79	5.22

MXFP4

Prompt Processing

Model	Prompt Size	Repack GEMM 4x16x1 (Tok/s)	Vec Dot (Tok/s)
Tinyllama MXFP4 1.1B	32	21.08	8.67
Tinyllama MXFP4 1.1B	64	20.13	8.91
Tinyllama MXFP4 1.1B	128	20.39	8.77
Tinyllama MXFP4 1.1B	256	19.04	8.60
Tinyllama MXFP4 1.1B	512	19.47	8.33

Token Generation

Model	Tokens Generated	Repack GEMV 1x16x1 (Tok/s)	Vec Dot (Tok/s)
Tinyllama MXFP4 1.1B	10	8.53	7.58
Tinyllama MXFP4 1.1B	16	8.25	7.66
Tinyllama MXFP4 1.1B	32	8.17	7.57
Tinyllama MXFP4 1.1B	64	8.12	7.06
Tinyllama MXFP4 1.1B	100	7.84	6.53

Future Work

Subsequent PRs plan to add these kernels for Q3_K and Q6_K