[Parquet] Add SIMD-accelerated byte-stream-split decoding#654
[Parquet] Add SIMD-accelerated byte-stream-split decoding#654daniel-adam-tfs wants to merge 2 commits intoapache:mainfrom
Conversation
- Move all byte-stream-split decoding routines to new file byte_stream_split_decode.go. - Add architecture-specific SIMD implementations: - AVX2 for amd64 (byte_stream_split_decode_avx2_amd64.go/.s) - NEON for arm64 (byte_stream_split_decode_neon_arm64.go/.s) - Add runtime dispatch for SIMD decoding based on CPU features (AVX2/NEON). - Ensure correct build tags and file suffixes for cross-platform compatibility.
|
Benchmarks comparison on my Intel macbook: |
|
@zeroshade One of our departments has integrated byte-stream-split encoding/decoding into the currently used proprietary format that is used to store data. We did some comparisons and they were getting faster decoding so looked into their code and they were using SIMD implementation with I've actually wrote a C code first and tried c2goasm with AppleClang and clang21.0 and I couldn't get that to generate me a compilable code. Anyway, most of our data is float32s or float64s, so the bss decoding function was at the top. After trying processing some files with this change it fell to like to 10th place, |
|
This is awesome, thanks! I agree that if the new fallback is faster then we should replace the current with it. I'm still traveling currently but I'll take a look later this week. The lint is failing because you need the apache license header on the files. |
|
@daniel-adam-tfs any further work needed here to make it ready for review? |
2 participants
Rationale for this change
The byte-stream-split encoding is commonly used in Parquet for floating-point data, as it improves compression ratios by grouping similar bytes together. However, the existing Go implementation uses a simple scalar loop which is inefficient for large datasets. By leveraging SIMD instructions (AVX2 on x86 and NEON on ARM), we can significantly accelerate the decoding process and improve overall Parquet read performance.
What changes are included in this PR?
Optimized implementation of byte-stream split decoding algorithm.
Added SIMD-accelerated implementations:
AVX2 implementation for amd64 architecture using 256-bit vectors processing 32 values per block
NEON implementation for arm64 architecture using 128-bit vectors processing 16 values per block
Both use 2-stage byte unpacking hierarchy following the same algorithm structure
Implemented runtime CPU feature detection with automatic dispatch to the best available implementation (SIMD vs scalar fallback)
Added proper build tags and file suffixes for cross-platform compatibility
Included an optimized V2 scalar implementation using unsafe pointer casting as a fallback
Are these changes tested?
Yes. Various tests were added:
Are there any user-facing changes?
No user-facing API changes. This is a performance optimization that maintains full backward compatibility. Users will automatically benefit from faster Parquet decoding when reading files with byte-stream-split encoded floating-point columns, with no code changes required.