Hadoop parquet reader (and parquet-CLI) fails on some files from parquet-testing

[ccleva]

Describe the bug, including details regarding any error messages, version, and platform.

Tested using v1.16.0 on openJDK 11 and 17.

1. nation.dict-malformed.parquet

> java -cp parquet-cli-1.16.0.jar:dependency/* org.apache.parquet.cli.Main cat nation.dict-malformed.parquet
Unknown error
java.lang.RuntimeException: Failed on record 0 in file nation.dict-malformed.parquet
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:89)
	at org.apache.parquet.cli.Main.run(Main.java:169)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:197)
Caused by: java.lang.RuntimeException: Failed while reading Parquet file: nation.dict-malformed.parquet
	at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:360)
	at org.apache.parquet.cli.BaseCommand$1$1.<init>(BaseCommand.java:337)
	at org.apache.parquet.cli.BaseCommand$1.iterator(BaseCommand.java:335)
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:76)
	... 3 more
Caused by: java.io.EOFException
	at org.apache.parquet.bytes.SingleBufferInputStream.sliceBuffers(SingleBufferInputStream.java:134)
	at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAsBytesInput(ParquetFileReader.java:2100)
	at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1990)
	at org.apache.parquet.hadoop.ParquetFileReader$Chunk.readAllPages(ParquetFileReader.java:1920)
	at org.apache.parquet.hadoop.ParquetFileReader.readChunkPages(ParquetFileReader.java:1454)
	at org.apache.parquet.hadoop.ParquetFileReader.internalReadRowGroup(ParquetFileReader.java:1188)
	at org.apache.parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:1135)
	at org.apache.parquet.hadoop.ParquetFileReader.readNextFilteredRowGroup(ParquetFileReader.java:1380)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:140)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:245)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:140)
	at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:356)
	... 6 more

This seems related to an issue with an older version of the java writer: apache/arrow#42298

It's been fixed in the C++/python version by apache/parquet-cpp#209 (file loads fine with pyArrow), but maybe not in the hadoop reader ?

Links to the fix and the test in current version:
https://github.com/apache/arrow/blob/64f2055ffb68e5077420f4253e76d78952438cab/cpp/src/parquet/file_reader.cc#L199
https://github.com/apache/arrow/blob/64f2055ffb68e5077420f4253e76d78952438cab/cpp/src/parquet/reader_test.cc#L977

Note that the file can be read by the old parquet-tools (tested with v1.10.1).

2. fixed_length_byte_array.parquet

pyArrow (and parquet-tools) also fails to read this one, so I think it's a wider problem. I'll open an issue on their repository, this is more to let you know.

> java -cp parquet-cli-1.16.0.jar:dependency/* org.apache.parquet.cli.Main cat fixed_length_byte_array.parquet
{"flba_field": [0, 0, 3, -24]}
[...]
{"flba_field": [0, 0, 3, -122]}
Unknown error
java.lang.RuntimeException: Failed on record 90 in file fixed_length_byte_array.parquet
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:89)
	at org.apache.parquet.cli.Main.run(Main.java:169)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.parquet.cli.Main.main(Main.java:197)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 92 in block 0 in file file:/home/ccleva/dev/tlabs-data/tablesaw-parquet/target/test/data/parquet-testing-master/data/fixed_length_byte_array.parquet
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:280)
	at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
	at org.apache.parquet.cli.BaseCommand$1$1.advance(BaseCommand.java:356)
	at org.apache.parquet.cli.BaseCommand$1$1.next(BaseCommand.java:350)
	at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:76)
	... 3 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can't read value in column [flba_field] required fixed_len_byte_array(4) flba_field at value 92 out of 1000, 92 out of 100 in currentPage. repetition level: 0, definition level: 0
	at org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:604)
	at org.apache.parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:30)
	at org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:477)
	at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
	at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:425)
	at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:249)
	... 7 more
Caused by: org.apache.parquet.io.ParquetDecodingException: could not read bytes at offset 364
	at org.apache.parquet.column.values.plain.FixedLenByteArrayPlainValuesReader.readBytes(FixedLenByteArrayPlainValuesReader.java:47)
	at org.apache.parquet.column.impl.ColumnReaderBase$2$6.read(ColumnReaderBase.java:411)
	at org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:579)
	... 12 more
Caused by: java.io.EOFException
	at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
	at org.apache.parquet.column.values.plain.FixedLenByteArrayPlainValuesReader.readBytes(FixedLenByteArrayPlainValuesReader.java:45)
	... 14 more

I recreated the file using the script in apache/parquet-testing#31 in case it was corrupted but got the same result (at a different offset).

Component(s)

No response

[zhaoyudi-creator]

Can you confirm whether this issue occurs during writing or reading? I encountered the same situation when using Hudi. When writing to a Hudi table using Spark, no exceptions occurred, but the same error appeared during read operations. The exception is as follows:

[zhaoyudi-creator]

org.apache.hudi.exception.HoodieException: unable to read next record from parquet file
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:53)
at org.apache.hudi.common.util.MappingIterator.hasNext(MappingIterator.java:35)
at org.apache.hudi.common.util.queue.SimpleExecutor.execute(SimpleExecutor.java:65)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:154)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdateInternal(HoodieSparkCopyOnWriteTable.java:230)
at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.handleUpdate(HoodieSparkCopyOnWriteTable.java:221)
at org.apache.hudi.table.action.compact.CompactionExecutionHelper.writeFileAndGetWriteStats(CompactionExecutionHelper.java:64)
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:248)
at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compactInParallel$6d347a86$1(HoodieCompactor.java:155)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:223)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1556)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1466)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1530)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1357)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:335)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
at org.apache.spark.scheduler.Task.run(Task.scala:140)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:562)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1555)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:565)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1771814 in block 4 in file 00000002-628c-4456-a893-f9ea6b31122b-0_11164-0-11164_20251221141817486.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:264)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:48)
... 32 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can't read value in column [created_tm] optional binary created_tm (STRING) at value 267595 out of 716627, 7595 out of 20000 in currentPage. repetition level: 0, definition level: 1
at org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:553)
at org.apache.parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:30)
at org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:439)
at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:234)
... 34 more
Caused by: org.apache.parquet.io.ParquetDecodingException: could not read bytes at offset 174666
at org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:42)
at org.apache.parquet.column.impl.ColumnReaderBase$2$6.read(ColumnReaderBase.java:372)
at org.apache.parquet.column.impl.ColumnReaderBase.readValue(ColumnReaderBase.java:533)
... 39 more
Caused by: java.io.EOFException
at org.apache.parquet.bytes.SingleBufferInputStream.slice(SingleBufferInputStream.java:116)
at org.apache.parquet.column.values.plain.BinaryPlainValuesReader.readBytes(BinaryPlainValuesReader.java:40)
... 41 more

[ccleva]

I confirm the issue occurs during reading the file. No issue when writing it. This looks very similar.