[lake][iceberg] Iceberg does not support union read for primary key table

[Yohahaha]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.9.0 (latest release)

Please describe the bug 🐞

I find Iceberg does not support union read primary key table, IcebergLakeSource#createRecordReader returns IcebergRecordReader which not a subclass of SortedRecordReader, so LakeSnapshotAndLogSplitScanner will fail when pollBatch.

I check testReadIcebergLakeTable in FlinkUnionReadPrimaryKeyTableITCase, seems it just read records from $lake, but not the union read.

Is there something I'm misunderstanding?

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[MehulBatra]

Hi @Yohahaha It's a known feature gap due to current acrchietecture if we enable union read on pk table it will end up heavy merge-sort join, hence we are working on better design implementation via Deletion vectors to speed up union reads.
cc: @wuchong @luoyuxia