Spark: Add support for 4.2.0-preview

[manuzhang]

No description provided.

[manuzhang]

This failure from testing Spark 4.2.0-preview2 is caused by apache/spark#53788, after which an AnalysisException would be thrown on Iceberg metadata tables like default.table.partitions.

TestAddFilesProcedure > addPartitionsWithNullValueShouldAddFilesToNullPartition() > catalogName = spark_catalog, implementation = org.apache.iceberg.spark.SparkSessionCatalog, config = {type=hive, default-namespace=default, parquet-enabled=true, cache-enabled=false}, formatVersion = 2 FAILED
    org.apache.spark.sql.AnalysisException: [REQUIRES_SINGLE_PART_NAMESPACE] spark_catalog requires a single-part namespace, but got `default`.`table`. SQLSTATE: 42K05

[manuzhang]

Failed tests after upgrading to Spark 4.2.0-preview3-rc1

1. testJoinsHourToDays() in TestStoragePartitionedJoins.java:

Assertion failed: "SPJ should not change query output: number of results should match"
The actual and expected query result sizes differ, indicating that either the join logic or test data setup causes a mismatch.

2. readFromViewReferencingTempFunction() in TestViews.java:

Assertion failed: Expected a specific routine not found error, but got an AnalysisException with different message details.

[manuzhang]

apache/spark#54884 has been opened to fix the first failure.

[manuzhang]

I will update HourToDaysReducer following interface changes from apache/spark#54884 in next preview release. All other test failures have been fixed.

[szehon-ho]

leave a note here to implement the new method in the Reducer once apache/spark#54884 is in (next Spark 4.2 preview)

Sorry just saw, it is the same comment