Skip to content

[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018

Open
slfan1989 wants to merge 2 commits intoapache:masterfrom
slfan1989:auron-2017
Open

[AURON #2017] [BUILD] Add Spark 4.x support to dev/reformat script.#2018
slfan1989 wants to merge 2 commits intoapache:masterfrom
slfan1989:auron-2017

Conversation

@slfan1989
Copy link
Contributor

Which issue does this PR close?

Closes #2017

Rationale for this change

With Spark 4.0 and 4.1 support added to the project, the dev/reformat script needs to be updated to handle formatting and style checks for these new versions. Spark 4.x requires JDK 17+ and Scala 2.13, while Spark 3.x uses JDK 8 and Scala 2.12. The script should automatically switch between these environments.

What changes are included in this PR?

1. Fix Flink Maven profile

  • Before: -Pflink,flink-1.18
  • After: -Pflink-1.18
  • Reason: Avoid activating non-existent flink profile

2.Add Spark 4.x support

  • Add spark-4.0 and spark-4.1 to the version sweep list
  • Auto-switch to scala-2.13 profile for Spark 4.x (Spark 4.x requires Scala 2.13)
  • Auto-switch to JDK 17 for Spark 4.x (Spark 4.x requires JDK 17+)
  • Auto-switch back to JDK 8 for Spark 3.x versions

3.Update CI workflow (.github/workflows/style.yml)

  • Add JDK 17 setup alongside existing JDK 8
  • Enable style check to work with both Spark 3.x and Spark 4.x versions

Are there any user-facing changes?

No.

How was this patch tested?

Verified automatic JDK switching works for Spark 3.x (JDK 8) and Spark 4.x (JDK 17)

…ipt.

Signed-off-by: slfan1989 <slfan1989@apache.org>
…ipt.

Signed-off-by: slfan1989 <slfan1989@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUILD] Add Spark 4.x support to dev/reformat script

1 participant

Comments