WIP: AVRO-4223 Gradle plugin for generating Java code

[frevib]

What is the purpose of the change

Gradle plugin to generate Java code from Avro files

Verifying this change

This change added tests and can be verified as follows:

cd to avro/lang/java/gradle-plugin

./gradlew test

Documentation

• Does this pull request introduce a new feature? (yes)
• If yes, how is the feature documented? (docs): https://github.com/frevib/avro/blob/AVRO-4223-gradle-plugin/lang/java/gradle-plugin/README.md

Release

https://plugins.gradle.org/plugin/eu.eventloopsoftware.avro-gradle-plugin

0.0.2 is released and fully works with AVSC files:

0.0.5

0.0.8

0.1.0 this release adds Protocol support.

0.1.1 Fix issue with Gradle multi project, where sources would not appear on the classpath

Installation instructions: https://github.com/frevib/avro/blob/AVRO-4223-gradle-plugin/lang/java/gradle-plugin/README.md#version

An official release will be done in the coming month

[frevib]

@raphaelauv did you test the latest release https://plugins.gradle.org/plugin/eu.eventloopsoftware.avro-gradle-plugin? No need to add compileJava.source(avroGenerateJavaClasses) any more.

[opwvhk]

This is yet another marked improvement. Thank you!

I have a few open comments, and we'll want to fix the build, but otherwise I'm happy with the result.

[martin-g]

I have created https://issues.apache.org/jira/browse/INFRA-27616 for the requirement from Gradle to prove the ownership of avro.apache.org DNS domain.

[martin-g]

Why this uses the current class' loader ?
loadLogicalTypesFactories() and doCompile() use the thread's class loader

[frevib]

I've used the same as in the Maven plugin:

avro/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/AbstractAvroMojo.java

Line 391 in 4e37673

Class<?> klass = Class.forName(velocityToolClassName);

and:

avro/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/AbstractAvroMojo.java

Line 462 in 4e37673

protected URLClassLoader createClassLoader() throws DependencyResolutionRequiredException, MalformedURLException {

[martin-g]

Same here - testExcludes

[frevib]

We're not making a distinction between main and test includes/excludes like in the Maven plugin. If you compile for the test classpath, the same value is used from includedSchemaFiles, excludedSchemaFiles, includedProtocolFiles and excludedProtocolFiles.

It's a bit of design choice, but I thought it's a bit unnecessary to have this distinction. Thoughts?

[martin-g]

We use deploy to deploy -SNAPSHOTs for the Maven artefacts.
AFAIK Gradle plugins repo does not allow -SNAPSHOTs

[frevib]

Correct, the Gradle plugin portal does not support snapshots. It works different than Sonatype, where you first push and later release via the Sonatype UI. With the Gradle publish plugin when you run ./gradlew publishPlugins you'll immediately publish the plugin to the portal.

You can however publish -SNAPSHOT to the local Maven repo with ./gradlew publishToMavenLocal

[RyanSkraba]

Hello! I haven't been following this, and I'm catching up, but I'm really grateful for the work you've already put into this! I am wholeheartedly for getting the gradle plugin moving for the next Avro release.

As a donation, are you willing to move the package into the org.apache.avro instead of `eu.eventloop namespace? We might need to recreate the ticket with INFRA.

Does anybody have concerns about integrating kotlin code into the Avro repo? I'm not against it but I'm not as familiar with Kotlin.

[ebAtUelzener]

This loop assumes that a single pass of all files is sufficient to parse all input files, however in the case of an unresolved schema exception multiple passes may be required.

The old plugin solves this inside its SchemaResolver by delaying the parsing of these unresolved schemas (see call on line 67) and then requeueing them after each successfully parsed schema (see the call on line 55).

This can easily happen for instance in the default value validation step for record fields when using an enum that has not been parsed yet due to how the filetree walk has ordered the schema files.

[ebAtUelzener]

As a POC i've tried using a custom task definition extending from the AbstractCompileTask that changes the loop a bit to allow multiple passes until nothing successfully parses anymore. This works for the mentioned issue but i am not sure whether the protocol parsing also needs logic like this.

val remainingWork: MutableList<File> = schemaFileTree.files.toMutableList()
val delayedWork: MutableList<File> = mutableListOf()
val suppressedExceptions: MutableList<Exception> = mutableListOf()
while (remainingWork.isNotEmpty()) {
  val nextWork = remainingWork.removeFirst()
  try {
    parser.parse(nextWork)

    if (delayedWork.isNotEmpty()) {
      delayedWork.forEach { remainingWork.addLast(it) }
      delayedWork.clear()
    }
  } catch (e: RuntimeException) {
    if (e.message?.contains("unresolved schema") == true) {
      delayedWork.add(nextWork)
      suppressedExceptions.add(e)
    } else {
      throw e
    }
  }
}
if (delayedWork.isNotEmpty()) {
  val e = GradleException("Unable to parse some schema files, see suppressed exceptions.")
  suppressedExceptions.forEach { e.addSuppressed(it) }
  throw e
}