Pin testLatestDeps versions for reproducible builds by zeitlinger · Pull Request #16344 · open-telemetry/opentelemetry-java-instrumentation

zeitlinger · 2026-03-02T14:35:56Z

Summary

Add version pinning for testLatestDeps builds so that the same commit always resolves the same dependency versions
A JSON file (.github/config/latest-dep-versions.json) stores resolved versions, used by the convention plugin and settings to replace dynamic versions (latest.release, + ranges)
Add a resolveLatestDepVersions Gradle task and a daily CI workflow to keep the pinned versions up to date

Details

When -PtestLatestDeps=true, dependency versions were resolved dynamically at build time via latest.release and + ranges. This made builds non-reproducible — the same commit could produce different results depending on when it was built.

Three sources of dynamic versions are now pinned:

Convention plugin (library/testLibrary configs) — whenObjectAdded hooks use pinned version instead of latest.release
latestDepTestLibrary configs — whenObjectAdded hooks use pinned version instead of declared range
Inline latest.release / + range strings in ~70 individual build files — caught by resolutionStrategy.eachDependency in the convention plugin (no individual build file changes needed)
Spring Boot catalogs in settings.gradle.kts — pinned via JSON lookup

To bypass pinning and resolve against live latest versions (e.g. for debugging):

./gradlew test -PtestLatestDeps=true -PresolveLatestDeps=true

To update the pinned versions locally:

./gradlew resolveLatestDepVersions -PtestLatestDeps=true -PresolveLatestDeps=true

If a dependency is not present in the JSON file, it falls back to the original dynamic version.

Why a GitHub Action instead of Renovate?

Renovate is the natural tool for dependency updates, but this file is a poor fit:

Same artifact, multiple version ranges — entries like java-client, java-client#2.+, java-client#2.5.+ coexist. Renovate identifies deps by name, so it can't handle the same Maven coordinate with different range constraints without extensive packageRules.
Dynamic version constraints — the #2.+, #4.3.+ suffixes encode allowed version ranges. Renovate can't derive allowedVersions from a regex capture; you'd need explicit packageRules for each of the dozens of unique range patterns.
Custom pre-release filtering — the Gradle task filters -alpha, -beta, -rc, .m (Lettuce milestones), -nf-execution (GraphQL), git SHAs, and datetime versions. Renovate's stability filtering won't match these project-specific quirks.
Gradle resolution semantics — the task resolves versions through Gradle's actual dependency resolution (respecting BOMs, platforms, transitive constraints). Renovate queries Maven Central independently, which could diverge for BOM-managed dependencies.

The GitHub Action runs Gradle's own resolution, so the pinned versions exactly match what testLatestDeps builds see.

Test plan

Ran resolveLatestDepVersions — resolved 475 versions
Verified pinning via dependency report (okhttp resolved to pinned 5.3.2)
Ran okhttp instrumentation tests with -PtestLatestDeps=true — all passed
CI passes

trask · 2026-03-05T16:25:34Z

btw, nice idea 👍

- Fail-fast when a range-specific pinned version key (e.g. "com.example:lib#2.+") is missing from the JSON instead of silently falling back to the base key which could resolve to a wrong major version. - Make test-latest-deps a required status check now that versions are pinned. Added TODO for branch protection settings (requires admin). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger · 2026-03-06T09:16:37Z

Merged your fork PR #9 (shared VersionFilters), addressed both review comments:

Silent fallback → replaced with fail-fast GradleException when a range-specific key is missing
Missing patterns → covered by the shared VersionFilters.isUnstable() from your PR
Also added test-latest-deps as a required status check in CI (branch protection still needs admin update)

- Remove unnecessary TODO about branch protection (trask feedback) - Cap muzzle open-ended version ranges using pinned versions from latest-dep-versions.json to prevent failures from newly released library versions (trask feedback) - Document version pinning in docs/safety-mechanisms.md Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

trask · 2026-03-11T17:29:51Z

sent a PR with a bit more feedback: zeitlinger#12

- Fail-fast when a range-specific pinned version key (e.g. "com.example:lib#2.+") is missing from the JSON instead of silently falling back to the base key which could resolve to a wrong major version. - Make test-latest-deps a required status check now that versions are pinned. Added TODO for branch protection settings (requires admin). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Remove unnecessary TODO about branch protection (trask feedback) - Cap muzzle open-ended version ranges using pinned versions from latest-dep-versions.json to prevent failures from newly released library versions (trask feedback) - Document version pinning in docs/safety-mechanisms.md Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Fail-fast when a range-specific pinned version key (e.g. "com.example:lib#2.+") is missing from the JSON instead of silently falling back to the base key which could resolve to a wrong major version. - Make test-latest-deps a required status check now that versions are pinned. Added TODO for branch protection settings (requires admin). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Remove unnecessary TODO about branch protection (trask feedback) - Cap muzzle open-ended version ranges using pinned versions from latest-dep-versions.json to prevent failures from newly released library versions (trask feedback) - Document version pinning in docs/safety-mechanisms.md Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Fail-fast when a range-specific pinned version key (e.g. "com.example:lib#2.+") is missing from the JSON instead of silently falling back to the base key which could resolve to a wrong major version. - Make test-latest-deps a required status check now that versions are pinned. Added TODO for branch protection settings (requires admin). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Remove unnecessary TODO about branch protection (trask feedback) - Cap muzzle open-ended version ranges using pinned versions from latest-dep-versions.json to prevent failures from newly released library versions (trask feedback) - Document version pinning in docs/safety-mechanisms.md Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger · 2026-03-16T14:44:08Z

Muzzle artifacts that fall back to dynamic Aether resolution

Six muzzle artifacts can't be pinned in latest-dep-versions.json because Gradle's dependency resolution fails for them. The muzzle-check plugin falls back to Aether (Maven resolver) for these, which matches the original behavior on main:

org.glassfish.grizzly:grizzly-http — latest version (5.0.0) depends on a SNAPSHOT (org.glassfish.grizzly:grizzly-framework:5.0.1-SNAPSHOT) that doesn't exist on Maven Central, causing transitive resolution failure.
org.restlet.jse:org.restlet — not published on Maven Central; hosted on maven.restlet.talend.com. Gradle's detached configuration only uses the standard repos.
io.opentelemetry:opentelemetry-instrumentation-annotations — relocated to group io.opentelemetry.instrumentation. The old coordinates don't resolve.
software.amazon.awssdk:bedrock-runtime — the actual artifact name is bedrockruntime (no hyphen). The muzzle directive uses a non-existent name.
com.clickhouse.client:clickhouse-client — the actual group is com.clickhouse, not com.clickhouse.client.
org.opensearch.client:rest — intentionally non-existent artifact used in a muzzle fail directive.

Items 3–5 are wrong/relocated coordinates in muzzle directives. We could add a coordinate-remapping table in the resolve task to pin these under their original keys (resolving via the correct coordinates). That would reduce the unpinnable set to just 3 (grizzly, restlet, opensearch).

@trask Worth adding coordinate remapping, or keep it simple with the Aether fallback for all 6?

trask · 2026-03-16T23:57:35Z

software.amazon.awssdk:bedrock-runtime — the actual artifact name is bedrockruntime (no hyphen). The muzzle directive uses a non-existent name.

com.clickhouse.client:clickhouse-client — the actual group is com.clickhouse, not com.clickhouse.client.

can we just fix the muzzle blocks to fix these?

zeitlinger · 2026-03-17T15:03:04Z

software.amazon.awssdk:bedrock-runtime — the actual artifact name is bedrockruntime (no hyphen). The muzzle directive uses a non-existent name.

com.clickhouse.client:clickhouse-client — the actual group is com.clickhouse, not com.clickhouse.client.

can we just fix the muzzle blocks to fix these?

good idea - done

in addition

the old group for clickhouse never resolved to anything, including assertInverse. With the correct group, older versions (0.4.x, 0.3.x) actually pass muzzle, so removed assertInverse for now, lowering the floor is a separate thing
Down to 4 unpinnable artifacts now (grizzly, restlet, otel-instrumentation-annotations, opensearch)

- Fail-fast when a range-specific pinned version key (e.g. "com.example:lib#2.+") is missing from the JSON instead of silently falling back to the base key which could resolve to a wrong major version. - Make test-latest-deps a required status check now that versions are pinned. Added TODO for branch protection settings (requires admin). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Remove unnecessary TODO about branch protection (trask feedback) - Cap muzzle open-ended version ranges using pinned versions from latest-dep-versions.json to prevent failures from newly released library versions (trask feedback) - Document version pinning in docs/safety-mechanisms.md Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

When -PtestLatestDeps=true, dependency versions are now pinned via .github/config/latest-dep-versions.json instead of resolving dynamically at build time. This makes builds reproducible across time. A new resolveLatestDepVersions task populates the JSON file, and a daily CI workflow opens a PR when versions change. The -PresolveLatestDeps=true flag bypasses pinning for live resolution. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

…docs Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Remove trailing blank line in build.gradle.kts for spotless. Use prefer instead of require for library/testLibrary version upgrades when pinLatestDeps is enabled. This prevents conflicts when a latestDepTestLibrary dependency uses strictly to constrain to an older major version (e.g. spring-boot-starter-test 3.3.x) while the pinned latest version is a newer major (4.0.x). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Two bugs in resolveLatestDepVersions: - When a range (e.g. activej-http#6.+) has only pre-release versions, fall back to the resolved pre-release instead of skipping the entry entirely. - When latest.release for a muzzle artifact resolves to a broken version (grizzly 5.0.0 depends on a SNAPSHOT BOM), fall back to the highest already-pinned version for that artifact instead of emitting no entry. Also regenerate latest-dep-versions.json. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- Fix muzzle: update otel-annotations muzzle group from io.opentelemetry to io.opentelemetry.instrumentation (correct Maven Central group) and add the corresponding pinned version entry - Fix testLatestDeps: revert async-http-client to 3.0.7; 3.0.8 has a broken POM declaring netty-codec-http2:4.0.34.Final which doesn't exist (codec-http2 only starts at Netty 4.1.x) - Fix resolveLatestDepVersions task: seed versions map from existing JSON so entries that fail to resolve are preserved rather than deleted, and add transitive-dep check to reject broken versions like 3.0.8 Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

- com.typesafe.akka:akka-actor_2.12 (latest.release is pre-release 2.9.0-M1) - io.netty:netty-all - io.opentelemetry:opentelemetry-instrumentation-annotations (sentinel 0.0 — artifact doesn't exist; this muzzle directive is intentionally a no-op, tests verify compatibility instead) - javax.servlet:servlet-api (latest.release is pre-release 3.0-alpha-1) - org.apache.pekko:pekko-actor_3 and pekko-http_3 (latest.release is pre-release) - org.opensearch.client:rest (sentinel 0.0 — artifact doesn't exist; fail directive) - org.restlet.jse:org.restlet (custom Maven repo, not auto-resolvable) - software.amazon.awssdk:bedrockruntime (fixes muzzle directive typo: bedrock-runtime) Also add "+" fallback in resolveLatestDepVersions for cases where maven <release> metadata points to a pre-release version. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

async-http-client 3.x depends on io.netty:*:4.0.34.Final for modules (netty-handler-proxy, netty-resolver-dns) that only exist from Netty 4.1.x, so the entire 3.x line is unresolvable. Pin to 2.12.4, the latest working release. Also add AcceptableVersions.isStable() to filterVersions() so that pre-release versions (RC, alpha, beta) are excluded from muzzle testing. Without this, assertInverse would fail when a pre-release of a supported version (e.g. 2.0.0-RC15) has a compatible API — producing a spurious "instrumentation unexpectedly passed" error. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

PR open-telemetry#17103 already removed this from main, but this branch pre-dated that fix and had re-introduced the line. When main was merged back in, the line survived because open-telemetry#17103 touched a different region of the file. extra["testLatestDeps"] stored a Boolean, which Gradle's findProperty() returns with higher precedence than the -P project property. All build scripts check findProperty("testLatestDeps") == "true", but Boolean(true) != "true" in Kotlin, so latestDepTest was always false. This caused the Netty force-to-4.0.34.Final block in async-http-client-2.0's build.gradle.kts to fire even in testLatestDeps mode, breaking dependency resolution since that old Netty version is gone from Maven Central.

quarkus-3.0-testing and quarkus-3.9-testing use version ranges "3.5.+" and "3.9.+" in testLatestDeps mode. The pin mechanism requires explicit entries for range keys or it throws an error. Latest stable: 3.5.3 and 3.9.5 (from Maven Central). Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

…deps fail Two bugs caused pre-release versions (e.g. mongodb-driver-sync:5.7.0-beta1) to get pinned instead of the latest stable: 1. resolveStableVersion() checked for unresolvable transitive dependencies, returning null if any were found (e.g. classifier-specific native JARs). Fix: disable transitive resolution — we only need the artifact itself. 2. recordVersion() used a pure numeric comparison, so a stable 5.6.4 would never replace a pre-release 5.7.0-beta1 already in the JSON. Fix: prefer stable over pre-release regardless of numeric ordering. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

…ase pin) Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger · 2026-04-08T14:39:57Z

@trask please have another look?

trask

Purely AI review, I take no credit or accountability 😂 (see #17889)...

Nice cleanup — pinning testLatestDeps and capping muzzle ranges with the same JSON gives us reproducible CI and removes the release-branch carve-out cleanly. A couple of small nits below; nothing blocking.

- Move AcceptableVersions import below the kdoc so the import block is contiguous with the file header comment. - Drop unused system/session/repos params from resolveUpperBound (the body only consults the pinned-versions map) and update the two callsites in inverseOf and muzzleDirectiveToArtifacts. Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

The 5.5.1 pin was added to work around a test failure with later versions, but pinning testLatestDeps versions in the PR's JSON config resolves httpclient5 to 5.6.1 where the test passes. Verified locally: ./gradlew :instrumentation:apache-httpclient:apache-httpclient-5.0:javaagent:test -PtestLatestDeps=true BUILD SUCCESSFUL.

trask · 2026-04-23T03:01:00Z

CI failing due to #18217

zeitlinger requested a review from a team as a code owner March 2, 2026 14:35

trask reviewed Mar 5, 2026

View reviewed changes

Comment thread conventions/src/main/kotlin/io.opentelemetry.instrumentation.base.gradle.kts Outdated

Comment thread conventions/src/main/kotlin/otel.resolve-latest-dep-versions.gradle.kts Outdated

trask mentioned this pull request Mar 6, 2026

Add consumer group flag to experimental suite #16398

Merged

trask reviewed Mar 6, 2026

View reviewed changes

Comment thread .github/workflows/build-pull-request.yml Outdated

Comment thread .github/workflows/build-pull-request.yml Outdated

trask reviewed Mar 9, 2026

View reviewed changes

zeitlinger force-pushed the pin-latest-dep-versions branch from 4c5245a to 0b4adf1 Compare March 12, 2026 15:36

zeitlinger force-pushed the pin-latest-dep-versions branch 2 times, most recently from e61e46e to f2ab30e Compare March 16, 2026 13:30

jaydeluca mentioned this pull request Mar 17, 2026

Fix clickhouse client v1 muzzle config #16644

Merged

zeitlinger force-pushed the pin-latest-dep-versions branch from 8debde7 to 887fade Compare March 31, 2026 15:36

otelbot-java-instrumentation Bot added the test native This label can be applied to PRs to trigger them to run native tests label Mar 31, 2026

zeitlinger added 3 commits April 7, 2026 11:19

Move resolveLatestDepVersions task to conventions plugin and improve …

82adc44

…docs Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger added 10 commits April 7, 2026 11:19

Document sentinel value 0.0 in latest-dep-versions.json

40a12b5

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Regenerate latest-dep-versions.json

2669cf4

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Regenerate latest-dep-versions.json (fix mongodb-driver-sync pre-rele…

3fce9ba

…ase pin) Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

zeitlinger force-pushed the pin-latest-dep-versions branch from 8e040ab to 3fce9ba Compare April 7, 2026 11:19

trask reviewed Apr 20, 2026

View reviewed changes

Comment thread conventions/src/main/kotlin/io.opentelemetry.instrumentation.base.gradle.kts Outdated

Comment thread gradle-plugins/src/main/kotlin/io.opentelemetry.instrumentation.muzzle-check.gradle.kts Outdated

zeitlinger and others added 7 commits April 20, 2026 08:32

Add missing akka 2.13 testLatestDeps pin

23e214f

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Limit latest dep test for apache-httpasyncclient

c591291

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Pin latest-deps Apache HttpClient 5.x test library

6a3a414

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Mark Apache HttpClient latest-deps pin as documented limitation

4a9d33d

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>

Merge branch 'main' into pin-latest-dep-versions

e7e5c71

update versions

72d4fe5

trask mentioned this pull request Apr 23, 2026

Fix duplicate examples key in log4j-appender-2.17 metadata.yaml #18217

Merged

trask added 5 commits April 23, 2026 07:14

Merge branch 'main' into pin-latest-dep-versions

31e66e7

updates

911d81c

Merge branch 'main' into pin-latest-dep-versions

574c2b4

updates

56e7874

Revert apache-httpasyncclient 4.1.4 cap (fixed in open-telemetry#18270)

9fd1700

trask force-pushed the pin-latest-dep-versions branch from 8c474cf to 9fd1700 Compare April 24, 2026 20:33

up

068b5c2

Conversation

zeitlinger commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Why a GitHub Action instead of Renovate?

Test plan

Uh oh!

Uh oh!

Uh oh!

trask commented Mar 5, 2026

Uh oh!

zeitlinger commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

trask commented Mar 11, 2026

Uh oh!

zeitlinger commented Mar 16, 2026

Uh oh!

trask commented Mar 16, 2026

Uh oh!

zeitlinger commented Mar 17, 2026

Uh oh!

zeitlinger commented Apr 8, 2026

Uh oh!

trask left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

trask commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zeitlinger commented Mar 2, 2026 •

edited

Loading