[DISCUSS] Release retrospective

[timsaucer]

This topic came up in the recent DataFusion community meeting. There is interest in having a retrospective of our current release practices.

Background

DataFusion major version 52 ended up with 4 releases and another underway:

• Release DataFusion 52.4.0 (minor/) Release (Mar 2026) #20855
• Release DataFusion 52.3.0 (minor/) Release (Mar 2026) #20681
• Release DataFusion 52.2.0 (minor/) Release (Feb 2026) #20287
• Release DataFusion 52.1.0 (minor/patch) Release (Jan 2026) #19784
• Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) #18566

At the same time we are trying to release 53.0.0. Trying to release patches on past versions while simultaneously releasing a new major version may split maintainer time and focus in undesirable ways.

Additional references

Discussion about the frequency of releasing: #16622

We are currently updating the release documentation: #21001

Request

This issue is to collect feedback from all community members about the recent release process. The discussion above is where the topic of frequency should happen.

There is a plan to have an in-person retrospective at the next community meeting, March 25. @mbutrovich has volunteered to try capturing notes from that meeting.

For people who want to share thoughts asynchronously or cannot attend the community meeting, please give us your feedback about the recent release process.

• What went well?
• What could be improved?
• What did we learn?
• What should we commit to changing?

A list of things to ensure nothing gets lost

• git tag validation requested @coracuity
• release management doc update

[comphead]

Thanks @timsaucer for driving this.

While fighting with DF 53 release I would like to share some experience that seemed reasonable to me

What went well?

Overall community reaction went well IMO, community made its parts brillianty:

• responsibly tests RC
• make quick turnaround on found bugs or provide workarounds
• support with backports

What could be improved?

Release documentation can be improved:

• some steps(more specific backports and some other minors) in release process can be elaborated, I'll create a PR soon to address this

Code quality can be improved:

• it is fine to support more than 1 release, like Apache Spark supports versions 3 and 4 at the same time, but having so many hotfixes suggests some regressions are found on later stages.

What did we learn?

Still processing this

What should we commit to changing?

• IMO release process needs to be refined
• Prerelease verification might include fuzz testing(currently it is pretty limited set), or quasi fuzz testing where genAI generates thousands of valid queries with normal and extreme values
• Prerelease verification can also include planner and query benchmarking to identify performance regressions

[alamb]

some steps(more specific backports and some other minors) in release process can be elaborated, I'll create a PR soon to address this

FYI I made one here to try and improve the backporting docs:

• Add release management page to the documentation #21001

[alamb]

What went well?

We fixed many bugs quickly and efficiently

It was a great team effort (we had multiple releases run by multiple different people - @timsaucer @comphead @xudong963 and myself 🎉

I actually also think the 53 release will be higher quality for having sat as a branch for a while (not immediately making a RC). While it was somewhat of a pain to backport PRs, that can be mostly automated and I think it gave us a chance to stabalize the release more.

What could be improved?

I think it would help to more explicitly focus on regression bugs (most of what triggered the 52 releases) -- in my mind this means

1. Having some way to flag regressions when they are filed in issues/PRs (so we can find and prioritize their reviews and fixes)
2. Making it clear that our priority is (at least my priority is) fixing regression bugs

What did we learn?

Running these releases takes a long time

What should we commit to changing?

I think we need to consider how to potentially slow down our release scheduled (to one major release every quarter, for example, rather than every month) or marking some as LTS releases where we have some way to make bug fixes without also new features.

[coracuity]

I wanted to call attention to this situation as an opportunity to improve the release process: #19692 (comment)

Specifically, how did bad tags get pushed up? Can we prevent it or automate it such that it's impossible or improbable?