[Meta-EPIC] Organizing DataFusion's Growth #21036
Description
I feel that DataFusion's community (users, contributors, stakeholders) has grown to a point where we may need to devote some more time ("resources") to more explicit organization to support that growth.
There are several factors that are driving growth
- More users!
- More systems "in critical production" in the sense that even small changes in behavior can be perceived as regressions for existing users
- More contributors and parallel work, which increases the need for explicit roadmap and coordination.
This is my attempt to gather a meta summary of challenges related to growth in one place.
Release Challenges
- [DISCUSS] Release retrospective #21034
- [Discuss] Release cadence / patch releases / Long Term Supported (lts) minor releases #5269
- Increase the cadence for publishing patch releases. #20230
Communication channels and “where should this conversation happen?” are a source of friction.
Possibly related to above, As @neilconway recently told me, the current DataFusion communication system seems a bit "everything everywhere all the time" in the sense that there is discord, github discussions, github issues, github pull requests, slack the website, and the weekly sync call and it is not always clear how to make sure you see relevant conversations
- [EPIC] Redesign DataFusion main page #14389
- Where do I ask questions about DataFusion? #2861
- Move discussion from ASF Slack to discord #7320
- [EPIC] Redesign DataFusion main page #14389
Not clear how to get consensus for large changes
I think in general it is not clear
- What "large changes" are currently being considered
- What criteria is used to decide what constitutes a large change
- Who "decides" if we will make a large change
This is definitely related to roadmap discussions as well
- More thorough contribution guideline #15365
- Q2 2026 roadmap planning #21017
- [DISCUSSION] DataFusion Road Map: Q1 2026 #18494
Hard to know how to help for new contributors
I don't always think it is easy for people to find ways to help. For example, if I have some time to review PRs, which one would be the most helpful?