[DISCUSS] Move Apache Arrow Datafusion to a new top level Apache project

[alamb]

TLDR

Apache Arrow DataFusion --> Apache DataFusion ?

Introduction

Arrow has become a large and widely adopted project, with at least 12 language implementations, 96 committers, and 47 PMC members) at the time of this writing. It also has several sub projects such as FlightSQL, and at least two query engine sub-projects.

Apache Arrow DataFusion, a query engine written in Rust, has been part of Arrow since February 2019. In that time, it has helped drive the Rust implementations of Arrow arrow-rs and parquet. It also includes the Ballista engine since 2021 and datafusion-python bindings

Challenges

Being part of Arrow has been great for DataFusion. DataFusion has benefited from the existing stable and well understood governance structures, processes, and community. DataFusion contributors learned many things from the Arrow community, both technically as well as organizationally. The community has grown the point where there is now very active development from many different contributors and a history of multiple commits to the main branch daily for several years.

However, DataFusion arguably has a different mission than Arrow. Arrow powers and enables interoperability for in-memory analytics. DataFusion is a very fast, extensible query engine for building high-quality data-centric systems in Rust. DataFusion does use Arrow extensively and to good effect but not the other way around.

The DataFusion community is largely different as well. It has its own core set of committers, which largely don't overlap with other Arrow implementations or Arrow repositories. There are also many PMC members whose near sole focus appears to be DataFusion. Having non overlapping communities is sometimes challenging for communications such as on the mailing lists dev@apache.arrow.org because relevant and irrelevant content is intermixed. I have also head anecdotally that writing to the main dev@arrow.apache.org mailing list for questions about DataFusion is intimidating given its perceived broad reach.

Finally, DataFusion is technically different than Arrow. DataFusion is developed in its own repository, at a different release cadence (every 2 weeks vs 3 months for the main repo), and has a different documentation site.

Thus given the differences in mission, code, and community I believe both Arrow and DataFusion could benefit from being separated so they can continue to focus and grow.

Proposal:

I propose we move DataFusion to its own top level Apache project (graduate the subproject in Arrow to its own top level project). The project could be "Apache DataFusion"

I think we need a community discussion and consensus before taking any steps so I wanted to start the conversation here.

Specifics

1. Follow the model here: https://incubator.apache.org/guides/graduation.html#whether_to_graduate_to_subproject_or_to_top_level_project
2. All existing Arrow Committers and PMC members who so desired, would start as committers or PMC members on the new DataFusion project (assuming this is allowed by the process)
3. Existing DataFusion focused Arrow Committers / PMCs decide on a case by case basis if they want to resign from Arrow

Benefits to DataFusion being its own top level project:

1. DataFuson will remain an Apache project, with the same governance structure and procedures, though the PMC would be more focused
2. Relationship between Arrow and DataFusion will be clearer, hopefully improving the external communication, and adoption of both systems
3. Internal communications will be clearer: it will be clearer to which mailing list / chat discussions and people won't have to filter mailing list traffic manually
4. Governance bandwidth will be more focused -- both in Arrow as well as DataFusion the PMCs will have

Potential Downsides

1. One time switching cost for ASF and github process changes (I am willing to personally do a substantial amount of this)
2. Support we get today from Arrow (e.g website, etc) and backup governance would no longer be present (however, I think we now have sufficient momentum and size of community that DataFusion can be self sustaining at this point)
3. Communication and flow of ideas between arrow-rs and arrow-datafusion may be less (However, I think the communities can and will work closely together)

Open questions:

1. Should we consider bringing Acero along (the C++ query engine) as it is similar in technology (though the community is mostly different)

Related

• DataFusion Vision Discussion

[tustvold]

Having non overlapping communities is sometimes challenging for communications such as on the mailing lists

I could be mistaken, but I'm not sure there is much overlap between the various language implementations in general. I'm personally not aware of this being a problem.

I definitely think this relates to #6441, as if the focus is an arrow-native query engine, it seems in both arrow and DFs interests to promote cross-pollination between the projects by not introducing additional beuracratic hurdles between the two. If the focus is instead on the non-execution components, then perhaps that level is a more logical place to draw some kind of separation, similar to Velox?

[alamb]

I want to be clear that I view this proposal as mostly a "marketing" / "branding" exercise with some governance improvements thrown in -- I don't expect there to be much impact on the day to day development activities of DataFusion and arrow-rs

[rtyler]

I agree with @alamb's proposal in that the distinction does greatly benefit Datafusion from a marketing and branding perspective.

[julianhyde]

ASF has a mantra "community over code". Applying that here, Data Fusion seems to have a separate community and therefore should be a separate project.

This wouldn't be cutting ties with Arrow. The projects can still interoperate. But being separate projects, they don't have to coordinate and remain in lock-step. Their release schedule, governance, and community practices can drift the way their communities want them to go.

People have asked whether "in Rust" should be part of the project description. From what I can see, Rust is a big unifying factor in the Data Fusion community, so I am inclined to say yes.

[andygrove]

ASF has a mantra "community over code". Applying that here, Data Fusion seems to have a separate community and therefore should be a separate project.

I totally agree with this. I am +1 for this proposal. I believe that there is enough momentum in the project to justify moving it to a top-level project.

[westonpace]

This makes a lot of sense to me and I think it's a great thing. I see this as the ecosystem growing to the point where we can support two complete projects.

Should we consider bringing Acero along (the C++ query engine) as it is similar in technology (though the community is mostly different)

If others are interested then I think that's fine. However, my personal opinion is that this probably isn't necessary at this point. I'm just not sure what benefits there would be.

[ozankabak]

I think this is a good idea overall.

ASF has a mantra "community over code". Applying that here, Data Fusion seems to have a separate community and therefore should be a separate project.

I agree with this and I think being a project on its own would benefit DataFusion. A PMC/committer roster with members actively contributing to the Datafusion project would improve governance and speed of progress IMO.

[alamb]

Yes, this is my hope as well -- and hopefully it will make it clearer where each PMC / committer spends their time

[mingmwang]

Totally agree!

[liurenjie1024]

Looking forward to it!

[liukun4515]

agree with this proposal

[jackwener]

Agree with this idea.

[stormasm]

I think this is a great idea !!

On the Acero question ?

Should we consider bringing Acero along (the C++ query engine) as it is similar in technology (though the community is mostly different)

Because the community is different I don't think it makes much sense....

If others are interested then I think that's fine. However, my personal opinion is that this probably isn't necessary at this point. I'm just not sure what benefits there would be.

I agree with the above point as well that there is no need to bring Acero along...

[alamb]

FWIW an update here is that once my term as Arrow PMC chair runs out, I will put my organizational energy into figuring out the next steps and moving this proposal forward. I expect that to be in a few more months

[alamb]

It has been brought to my attention that changing the project structure (even a new project structure in Apache) may cause some disruption, espeically for contributors from large companies where contributing to each Apache project must be specifically vetted.

WIth this in mind, I will ensure to over communicate any movement on this issue and will ensure consensus prior to making anything official

[alamb]

For anyone following along we are working on a more specific proposal via #8491

[alamb]

Update:

Here is the specific proposal for the new top level project
https://docs.google.com/document/d/11WTNYS8KWScOt3ySTX39WVS6krPhUvHsuJRY9PZQx4g.

The expected timeline is

1. A formal vote in February 2024
2. Submit for the board's approval in April 2024.

See mailing list discussion for more details

[ozankabak]

Thank you @alamb for working on this 💪

[liurenjie1024]

Hi, what's the status now?

[alamb]

Hi @liurenjie1024 - I still plan to submit for the formal vote this week (I am a bit behind -- thank you for the reminder) and then submit for formal ASF board approval at the April Board meeting

[liurenjie1024]

Thanks for the effort!

[alamb]

I have updated the proposal document https://docs.google.com/document/d/11WTNYS8KWScOt3ySTX39WVS6krPhUvHsuJRY9PZQx4g/edit with additional information (new committers Jeffrey Vo and Jay Zhan, and the new datafusion-comet repository)

I plan to:

1. Call for a formal vote on the dev@arrow.apache.org mailing list this Friday March 2
2. If the vote passes, submit the proposal to the ASF board as part of the April 2024 Arrow report.

This extended timeline is designed to balance the needs of some contributors to prepare for the changed structure with their employers.

[alamb]

https://lists.apache.org/thread/pq2k8o45trgp15g4d6xvyx5f7hvrcj1s (cross posted to mailing list)

[alamb]

Here is the official vote thread on the arrow dev list: https://lists.apache.org/thread/tv8s8ootxf7nrsp3vo1mt8mtxxt5qcor

(if that passes, we'll submit to the board for approval in April)

[alamb]

Update -- vote passed, and we plan to proceed with the board vote in April as planned

[phillipleblanc]

Thanks for all your work in moving this forward!

[alamb]

Added the board resolution to the draft ASF Arrow board report: https://docs.google.com/document/d/1q6uBW4MNijY8cThZf0d-XZTcCxAOhUgp1sUG6_eOBMY/edit (mailing list link: https://lists.apache.org/thread/qyg07kdvjnb0178087kpt5osqglrrw8x)

[alamb]

We are also tracking todo items after we become a top level project in #9691

[alamb]

Its official 🎉 Apache DataFusion is now its own top level project in the Apache Software Foundation

https://projects.apache.org/project.html?datafusion

We are tracking various todos here: #9691