1.2 Release Planning

[vinothchandar]

Starting this discussion to plan out the 1.2 release. I propose moving to a faster release cycle this time, where we pick 1-2 key features per track and release when they are ready to go out.

Will start separate comments on each area and mention contributors I know, who are working on those.

[vinothchandar]

Spark

• MIT rewrite
• Explore removing caching on upsert path (JIRA)
• DataSource V2 support (?)
• End-to-end row writing including Hudi streamer (?)
• Deprecation and clean-up
  • Deprecate Spark 3.3 support
  • Remove glob support, unused relations

[geserdugarov]

@yihua Can we keep Spark 3.3 version and deprecate the older spark versions like 2.4, 3.1, 3.2? As part of Hudi 2.0 we can deprecate all versions older than Spark 3.5.

But we already removed support of Spark 2.4 #11788 and 3.0-3.2 #11692 .

[suryaprasanna]

My bad, I missed it. Then can we not do deprecate Spark 3.3 for Hudi 1.2 release and directly remove 3.3 and 3.4 as part of 2.x release?
Is there a specific reason we want to remove this support, or is it simply part of a standard deprecation process due to the feature being outdated?

[yihua]

@geserdugarov Yes, DataSource V2 support takes more than 1 release cycle based on the current timeline. If we can scope out the work to be done for DataSource V2 support, especially for the performance, we should get some ground work done in this story.

[yihua]

@suryaprasanna Maintaining the Hudi integration for more than 3 or 4 Spark versions brings overhead to development, as we need to test against all these versions for new features and account for version-specific APIs as Hudi is deeply integrated with Spark internals. There are certain logic that's Spark 3.3 specific which can be removed once Spark 3.3 is deprecated, simplifying the code base. The latest patch release of Spark 3.3 is Spark 3.3.4, which is released almost 2 years ago. Spark 4.1.0 is coming soon (https://spark.apache.org/news/spark-4-1-0-preview4-released.html). So IMO we should gradually remove old Spark version support based on the latest Spark releases, and focus more on new Spark versions (e.g., Spark 3.5.x and latest Spark 4.x which are actively maintained by Spark community) to save our development cycles. Wdyt?

[suryaprasanna]

@yihua If maintaining compatibility with older Spark versions is blocking or hindering progress on datasource V2 APIs, we can proceed with deprecating Spark 3.3.
CC @prashantwason

[vinothchandar]

Storage Format

• Type system (RFC-99) (to have support for) discussion thread
• Blob type & Vector types for RFC-100
• New file format support to ensure for storing unstructured content (RFC-100) (Lance file format integration (board)
• Vector Index (ANN) (Review Suryas [PR]
• Spark TVF for Vector Search RFC 102; with End-End integration with RAG application to query tables using text

[vinothchandar]

@rahil-c @the-other-tim-brown @voonhous to confirm/add/remove

[the-other-tim-brown]

Do we want to include Variant support or is that expanding the scope too far?

[vinothchandar]

Storage Engine

• Performance certification of SI/RLI reads/writes.
• Schema abstraction – don’t use Avro directly.
• HoodieStorage enhancement on reader and writer paths

[yihua]

There are HoodieStorage enhancements that @mansipp and @CTTY would like to contribute.

[vinothchandar]

Flink

• Automatic Schema Evolution: internal doc
• Async clustering for Flink
• RLI support for Flink writing: internal doc
• New Flink source with FLIP 27: RFC-design
• LSM file layout for streaming: issue
• Dynamic bucket scaling for bucket index: (still planning)
• Virtual column read support, e.g, the _hoodie_commit_time: issue
• Flink 2.1 support

[vinothchandar]

@cshuo @danny0405 to update items along the theme of making Flink more performant, reliable. and have total equivalent parity with the Spark engine support.

[voonhous]

CMIIW, automatic schema evolution refers to Hudi support as a sink for Flink-CDC? @cshuo @danny0405

[danny0405]

@voonhous yes.

[cshuo]

maybe we should also include flink 2.1 support, which contains some new useful features, e.g., variant type and delta join.

[parisni]

May contributor suggest stuff ? If so let me propose full support for table documentation (doc, flat and nested comments) for both reading hudi and writing into hudi from spark, trino or any hms/glue/s3 hudi compliant readers

• fix: aws glue sync set comment #14234
• [HUDI-5533] Support spark nested columns comments #14235

Comment support has been commited back in 2022 #4960 but the feature has never been working due to lack of suppprt for comment extraction from the hudi avro schema.

The nested comment suppports allows spark io to propagate the comments transparently thought the transformation lineage, which is AFAIK unique among acid format (delta/iceberg) do not support nested comment, only first level.

If this is of interest i can also work to backport this work for 0.15 since it can be considered as a bug fix.

[vinothchandar]

yes absolutely @parisni ..

[danny0405]

yes, it's useful, I can help for the code review of these two PRs.