feat: [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines #14261

PavithranRick · 2025-11-13T21:59:07Z

Describe the issue this Pull Request addresses

This PR introduces a new comprehensive show_timeline procedure for Hudi Spark SQL that provides detailed timeline information for all table operations. The procedure displays timeline instants including commits, deltacommits, compactions, clustering, cleaning, and rollback operations with support for both active and archived timelines and completed/pending state instants.

Summary and Changelog

Comprehensive timeline view:
Shows all timeline instants with detailed metadata including state transitions (REQUESTED, INFLIGHT, COMPLETED)

Time-based filtering:
Support for startTime and endTime parameters to filter results within specific time ranges

Archive timeline support:
showArchived parameter to include archived timeline data for complete historical view

Generic SQL filtering:
filter parameter supporting SQL expressions for flexible result filtering

Rich metadata output:
Includes formatted timestamps, rollback information, and table type details

The procedure replaces multiple fragmented timeline-related procedures with a single unified interface that provides both pending and completed instant information with partition-specific metadata support.

Impact-related changelog details:

New procedure: show_timeline with parameters (table, path, limit, showArchived, filter, startTime, endTime)
Enhanced schema: 8-column output including instant_time, action, state, requested_time, inflight_time, completed_time, timeline_type, rollback_info
Backward compatibility: existing timeline procedures remain functional (deprecated with guidance to use the new procedure)

Impact

User-facing Features:

Unified timeline interface replacing multiple specialized procedures
Advanced filtering capabilities: time-based + SQL expression filtering
Historical data access through archive timeline support

Performance Impact:

Optimized timeline scanning with proper extension filtering
Configurable limits (default 20 entries if no start/end time provided)
Archive timeline accessed only when explicitly requested

Risk Level

Low

Verification performed

Comprehensive test coverage: 8 focused test cases covering basic functionality, MoR tables, rollback operations, and state transitions, timeline
Schema validation: all output fields properly typed and validated
Error handling: graceful handling of invalid filters, missing tables, and timeline access failures
Timeline consistency: proper handling of both active and archived timelines with correct state mapping

Documentation Update

Add show_timeline procedure to Hudi Spark SQL procedures documentation
Update timeline management examples to use new procedure
Add advanced filtering examples and use cases

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

…ring and generic SQL filters on top

…-for-show_timeline-Procedure-with-appropriate-start-and-end-time-for-both-active-and-archive-timelines

…ring and generic SQL filters on top

…Support-for-show_timeline-Procedure-with-appropriate-start-and-end-time-for-both-active-and-archive-timelines

…9766

…ests for show timelies

…oodieInstant

…from timeline

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineV2.java

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java

...park/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala

danny0405 · 2025-11-20T03:47:55Z

...park/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala

+    val limitedInstants = if (startTime.trim.nonEmpty && endTime.trim.nonEmpty) {
+      sortedInstants
+    } else {
+      sortedInstants.take(limit)


if we have a final limit on the timeline instants, do we stil need to add limit to the timeline API?

…rchived timeline partial loading (stop after limit / avoid loading entire archived timeline)

nsivabalan

sharing some feedback for now. will review the rest later today.

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java

hudi-common/src/main/java/org/apache/hudi/common/table/timeline/StoppableRecordConsumer.java

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java

.../main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineLoaderV1.java

.../main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineLoaderV2.java

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineV2.java

.../main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineLoaderV2.java

...park/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala

nsivabalan · 2025-11-22T02:23:51Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+
+import org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase
+
+class TestShowTimelineProcedure extends HoodieSparkSqlTestBase {


lets try to cover all cases listed below

both v1 and v2

limit 50.

both active and archived 20. where active contains 40.

both active and archived 20, where active contains 10.

start within active timeline.

end within active timeline.

start and end w/n active timeline.

start in archived. but arch not explicitly enabled.

"". archived enabled.

start in archived and end in active timeline. archived not enabled.

start in archived and end in active timeline. archived enabled.

start and end in archived. arch not enabled.

start and end in archived. arch enabled.

completed commits.
infight commits.
completed dc (mor)
inflight dc (mor)
clean commits.
clean inflight commits.
completed compaction.
infligth compaction.
completed replace commits. -> clustering
inflight replace commits. -> clustering
completed replace commits. -> insert overwrite
inflight replace commits. -> insert overwrite
completed rollback.
pending rollback.

In active timeline, we should have mix of inflight and completed. but in archived, we can only have completed and its fine.

danny0405 · 2025-11-22T04:41:55Z

...park/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala

+    // This way, if all archived instants are older than the active timeline's max instant,
+    // the archived timeline will be empty and won't load anything, avoiding unnecessary loading.
+    // Instead of getArchivedTimeline() which loads with LoadMode.ACTION, we use the startTs
+    // constructor which loads with LoadMode.METADATA, and then load specific details (PLAN for compactions).


The LoadMode.METADATA does not include the plan, are you saying the LoadMode.PLAN or LoadMode.FULL ?

Even if we use the constructor ArchivedTimelineV2(HoodieTableMetaClient metaClient, String startTs), the whole timeline would still be loaded with filtering, a better way to avoid the eager loading is adding a new API in TimelineFactory.createArchivedTimeline(HoodieTableMetaClient metaClient, boolean loadingInstants), add in each ArchivedTimeline, we add a new constructor that accepts the meta client but does not load any instants by default.

Because in the use cases of this PR, we always want a lazy loading of archived timeline and the load is triggered as needed.

@danny0405 - ArchivedTimelineV2(HoodieTableMetaClient metaClient, String startTs) - we are using the startTs as the firstInstantTime from active timeline. So all instants in archivedTimeline will be before this starting point. So, we won't be reading the records inside the files.

The archived files will skipped (based on start timestamp comparison of each file). Tested locally that we are not reading the records inside the file like before.
CC: @nsivabalan

Also, we are using empty archived timeline only in SQL procedure. currently we do not have a use case outside of this SQL procedure

The archived files will skipped (based on start timestamp comparison of each file). Tested locally that we are not reading the records inside the file like before.

For V2 this might be true, but not for V1?

danny0405 · 2025-11-22T10:59:03Z

.../main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineLoaderV2.java

+      }
+
+      // Sort files in reverse chronological order if needed (newest first for limit queries)
+      if (stoppable != null && stoppable.needsReverseOrder()) {


If we do not have good way to plugin the limit logic simply and clean, maybe we just add a separate method in ArchivedTimelineLoader.loadInstants with an explicit param StoppableRecordConsumer, the benefits:

get rid of the null check and instance of check;

always sort the files in reverse chronological order;

read the files in single thread instead of in parallel.

Read with limit is somehow a range query instead of full scan, by doting this, we can freely plugin in the logic required for limit while still keep the basic scan query efficient and clean. We can always comeback to this for better abstraction when it is necessary.

Thank you for the suggestion. My concern is by adding a separate method, we would be duplicating the same logic and additional maintenance overhead.

makes sense. Accepted the suggestion and made relevant changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java

… tests

…use atomic boolean flag

…9766

PavithranRick

addressed the comments and rebasing with master to resolve conflicts

…d times of archived timeline

.../main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineLoaderV1.java

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java

nsivabalan · 2025-11-25T22:27:50Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+   * Helper method to create a table with all types of commits for comprehensive testing.
+   * Creates: completed/inflight commits, deltacommits (MOR), clean, compaction, clustering, insert overwrite, rollback
+   */
+  private def setupTableWithAllCommitTypes(tableName: String, tableLocation: String, tableType: String): Map[String, String] = {


didn't I point you to test code where we prepare the timeline directly?
we should avoid doing end to end functional set up if not really required.

refer to TestArchivedTimelineV1.createInstants() and createAdditionalInstants()
and TestArchivedTimelineV2.writeArchivedTimeline(...)

nsivabalan · 2025-11-25T22:28:44Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+
+    // 2. Inflight commits - create and leave one inflight by interrupting
+    spark.sql(s"insert into $tableName values(4, 'a4', 40, 4000)")
+    Thread.sleep(100) // Small delay


why do we need the sleep in many places ?

nsivabalan · 2025-11-25T22:30:43Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+
+    // Scenario 1: limit 50
+    val limit50Result = spark.sql(s"call show_timeline(table => '$tableName', limit => 50)").collect()
+    assert(limit50Result.length <= 50, s"Scenario 1: Should have at most 50 entries, got ${limit50Result.length}")


can we validate the instant times as well.
would be much easier if we have prepared the timeline in test.
something to think about.

nsivabalan · 2025-11-25T22:31:44Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+      result6.foreach { row =>
+        val instantTime = row.getString(0)
+        assert(instantTime >= startTime && instantTime <= endTime,
+          s"Scenario 6: Entry $instantTime should be in range [$startTime, $endTime]")


lets revisit all validations and make it tight

nsivabalan · 2025-11-25T22:32:33Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+        val archivedEnd = archivedTimes.last
+        val result11 = spark.sql(s"call show_timeline(table => '$tableName', startTime => '$archivedStart', endTime => '$archivedEnd')").collect()
+        result11.foreach { row =>
+          assert(row.getString(6) == "ACTIVE" || row.getString(0) >= archivedStart,


shouldn't result11 expected to be an empty list

nsivabalan · 2025-11-25T22:33:19Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+        println(s"V1 COW test: Table version after downgrade: $versionAfter")
+//        // Verify table version was downgraded to SIX
+//        assert(versionAfter == HoodieTableVersion.SIX,
+//          s"V1 COW test: Table version should be SIX after downgrade, but got $versionAfter (was $versionBefore)")


why commented out ?

nsivabalan · 2025-11-25T22:36:17Z

...udi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestShowTimelineProcedure.scala

+            s"V1 COW: Archived timeline should only have COMPLETED states, got: ${archivedStates.mkString(", ")}")
+        }
+
+        // Verify specific commit types exist for V1 COW


why do this for V1 only?

…loading with limit param

hudi-bot · 2025-11-26T02:15:02Z

CI report:

c0a4a97 UNKNOWN
3bd5aa3 UNKNOWN
bc8da37 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

vamshikrishnakyatham and others added 7 commits August 28, 2025 17:46

[HUDI-9766] Support for show_timeline procedure with time-based filte…

ffe4ca1

…ring and generic SQL filters on top

[HUDI-9766] Support for show_timeline procedure with time-based filte…

1827abc

…ring and generic SQL filters on top

Merge remote-tracking branch 'upstream/master' into HUDI-9766-Support…

09594b3

…-for-show_timeline-Procedure-with-appropriate-start-and-end-time-for-both-active-and-archive-timelines

[HUDI-9766] Support for show_timeline procedure with time-based filte…

afd7d6b

…ring and generic SQL filters on top

optimization on driver for reading instants

521ed76

Merge branch 'master' of ssh://github.com/apache/hudi into HUDI-9766-…

a0d33b5

…Support-for-show_timeline-Procedure-with-appropriate-start-and-end-time-for-both-active-and-archive-timelines

Merge branch 'master' of ssh://github.com/apache/hudi into pavi-hudi-…

c0a4a97

…9766

github-actions bot added the size:L PR with lines of changes in (300, 1000] label Nov 13, 2025

Pavithran Ravichandiran added 2 commits November 17, 2025 11:11

HUDI-9766 - Added instant loader for archived timeline v2

45aec77

HUDI-9766 - Added instant loader for archived timeline v2 and added t…

5c0a7ae

…ests for show timelies

github-actions bot added size:XL PR with lines of changes > 1000 and removed size:L PR with lines of changes in (300, 1000] labels Nov 18, 2025

Pavithran Ravichandiran added 3 commits November 18, 2025 16:08

HUDI-9766 - populating timestamps for achived timeline entries from H…

77d9162

…oodieInstant

HUDI-9766 - using existing methods for filtering time range instants …

3d71552

…from timeline

HUDI-9766 - show_timeline new tests for start and end time

c2daa7a

PavithranRick changed the title ~~[HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines~~ feat: [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines Nov 19, 2025

danny0405 reviewed Nov 20, 2025

View reviewed changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 20, 2025

View reviewed changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v2/ArchivedTimelineV2.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 20, 2025

View reviewed changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java Outdated Show resolved Hide resolved

danny0405 reviewed Nov 20, 2025

View reviewed changes

...park/src/main/scala/org/apache/spark/sql/hudi/command/procedures/ShowTimelineProcedure.scala Outdated Show resolved Hide resolved

danny0405 reviewed Nov 20, 2025

View reviewed changes

Pavithran Ravichandiran added 2 commits November 21, 2025 13:02

HUDI-9766 - show_timeline - addressed PR comments, bug fixes around a…

47eee42

…rchived timeline partial loading (stop after limit / avoid loading entire archived timeline)

HUDI-9766 - show_timeline new tests for start and end time

3ce75dd

nsivabalan reviewed Nov 21, 2025

View reviewed changes

nsivabalan reviewed Nov 22, 2025

View reviewed changes

danny0405 reviewed Nov 22, 2025

View reviewed changes

linliu-code reviewed Nov 24, 2025

View reviewed changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java Outdated Show resolved Hide resolved

linliu-code reviewed Nov 24, 2025

View reviewed changes

...on/src/main/java/org/apache/hudi/common/table/timeline/versioning/v1/ArchivedTimelineV1.java Outdated Show resolved Hide resolved

HUDI-9766 - show_timeline - addressed PR comments and added extensive…

3bd5aa3

… tests

Pavithran Ravichandiran added 2 commits November 24, 2025 17:12

HUDI-9766 - show_timeline - timeline loaders extra check removal and …

81be85d

…use atomic boolean flag

Merge branch 'master' of ssh://github.com/apache/hudi into pavi-hudi-…

f97ff88

…9766

PavithranRick commented Nov 25, 2025

View reviewed changes

HUDI-9766 - show_timeline - bug fix for loading based on start and en…

0f7d5c8

…d times of archived timeline

nsivabalan reviewed Nov 25, 2025

View reviewed changes

HUDI-9766 - show_timeline - removed instantloaderlimit with func over…

bc8da37

…loading with limit param


		import org.apache.spark.sql.hudi.common.HoodieSparkSqlTestBase

		class TestShowTimelineProcedure extends HoodieSparkSqlTestBase {

feat: [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines #14261

Are you sure you want to change the base?

feat: [HUDI-9766] Support for show_timeline Procedure with appropriate start and end time for both active and archive timelines #14261

Conversation

PavithranRick commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Verification performed

Documentation Update

Contributor's checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

PavithranRick left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nsivabalan Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

PavithranRick commented Nov 13, 2025 •

edited

Loading

danny0405 Nov 22, 2025 •

edited

Loading

danny0405 Nov 22, 2025 •

edited

Loading

PavithranRick left a comment •

edited

Loading

nsivabalan Nov 25, 2025 •

edited

Loading