feat(frontend): add refreshable property & refresh command #22619

xxchan · 2025-07-16T02:29:18Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

First part of #22690. See the tracking issue for the whole picture of the user journey.

Frontend part.

Key changes include:

Added a refreshable flag to the table catalog to indicate whether a table supports refresh operations. In this PR, there's no way to set the flag yet.
Implemented a new REFRESH TABLE SQL command and corresponding handler. It takes effect for table with refreshable flag (so no way to use it in this PR).
Added new barrier mutations RefreshStart (send by REFRESH TABLE SQL) and LoadFinish (will have a new mechanism to send it, introduced in the following PR.)

Checklist

I have written necessary rustdoc comments.

xxchan · 2025-07-16T02:29:34Z

feat: refreshable source, SourceExec part #22620 : 2 dependent PRs (#22621 , #22733 )
feat(frontend): add refreshable property & refresh command #22619 👈 (View in Graphite)
feat: minor improve risedev d in cursorrule #22689
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

xxchan · 2025-07-22T08:56:44Z

src/connector/src/with_options.rs

+    fn is_refreshable_connector(&self) -> bool {
+        false
+    }


This decides whether a table can be refreshed. So not possible to create refreshable table in this PR

graphite-app · 2025-07-23T02:11:04Z

Looks like this PR extends new SQL syntax or updates existing ones. Make sure that:

Test cases about the new/updated syntax are added in src/sqlparser/tests/testdata. Especially, double check the formatted_sql is still a valid SQL #20713
The meaning of each enum variant is documented in PR description. Additionally, document what it means when each optional clause is omitted.

cursor

Bug: Refresh Syntax Parsing Error

The REFRESH statement parser incorrectly mandates the TABLE keyword (REFRESH TABLE <table_name>). This conflicts with existing test data and the AST display format, which both imply REFRESH <table_name> is the expected syntax, leading to parsing failures for the simpler form.

src/sqlparser/src/parser.rs#L377-L381

risingwave/src/sqlparser/src/parser.rs

Lines 377 to 381 in 6f04b2b

    
           pub fn parse_refresh(&mut self) -> ModalResult<Statement> { 
        
               self.expect_keyword(Keyword::TABLE)?; 
        
               let table_name = self.parse_object_name()?; 
        
               Ok(Statement::Refresh { table_name }) 
        
           }

Fix in Cursor • Fix in Web

Bugbot free trial expires on July 29, 2025
Learn more in the Cursor dashboard.

Was this report helpful? Give feedback by reacting with 👍 or 👎

Signed-off-by: xxchan <[email protected]>

chenzl25 · 2025-07-25T07:02:35Z

proto/stream_plan.proto

+message LoadFinishMutation {
+  // Associated source ID for this load operation.
+  uint32 associated_source_id = 1;
+}


Do we need table_id here as RefreshStartMutation?

Actually I think for RefreshStartMutation we also just need the source id, because currently StreamSource only has source_id, but not the corresponding table id, while MaterializeNode has the catalog.Table, containing both table_id and associated_source_id. 🤣

chenzl25 · 2025-07-25T07:05:58Z

src/meta/src/stream/refresh_manager.rs

+    /// 1. Validates that the table exists and is refreshable
+    /// 2. Sends a refresh command through the barrier system
+    /// 3. Returns the result of the refresh operation
+    pub async fn refresh_table(&self, request: RefreshRequest) -> MetaResult<RefreshResponse> {


When manually triggering a refresh command, do we need to send the loadFinish command as well?

The loadFinish command is triggered after SourceExec reporting data load finished. added in #22620

chenzl25

LGTM

chenzl25 · 2025-07-25T08:36:23Z

src/meta/src/stream/refresh_manager.rs

+    /// 1. Validates that the table exists and is refreshable
+    /// 2. Sends a refresh command through the barrier system
+    /// 3. Returns the result of the refresh operation
+    pub async fn refresh_table(&self, request: RefreshRequest) -> MetaResult<RefreshResponse> {


BugenZhao

Rest LGTM

src/frontend/src/handler/refresh.rs

BugenZhao · 2025-07-28T05:57:22Z

src/frontend/src/handler/refresh.rs

+            // Refresh command sent successfully
+            tracing::info!(
+                table_id = %table_id,
+                table_name = %table_name,
+                "Manual refresh initiated"
+            );


Does it mean that we don't wait for REFRESH to be finished?

Yes. But in the future, we can make it a blocking command

src/frontend/src/handler/refresh.rs

src/meta/src/stream/refresh_manager.rs

src/sqlparser/src/ast/mod.rs

This was referenced Jul 16, 2025

feat: refreshable source, SourceExec part #22620

Merged

feat(stream): support RefreshableMaterialize with staging table #22621

Closed

xxchan force-pushed the xxchan/protective-jackal branch from 75e6f3a to 09dedc4 Compare July 22, 2025 05:37

github-actions bot added type/feature Type: New feature. ci/run-backwards-compat-tests Run backwards compatibility tests in your PR. ci/run-e2e-single-node-tests ci/run-e2e-test-other-backends labels Jul 22, 2025

xxchan force-pushed the xxchan/protective-jackal branch from 09dedc4 to 5b518e5 Compare July 22, 2025 06:42

xxchan changed the base branch from main to graphite-base/22619 July 22, 2025 08:35

xxchan force-pushed the xxchan/protective-jackal branch from 5b518e5 to a409585 Compare July 22, 2025 08:35

xxchan changed the base branch from graphite-base/22619 to 07-22-feat_minor_improve_risedev_d_in_cursorrule July 22, 2025 08:35

xxchan mentioned this pull request Jul 22, 2025

feat: minor improve risedev d in cursorrule #22689

Merged

8 tasks

xxchan commented Jul 22, 2025

View reviewed changes

Base automatically changed from 07-22-feat_minor_improve_risedev_d_in_cursorrule to main July 22, 2025 09:19

xxchan force-pushed the xxchan/protective-jackal branch from a409585 to 6f04b2b Compare July 22, 2025 10:07

xxchan mentioned this pull request Jul 22, 2025

feat: refreshable materialize with staging table #22695

Open

10 tasks

xxchan marked this pull request as ready for review July 23, 2025 02:10

cursor bot reviewed Jul 23, 2025

View reviewed changes

xxchan force-pushed the xxchan/protective-jackal branch 4 times, most recently from 553fc73 to 23ca770 Compare July 23, 2025 05:19

feat(frontend): add refreshable property & refresh command

03a7909

Signed-off-by: xxchan <[email protected]>

xxchan force-pushed the xxchan/protective-jackal branch from 23ca770 to 03a7909 Compare July 24, 2025 05:24

xxchan requested review from BugenZhao and chenzl25 July 25, 2025 06:21

chenzl25 reviewed Jul 25, 2025

View reviewed changes

chenzl25 approved these changes Jul 25, 2025

View reviewed changes

BugenZhao reviewed Jul 28, 2025

View reviewed changes

resolve bugen comments

a602d3b

xxchan enabled auto-merge July 28, 2025 07:12

xxchan added this pull request to the merge queue Jul 28, 2025

Merged via the queue into main with commit 1d0947a Jul 28, 2025
35 of 36 checks passed

xxchan deleted the xxchan/protective-jackal branch July 28, 2025 08:27

This was referenced Jul 28, 2025

feat: batch posix fs connector (__for_testing_only_batch_posix_fs) #22733

Merged

feat: support refreshable bigquery source #22744

Draft

xxchan mentioned this pull request Aug 4, 2025

Tracking issue for refreshable batch source #22690

Open

5 tasks

	pub fn parse_refresh(&mut self) -> ModalResult<Statement> {
	self.expect_keyword(Keyword::TABLE)?;
	let table_name = self.parse_object_name()?;
	Ok(Statement::Refresh { table_name })
	}

feat(frontend): add refreshable property & refresh command #22619

feat(frontend): add refreshable property & refresh command #22619

Uh oh!

Conversation

xxchan commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Uh oh!

xxchan commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

graphite-app bot commented Jul 23, 2025

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Refresh Syntax Parsing Error

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chenzl25 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BugenZhao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xxchan commented Jul 16, 2025 •

edited

Loading

xxchan commented Jul 16, 2025 •

edited

Loading