Skip to content

Commit 0502a7c

Browse files
authored
feat(sql): migrate to DataFusion-based streaming SQL (#219)
<!-- CURSOR_SUMMARY --> > [!NOTE] > **High Risk** > Large dependency and planning/execution refactor: introduces DataFusion/Arrow/DataFusion-based streaming SQL planning plus persistent stream table catalog and job submission. Risk comes from new DDL paths (`CREATE STREAMING TABLE`, connector-backed `CREATE TABLE`, `DROP TABLE`) and major crate/version bumps (Arrow 55/DataFusion git forks, `bincode` v2) affecting runtime behavior and serialization. > > **Overview** > Switches the SQL stack to a **DataFusion-based streaming planner**: adds compilation support for `CREATE STREAMING TABLE ... AS SELECT` (including connector options like `connector`/`partition_by`) and connector-backed `CREATE TABLE ... WITH ('connector'=...)`, plus `DROP TABLE` planning. > > Wires these new plan nodes through coordinator execution by introducing a `CoordinatorRuntimeContext` (task manager + stream catalog + job manager), persisting source/sink definitions to the stream catalog, and submitting streaming jobs when creating a streaming sink. > > Expands the `protocol` crate with new protobuf APIs (`fs_api.proto`, `storage.proto`) and build output (serde-derived types + descriptor set), adds a new Arrow-backed `FsSchema` type, and performs a **major dependency refresh** (Arrow 55, git-pinned DataFusion/Arrow/parquet/sqlparser/typify, `bincode` 2 + new supporting crates) reflected in `Cargo.toml` and `Cargo.lock`. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 7842995. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 6ab4638 commit 0502a7c

278 files changed

Lines changed: 36390 additions & 1577 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/verify-package.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ jobs:
4949
sudo apt-get install -y --no-install-recommends \
5050
cmake \
5151
libssl-dev \
52+
libcurl4-openssl-dev \
5253
pkg-config \
5354
libsasl2-dev \
5455
protobuf-compiler

0 commit comments

Comments
 (0)