Streaming support #96
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces support for streaming workflows, adds new functionality for managing debug sessions, and improves the handling of execution flows in the StepFlow system. Key changes include adding a
Streaming
variant toFlowResult
, enhancing debug session management withtokio::sync::Mutex
, and updating theValueResolver
to handle streaming results.Streaming Workflow Support:
Streaming
variant to theFlowResult
enum to represent streaming data with associated metadata, chunk information, and a final chunk indicator (crates/stepflow-core/src/flow_result.rs
).ValueResolver
to pass through streaming results without applying skip actions and handle streaming results in objects and arrays (crates/stepflow-execution/src/value_resolver.rs
). [1] [2] [3]EvalComponent
to serialize streaming results into JSON format for external use (crates/stepflow-builtins/src/eval.rs
).Debug Session Management:
WorkflowExecutor
indebug_sessions
with anArc<tokio::sync::Mutex<WorkflowExecutor>>
to allow safe concurrent access and reuse of debug sessions (crates/stepflow-execution/src/executor.rs
). [1] [2]crates/stepflow-execution/src/executor.rs
). [1] [2] [3]Execution Flow Enhancements:
pending
executions and debug sessions to prevent memory leaks (crates/stepflow-execution/src/executor.rs
).Executor
trait forStepFlowExecutor
, enabling integration with the plugin system (crates/stepflow-execution/src/executor.rs
).Additional Improvements:
Clone
implementation forDependencyTracker
and other structs to support cloning where needed (crates/stepflow-analysis/src/tracker.rs
,crates/stepflow-core/src/workflow/step.rs
,crates/stepflow-execution/src/value_resolver.rs
). [1] [2] [3]streaming
flag in theStep
struct to indicate whether a step is a streaming step (crates/stepflow-core/src/workflow/step.rs
). [1] [2] [3]