Skip to content

Commit d75573f

Browse files
luoluoyuyuluoluoyuyu
committed
update README.md
1 parent dd402cc commit d75573f

1 file changed

Lines changed: 157 additions & 32 deletions

File tree

‎README.md‎

Lines changed: 157 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,158 @@
1-
# Rust Function Stream
2-
3-
Rust Function Stream is a stream processing framework built with Rust.
4-
5-
## Project Structure
6-
7-
```
8-
rust-function-stream/
9-
├── src/
10-
│ ├── client/
11-
│ │ ├── mod.rs # Client module entry
12-
│ │ └── api.rs # Client API implementation
13-
│ ├── config/
14-
│ │ ├── mod.rs # Configuration module entry and exports
15-
│ │ ├── types.rs # Global configuration type definitions and registry
16-
│ │ └── loader.rs # YAML reading functions
17-
│ ├── server/
18-
│ │ ├── mod.rs # Server module entry
19-
│ │ ├── connection_manager.rs # Connection state manager
20-
│ │ ├── session_manager.rs # Session manager
21-
│ │ └── service.rs # gRPC service implementation
22-
│ ├── lib.rs # Library entry
23-
│ └── main.rs # Demo program
24-
├── protocol/ # Pure protocol definition package
25-
│ ├── proto/
26-
│ │ └── function_stream.proto # Protocol Buffers definitions
27-
│ ├── src/
28-
│ │ ├── lib.rs # Protocol exports
29-
│ │ └── function_stream.rs # Generated message code
30-
│ └── Cargo.toml # Protocol package configuration
31-
├── Cargo.toml # Main project configuration
32-
└── README.md # Documentation
1+
# Function Stream
2+
3+
**Function Stream** is a high-performance, event-driven stream processing framework built in Rust. It provides a modular runtime to orchestrate serverless-style processing functions compiled to **WebAssembly (WASM)**, supporting functions written in **Go, Python, and Rust**.
4+
5+
6+
7+
## Key Features
8+
9+
* **Event-Driven WASM Runtime**: Executes polyglot functions (Go, Python, Rust) with near-native performance and sandboxed isolation.
10+
* **Durable State Management**: Built-in support for RocksDB-backed state stores for stateful stream processing.
11+
* **SQL-Powered CLI**: Interactive REPL for job management and stream inspection using SQL-like commands.
12+
13+
## Repository Layout
14+
15+
```
16+
function-stream/
17+
├── src/ # Core runtime, coordinator, server, config, SQL parser
18+
├── protocol/ # Protocol Buffers definitions and generated gRPC code
19+
├── cli/cli/ # SQL REPL client
20+
├── conf/ # Default runtime configuration
21+
├── examples/ # Sample processors and integration examples
22+
├── python/ # Python API, client, and runtime WASM generator
23+
├── Makefile # Build and packaging automation
24+
└── Cargo.toml # Workspace manifest
25+
```
26+
27+
## Prerequisites
28+
29+
- Rust toolchain (recommend `rustup` with stable >= 1.77)
30+
- `protoc` (Protocol Buffers compiler) for generating gRPC bindings
31+
- Build tooling for `rdkafka`: `cmake`, `pkg-config`, and OpenSSL headers
32+
- Optional (for Python processors and full package):
33+
- Python 3.9+ with `venv`
34+
- `componentize-py` (installed via `make install-deps` inside `python/functionstream-runtime`)
35+
36+
## Build From Source
37+
38+
Clone the repository and install dependencies:
39+
40+
```
41+
git clone https://github.com/<your-org>/function-stream.git
42+
cd function-stream
43+
cargo fetch
44+
```
45+
46+
Build targets:
47+
48+
- Debug build (fast iteration):
49+
```
50+
cargo build
51+
```
52+
- Release build with Python support (default features):
53+
```
54+
cargo build --release
55+
```
56+
- Release build without Python (lite):
57+
```
58+
cargo build --release --no-default-features --features incremental-cache
59+
```
60+
61+
To regenerate Protocol Buffers, rerun the build; `tonic-build` automatically recompiles when `protocol/proto/function_stream.proto` changes.
62+
63+
## Run Locally
64+
65+
### Prepare configuration
66+
67+
1. Review `conf/config.yaml` and adjust service host/port, logging, state storage, and Python runtime settings as needed.
68+
2. Optionally point `FUNCTION_STREAM_CONF` to a custom configuration file or directory:
69+
```
70+
export FUNCTION_STREAM_CONF=/path/to/config.yaml
71+
```
72+
3. Ensure `data/` and `logs/` directories are writable (they are created automatically on startup).
73+
74+
### Start the control plane
75+
76+
```
77+
cargo run --release --bin function-stream
78+
```
79+
80+
The server logs appear under `logs/app.log` (default JSON format). Stop the service with `Ctrl+C`.
81+
82+
### Use the SQL CLI
83+
84+
Run the interactive CLI from another terminal to issue SQL statements:
85+
3386
```
87+
cargo run -p function-stream-cli -- --ip 127.0.0.1 --port 8080
88+
```
89+
90+
The CLI connects to the gRPC endpoint exposed by the server.
91+
92+
### Try sample processors
93+
94+
- Python processor example:
95+
```
96+
cd examples/python-processor
97+
python main.py
98+
```
99+
- Kafka integration tests and Go processor examples are under `examples/`.
100+
101+
## Packaging
102+
103+
Packaging is orchestrated by the top-level `Makefile`. Outputs are placed in `dist/`.
104+
105+
### Full distribution (includes Python runtime)
106+
107+
1. Create the virtual environment once:
108+
```
109+
python3 -m venv .venv
110+
source .venv/bin/activate
111+
make -C python/functionstream-runtime install-deps build
112+
deactivate
113+
```
114+
2. Build and package:
115+
```
116+
make package-full
117+
```
118+
119+
Artifacts:
120+
- `dist/function-stream-<version>/` directory containing binaries, config, logs/data skeletons, README, and Python WASM runtime.
121+
- `dist/packages/function-stream-<version>.zip` and `.tar.gz`.
122+
123+
### Lite distribution (Rust-only)
124+
125+
```
126+
make package-lite
127+
```
128+
129+
Artifacts:
130+
- `dist/function-stream-<version>-lite/` directory with binaries and configs.
131+
- `dist/packages/function-stream-<version>-lite.zip` and `.tar.gz`.
132+
133+
### Package all variants
134+
135+
```
136+
make package-all
137+
```
138+
139+
The script cleans previous `dist/` contents, produces both full and lite packages, and lists generated archives.
140+
141+
## Testing
142+
143+
Run the Rust test suite:
144+
145+
```
146+
cargo test
147+
```
148+
149+
Python components expose their own `Makefile` targets (for example, `make -C python/functionstream-runtime test` if defined).
150+
151+
## Environment Variables
152+
153+
- `FUNCTION_STREAM_HOME`: Overrides the project root for resolving data/log directories.
154+
- `FUNCTION_STREAM_CONF`: Points to a configuration file or directory containing `config.yaml`.
155+
156+
## License
157+
158+
Licensed under the Apache License, Version 2.0. See `LICENSE` for details.

0 commit comments

Comments
 (0)