diff --git a/docs/v2/examples/_meta.yaml b/docs/v2/examples/_meta.yaml new file mode 100644 index 00000000..1048fbb7 --- /dev/null +++ b/docs/v2/examples/_meta.yaml @@ -0,0 +1,3 @@ +label: Examples +order: 75 +collapsed: true diff --git a/docs/v2/examples/index.mdx b/docs/v2/examples/index.mdx new file mode 100644 index 00000000..392e6c68 --- /dev/null +++ b/docs/v2/examples/index.mdx @@ -0,0 +1,210 @@ +--- +title: Examples +--- + +The _Oura_ repository ships a collection of ready-to-run configurations under the +[`examples/`](https://github.com/txpipe/oura/tree/main/examples) directory. Each one is a +complete `daemon.toml` wiring a source, an optional filter chain, and a sink, so you can see +how the pieces fit together end-to-end rather than one component at a time. + +## Running an example + +Most examples reference companion files and relative paths, so run them **from inside their +own directory** rather than from the repository root: + +```sh +cd examples/ +oura daemon --config daemon.toml +``` + +Running from the repository root would scatter relative output and state (`./output`, +`./snapshot`, cursor and database files) in the wrong place and break the path-based +examples. + +Examples that target an optional sink or filter (`kafka`, `redis`, `rabbitmq`, `zeromq`, +`elasticsearch`, the AWS and GCP sinks, `SqlDb`, `WasmPlugin`, …) require _Oura_ to be built +with the matching cargo feature. See [Install from source](/oura/v2/installation/from_source) +for the feature flags, or use a Docker image that bundles them. + +## Companion files & setup + +Some examples need more than the `daemon.toml`. Check the example's folder before running: + +- **Backing service via `docker-compose`** — start it first from the example directory + (`docker compose up -d`): `kafka`, `redis`, `rabbitmq`, `elasticsearch`, `postgresql`, + `redis_cursor`, `n2c_source`. +- **Build step** — `wasm_basic` loads `./extract_fee/plugin.wasm`; the Go plugin under + `extract_fee/` must be built first (see its README). +- **Relative output/state created in the working directory** — `into_json`, `parse_cbor` and + `file_rotate` write to `./output/`; `mithril` downloads to `./snapshot/`; `file_cursor` + writes `my_cursor.json`; `sqlite` writes `./mydatabase.db` and ships an `init.sql` and a + `Makefile`. +- **Has its own README with extra steps** — `metadata_regex_filter`, `postgresql`, + `redis_cursor`, `wasm_basic`. + +> The `n2c_source` example sets `socket_path = "examples/n2c_source/node/node.socket"`, a path +> written relative to the **repository root** (unlike the others). Run it from the repo root or +> edit the path to point at your node socket. + +## Sources + +| Example | What it shows | Setup | +| :------ | :------------ | :---- | +| [`n2c_source`](https://github.com/txpipe/oura/tree/main/examples/n2c_source) | Node-to-Client source over a unix socket → Stdout | docker compose · run from repo root | +| [`hydra`](https://github.com/txpipe/oura/tree/main/examples/hydra) | Hydra head source over a WebSocket → Stdout | needs a running Hydra node | +| [`mithril`](https://github.com/txpipe/oura/tree/main/examples/mithril) | Mithril snapshot bootstrap with a `Breadcrumbs` intersect | downloads `./snapshot` | +| [`dolos_source`](https://github.com/txpipe/oura/tree/main/examples/dolos_source) | UTxO RPC (U5C) source against a Dolos/Demeter endpoint | needs a running Dolos (or a hosted U5C endpoint) | + +## Filters & pipelines + +| Example | What it shows | Setup | +| :------ | :------------ | :---- | +| [`select`](https://github.com/txpipe/oura/tree/main/examples/select) | `SplitBlock` → `ParseCbor` → `Select` by address → Stdout | — | +| [`metadata_regex_filter`](https://github.com/txpipe/oura/tree/main/examples/metadata_regex_filter) | `Select` matching a metadata label + regex (preprod) → Stdout | see README | +| [`parse_cbor`](https://github.com/txpipe/oura/tree/main/examples/parse_cbor) | `SplitBlock` → `ParseCbor` → JSONL file output | writes `./output` | +| [`into_json`](https://github.com/txpipe/oura/tree/main/examples/into_json) | `ParseCbor` → `IntoJson` → compressed JSONL file output | writes `./output` | +| [`wasm_basic`](https://github.com/txpipe/oura/tree/main/examples/wasm_basic) | Custom `WasmPlugin` filter loaded from a `.wasm` file | build step · `wasm` feature · README | + +## Sinks + +| Example | What it shows | Setup | +| :------ | :------------ | :---- | +| [`stdout`](https://github.com/txpipe/oura/tree/main/examples/stdout) | Legacy v1 events printed to standard output | — | +| [`webhook_basics`](https://github.com/txpipe/oura/tree/main/examples/webhook_basics) | `WebHook` sink posting events over HTTP | needs an HTTP endpoint | +| [`kafka`](https://github.com/txpipe/oura/tree/main/examples/kafka) | `Kafka` sink with `ByBlock` partitioning | docker compose · `kafka` feature | +| [`rabbitmq`](https://github.com/txpipe/oura/tree/main/examples/rabbitmq) | `Rabbitmq` sink | docker compose · `rabbitmq` feature | +| [`zeromq`](https://github.com/txpipe/oura/tree/main/examples/zeromq) | `Zeromq` PUSH socket sink | `zeromq` feature | +| [`elasticsearch`](https://github.com/txpipe/oura/tree/main/examples/elasticsearch) | `ElasticSearch` sink | docker compose · `elasticsearch` feature | +| [`redis`](https://github.com/txpipe/oura/tree/main/examples/redis) | `Redis` streams sink | docker compose · `redis` feature | +| [`sqlite`](https://github.com/txpipe/oura/tree/main/examples/sqlite) | `SqlDb` sink writing raw CBOR into SQLite | writes `./mydatabase.db` · `sql` feature | +| [`postgresql`](https://github.com/txpipe/oura/tree/main/examples/postgresql) | `SqlDb` sink targeting PostgreSQL | docker compose · README · `sql` feature | +| [`file_rotate`](https://github.com/txpipe/oura/tree/main/examples/file_rotate) | `FileRotate` sink writing rotated, compressed JSONL | writes `./output` | +| [`aws_lambda`](https://github.com/txpipe/oura/tree/main/examples/aws_lambda) | `AwsLambda` sink invoking a function per event | AWS credentials · `aws` feature | +| [`aws_s3`](https://github.com/txpipe/oura/tree/main/examples/aws_s3) | `AwsS3` sink writing objects | AWS credentials · `aws` feature | +| [`aws_sqs`](https://github.com/txpipe/oura/tree/main/examples/aws_sqs) | `AwsSqs` sink enqueueing messages | AWS credentials · `aws` feature | +| [`gcp_pubsub`](https://github.com/txpipe/oura/tree/main/examples/gcp_pubsub) | `GcpPubSub` sink publishing to a topic | GCP credentials · `gcp` feature | +| [`gcp_cloudfunction`](https://github.com/txpipe/oura/tree/main/examples/gcp_cloudfunction) | `GcpCloudFunction` sink calling a function | GCP credentials · `gcp` feature | +| [`assert`](https://github.com/txpipe/oura/tree/main/examples/assert) | `Assert` sink used for testing/validation | — | + +## Cursors + +| Example | What it shows | Setup | +| :------ | :------------ | :---- | +| [`file_cursor`](https://github.com/txpipe/oura/tree/main/examples/file_cursor) | Persisting pipeline progress to a `File` cursor | writes `my_cursor.json` | +| [`redis_cursor`](https://github.com/txpipe/oura/tree/main/examples/redis_cursor) | Persisting pipeline progress to a `Redis` cursor | docker compose · README · `redis` feature | + +## Observability + +| Example | What it shows | Setup | +| :------ | :------------ | :---- | +| [`metrics`](https://github.com/txpipe/oura/tree/main/examples/metrics) | Exposing a Prometheus metrics endpoint via the `[metrics]` block | — | + +## As a library + +The [`examples/lib`](https://github.com/txpipe/oura/tree/main/examples/lib) project shows how +to embed _Oura_ in a Rust application with custom stages — a custom filter plus a custom sink +that persists events into SQLite. See its README for the `sqlx-cli` setup, and the +[Library usage](/oura/v2/usage/library) page for the API overview. + +## Selected recipes + +A few complete pipelines, taken verbatim from the examples above. + +### Select transactions by address + +`SplitBlock` breaks each block into individual transactions, `ParseCbor` decodes them, and +`Select` keeps only the ones touching a given address. See the +[Select filter](/oura/v2/filters/select). + +```toml +[source] +type = "N2N" +peers = ["backbone.mainnet.cardanofoundation.org:3001"] + +[intersect] +type = "Point" +value = [37225013, "65b3d40e6114e05b662ddde737da63bbab05b86d476148614e82cde98462a6f5"] + +[[filters]] +type = "SplitBlock" + +[[filters]] +type = "ParseCbor" + +[[filters]] +type = "Select" +skip_uncertain = true +predicate = "addr1qx2fxv2umyhttkxyxp8x0dlpdt3k6cwng5pxj3jhsydzer3n0d3vllmyqwsx5wktcd8cc3sq835lu7drv2xwl2wywfgse35a3x" + +[sink] +type = "Stdout" +``` + +### Stream parsed transactions to JSONL files + +`IntoJson` turns the parsed records into JSON, and the +[FileRotate sink](/oura/v2/sinks/file_rotate) writes them to rotated, compressed JSONL files +under `./output/`. + +```toml +[source] +type = "N2N" +peers = ["backbone.mainnet.cardanofoundation.org:3001"] + +[intersect] +type = "Point" +value = [4493860, "ce7f821d2140419fea1a7900cf71b0c0a0e94afbb1f814a6717cff071c3b6afc"] + +[[filters]] +type = "SplitBlock" + +[[filters]] +type = "ParseCbor" + +[[filters]] +type = "IntoJson" + +[sink] +type = "FileRotate" +max_total_files = 5 +output_format = "JSONL" +output_path = "./output/logs.jsonl" +max_bytes_per_file = 5_000_000 +compress_files = true +``` + +### Match a metadata label with a regex + +A [Select filter](/oura/v2/filters/select) predicate can match on transaction metadata — here, +label `674` whose text matches a regular expression. + +```toml +[chain] +type = "preprod" + +[source] +type = "N2N" +peers = ["preprod-node.world.dev.cardano.org:30000"] + +[intersect] +type = "Tip" + +[[filters]] +type = "SplitBlock" + +[[filters]] +type = "ParseCbor" + +[[filters]] +type = "Select" +skip_uncertain = false + +[filters.predicate.match.metadata] +label = 674 + +[filters.predicate.match.metadata.value.text] +regex = "Hello World" + +[sink] +type = "Stdout" +``` diff --git a/docs/v2/index.mdx b/docs/v2/index.mdx index 852b3c28..669dd64a 100644 --- a/docs/v2/index.mdx +++ b/docs/v2/index.mdx @@ -40,5 +40,9 @@ _Oura_ running in `daemon` mode can be configured to use custom filters to pinpo If the available out-of-the-box features don't satisfy your particular use-case, _Oura_ can be used a library in your Rust project to set up tailor-made pipelines. Each component (sources, filters, sinks, etc) in _Oura_ aims at being self-contained and reusable. For example, custom filters and sinks can be built while reusing the existing sources. +## Examples + +For ready-to-run configurations covering every source, filter and sink — plus end-to-end pipeline recipes — see the [Examples](/oura/v2/examples) section. + ## (Experimental) Windows Support _Oura_ Windows support is currently __experimental__, Windows build supports only [Node-to-Node](/oura/v2/sources/n2n) source with tcp socket bearer. diff --git a/examples/assert/daemon.toml b/examples/assert/daemon.toml index 9ab28852..c5c90174 100644 --- a/examples/assert/daemon.toml +++ b/examples/assert/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Tip" diff --git a/examples/aws_lambda/daemon.toml b/examples/aws_lambda/daemon.toml index 08d6171d..a0493fef 100644 --- a/examples/aws_lambda/daemon.toml +++ b/examples/aws_lambda/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/aws_s3/daemon.toml b/examples/aws_s3/daemon.toml index f1f4dd8d..7a6d3b53 100644 --- a/examples/aws_s3/daemon.toml +++ b/examples/aws_s3/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/aws_sqs/daemon.toml b/examples/aws_sqs/daemon.toml index d1a21219..635dfa95 100644 --- a/examples/aws_sqs/daemon.toml +++ b/examples/aws_sqs/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/dolos_source/daemon.toml b/examples/dolos_source/daemon.toml index 35f1a4d0..7d6f3266 100644 --- a/examples/dolos_source/daemon.toml +++ b/examples/dolos_source/daemon.toml @@ -1,6 +1,10 @@ [source] type = "U5C" -url = "https://50051-romantic-calmness-b55bqg.us1.demeter.run" +url = "http://localhost:50051" +# Extra gRPC headers sent with every request. Required field (use an empty table for a +# local, unauthenticated Dolos). For a hosted endpoint such as Demeter, pass the API key: +# metadata = { "dmtr-api-key" = "dmtr_..." } +metadata = {} [intersect] type = "Tip" diff --git a/examples/elasticsearch/daemon.toml b/examples/elasticsearch/daemon.toml index 73416b7d..bc27c8cc 100644 --- a/examples/elasticsearch/daemon.toml +++ b/examples/elasticsearch/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/file_cursor/daemon.toml b/examples/file_cursor/daemon.toml index b995a77f..9bc8d898 100644 --- a/examples/file_cursor/daemon.toml +++ b/examples/file_cursor/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/file_rotate/daemon.toml b/examples/file_rotate/daemon.toml index e9a3685e..48251013 100644 --- a/examples/file_rotate/daemon.toml +++ b/examples/file_rotate/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/gcp_cloudfunction/daemon.toml b/examples/gcp_cloudfunction/daemon.toml index 96cde347..b8913166 100644 --- a/examples/gcp_cloudfunction/daemon.toml +++ b/examples/gcp_cloudfunction/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Tip" diff --git a/examples/gcp_pubsub/daemon.toml b/examples/gcp_pubsub/daemon.toml index 2eb64c81..5bd86c14 100644 --- a/examples/gcp_pubsub/daemon.toml +++ b/examples/gcp_pubsub/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/into_json/daemon.toml b/examples/into_json/daemon.toml index 415407ec..debdc6a1 100644 --- a/examples/into_json/daemon.toml +++ b/examples/into_json/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/kafka/daemon.toml b/examples/kafka/daemon.toml index dc4574cb..fa90ca7a 100644 --- a/examples/kafka/daemon.toml +++ b/examples/kafka/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/metrics/daemon.toml b/examples/metrics/daemon.toml index 347dd8d9..9ba21541 100644 --- a/examples/metrics/daemon.toml +++ b/examples/metrics/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/parse_cbor/daemon.toml b/examples/parse_cbor/daemon.toml index 4c518ca3..61806fc2 100644 --- a/examples/parse_cbor/daemon.toml +++ b/examples/parse_cbor/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/postgresql/daemon.toml b/examples/postgresql/daemon.toml index 6be9568b..8715593e 100644 --- a/examples/postgresql/daemon.toml +++ b/examples/postgresql/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/rabbitmq/daemon.toml b/examples/rabbitmq/daemon.toml index 3b0846b1..262fa2dd 100644 --- a/examples/rabbitmq/daemon.toml +++ b/examples/rabbitmq/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/redis/daemon.toml b/examples/redis/daemon.toml index b1383386..ee1979e3 100644 --- a/examples/redis/daemon.toml +++ b/examples/redis/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/examples/select/daemon.toml b/examples/select/daemon.toml index 2a6ee5eb..15594596 100644 --- a/examples/select/daemon.toml +++ b/examples/select/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/stdout/daemon.toml b/examples/stdout/daemon.toml index b15b8232..dabf6f5c 100644 --- a/examples/stdout/daemon.toml +++ b/examples/stdout/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/wasm_basic/daemon.toml b/examples/wasm_basic/daemon.toml index b4d83252..a2733ae8 100644 --- a/examples/wasm_basic/daemon.toml +++ b/examples/wasm_basic/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/webhook_basics/daemon.toml b/examples/webhook_basics/daemon.toml index 200410e3..23ce518b 100644 --- a/examples/webhook_basics/daemon.toml +++ b/examples/webhook_basics/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [intersect] type = "Point" diff --git a/examples/zeromq/daemon.toml b/examples/zeromq/daemon.toml index 4066ea90..2c207d49 100644 --- a/examples/zeromq/daemon.toml +++ b/examples/zeromq/daemon.toml @@ -1,6 +1,6 @@ [source] type = "N2N" -peers = ["relays-new.cardano-mainnet.iohk.io:3001"] +peers = ["backbone.mainnet.cardanofoundation.org:3001"] [chain] type = "mainnet" diff --git a/src/sources/hydra.rs b/src/sources/hydra.rs index 7a2a2d44..37f16e9c 100644 --- a/src/sources/hydra.rs +++ b/src/sources/hydra.rs @@ -220,7 +220,12 @@ impl gasket::framework::Worker for Worker { debug!("connecting to hydra WebSocket"); let url = &stage.config.ws_url; - let (socket, _) = connect_async(url).await.expect("Can't connect"); + let (socket, _) = connect_async(url) + .await + .inspect_err(|err| { + error!(%err, %url, "failed to connect to hydra WebSocket"); + }) + .or_restart()?; let worker = Self { socket, intersect: intersect_from_config(&stage.intersect),