Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 16 additions & 5 deletions docs/v2/advanced/finalize_options.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,34 @@
title: Finalize Options
---

Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can also instruct oura to stop syncing when it reaches a specific block hash by defining a `[finalize]` config.
Advanced options for instructing Oura to stop on a chain point. If you'd like it to only sync a specific section of the chain, you can instruct Oura to stop syncing once it reaches a specific block hash, slot, or block count.

There is no top-level `[finalize]` block in v2. Finalization is configured through the [Work Stats filter](/oura/v2/filters/work_stats), which counts blocks as they pass through and stops the pipeline once a configured condition is met.

## Configuration

To modify the default behavior the daemon mode uses, a section named `[finalize]` needs to be added to the `daemon.toml` file.
Add a `WorkStats` filter to the `[[filters]]` chain in your `daemon.toml` and set any of the stopping conditions:

```toml
[finalize]
[[filters]]
type = "WorkStats"
until_hash = <BlockHash>
max_block_slot = <SlotNumber>
max_block_quantity = <BlockCount>
```

- `until_hash` (optional): stop once the block with this hash has been processed.
- `max_block_slot` (optional): stop once a block at or beyond this slot number has been processed.
- `max_block_quantity` (optional): stop after this many blocks have been processed.

The pipeline stops as soon as any configured condition is met.

## Examples

The following example show how to configure Oura to stop sync on Byron era
The following example shows how to configure Oura to stop syncing on the Byron era

```toml
[finalize]
[[filters]]
type = "WorkStats"
until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
```
2 changes: 1 addition & 1 deletion docs/v2/advanced/intersect_options.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ When running in daemon mode, Oura provides 4 different strategies for finding th

The default strategy use by Oura is `Tip`, unless an alternative option is specified via configuration.

You can also define a finalizing point by providing a block hash at which oura will stop reading from the the chain and exit gracefully.
You can also have Oura stop reading from the chain and exit gracefully once it reaches a given block hash, slot, or block count. This is configured through the [Work Stats filter](/oura/v2/filters/work_stats); see [finalize options](/oura/v2/advanced/finalize_options) for details.

## Configuration

Expand Down
4 changes: 1 addition & 3 deletions docs/v2/advanced/pipeline_metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,12 +33,10 @@ The _metrics_ feature is a configurable setting available when running in daemon

[metrics]
address = "0.0.0.0:9186"
endpoint = "/metrics"
```

- `[metrics]` section needs to be present to enable the feature. Absence of the section will not expose any HTTP endpoints.
- `address`: The address at which the HTTP server will be listening for request. Expected format is `<ip>:<port>`. Use the IP value `0.0.0.0` to allow connections on any of the available IP address of the network interface. Default value is `0.0.0.0:9186`.
- `endpoint`: The path at which the metrics will be exposed. Default value is `/metrics`.
- `address` (optional): The address at which the HTTP server will be listening for request. Expected format is `<ip>:<port>`. Use the IP value `0.0.0.0` to allow connections on any of the available IP address of the network interface. Default value is `0.0.0.0:9186`.

## Usage

Expand Down
46 changes: 41 additions & 5 deletions docs/v2/advanced/stateful_cursor.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ Although valid, the workaround described above is very inefficient. If the fixed

## Feature

Oura implements an optional stateful cursor that receives notifications from the sink stage of the pipeline to continuously track the current position of the chain. At certain checkpoints (every 10 secs by default), the position is persisted onto the file system at a configurable location.
Oura implements an optional stateful cursor that receives notifications from the sink stage of the pipeline to continuously track the current position of the chain. At certain checkpoints (every 10 secs by default), the position is persisted onto the configured backend.

Assuming that a restart occurs and the cursor feature is enabled, the process will attempt to locate and load the persisted value and instruct the source stage to begin reading chain data from the last known position.
Assuming that a restart occurs and the cursor feature is enabled, the process will attempt to locate and load the persisted value and instruct the source stage to begin reading chain data from the last known position.

## Configuration

Expand All @@ -27,9 +27,45 @@ The _cursor_ feature is a configurable setting available when running in daemon
```toml
[cursor]
type = "File"
path = "/var/oura/cursor"
path = "./cursor.json"
max_breadcrumbs = 10
flush_interval = 10
```

- `[cursor]`: The presence of this section in the toml file indicates Oura to enable the _cursor_ feature.
- `type`: The type of persistence backend to use for storing the state of the cursor. The only available option at the moment is `File`, which stores the cursor in the file system.
- `path`: The location of the cursor file within the file system. Default value is `var/oura/cursor`.
- `type`: The type of persistence backend to use for storing the state of the cursor. Available options are `Memory` (the default if no `[cursor]` section is present), `File`, and `Redis`.

### Backend: `Memory`

Keeps the cursor breadcrumbs in memory only; the position is **not** persisted across restarts. This is the default behavior when no `[cursor]` section is configured. It takes no extra fields:

```toml
[cursor]
type = "Memory"
```

### Backend: `File`

Stores the cursor in a JSON file on the local file system.

- `path` (optional): The location of the cursor file within the file system. Default value is `./cursor.json` (relative to the working directory).
- `max_breadcrumbs` (optional): The number of recent chain positions to retain (used to recover across rollbacks). Default value is `10`.
- `flush_interval` (optional): How often, in seconds, the position is flushed to disk. Default value is `10`.

### Backend: `Redis`

Stores the cursor in a Redis instance (requires the `redis` feature).

```toml
[cursor]
type = "Redis"
url = "redis://localhost:6379"
key = "oura-cursor"
max_breadcrumbs = 10
flush_interval = 10
```

- `url`: The Redis connection URL.
- `key`: The Redis key under which the cursor state is stored.
- `max_breadcrumbs` (optional): The number of recent chain positions to retain. Default value is `10`.
- `flush_interval` (optional): How often, in seconds, the position is flushed to Redis. Default value is `10`.
20 changes: 20 additions & 0 deletions docs/v2/filters/into_json.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
---
title: Into JSON filter
sidebar:
label: Into JSON
---

The `into_json` filter converts each record flowing through the pipeline into a generic JSON record. It is useful when a downstream sink expects JSON regardless of the original record type (raw CBOR, parsed transaction/block, or a legacy v1 event).

The JSON produced mirrors the serde representation of the incoming record, so the exact shape depends on the filters placed before it (for example, running `ParseCbor` first yields the UTxO RPC parsed representation, while `LegacyV1` yields the v1 event schema).

## Configuration

Adding the following section to the daemon config file will enable the filter as part of the pipeline:

```toml
[[filters]]
type = "IntoJson"
```

This filter takes no additional options.
26 changes: 26 additions & 0 deletions docs/v2/filters/work_stats.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
title: Work Stats filter
sidebar:
label: Work Stats
---

The `work_stats` filter tracks pipeline progress as events pass through it and can optionally finalize (stop) the pipeline once a configured stopping point is reached. It passes events through unchanged, so it can be placed anywhere in the filter chain.

This filter is how finalization is configured in v2 (see [finalize options](/oura/v2/advanced/finalize_options) for the conceptual overview): the pipeline stops as soon as any configured condition is met.

## Configuration

```toml
[[filters]]
type = "WorkStats"
until_hash = "aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de"
max_block_slot = 1000000
max_block_quantity = 5000
```

- `type`: the literal value `WorkStats`.
- `until_hash` (optional): stop once the block with this hash has been processed.
- `max_block_slot` (optional): stop once a block at or beyond this slot number has been processed.
- `max_block_quantity` (optional): stop after this many blocks have been processed.

All options are optional; if none are set, the filter only tracks statistics and never stops the pipeline.
21 changes: 18 additions & 3 deletions docs/v2/installation/docker.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,27 @@ docker run -d -v $(pwd)/daemon.toml:/etc/oura/daemon.toml \
ghcr.io/txpipe/oura:latest daemon
```

## Available Tags

Images are published to `ghcr.io/txpipe/oura` under several tags, each following a different update trigger (see the _Updated on_ column). Pick the one that matches how closely you want to track new changes:

| Tag | Example | Updated on | Description |
| :----------------- | :--------------- | :--------------------- | :---------------------------------------------------------------------- |
| `latest` | `latest` | every push to `main` | Tracks the tip of the `main` branch (development builds, not a release).|
| `stable` | `stable` | every release | Tracks the most recent released version. |
| `v{major}` | `v2` | every release | Latest release within a major version. |
| `v{major}.{minor}` | `v2.1` | every release | Latest release within a minor version. |
| `v{version}` | `v2.1.0` | its matching release | A specific, immutable release. |
| `sha-{commit}` | `sha-2dec504` | every build | A specific commit; fully immutable. |

## Versioned Images

Images are also tagged with the corresponding version number. It is highly recommended to use a fixed image version in production environments to avoid the effects of new features being included in each release (please remember Oura hasn't reached v2 stability guarantees).
It is highly recommended to use a fixed image version in production environments to avoid the effects of new features being included in each release.

To use a versioned image, replace the `latest` tag by the desired version with the `v` prefix. For example, to use version `v2.0.0-alpha.1`, use the following image:
To use a versioned image, replace the `latest` tag by the desired version with the `v` prefix. For example, to pin the `v2.1.0` release, use the following image:

```
ghcr.io/txpipe/oura:v2.0.0-alpha.1
ghcr.io/txpipe/oura:v2.1.0
```

You can also track a major or minor line (for example `v2` or `v2.1`) to automatically pick up compatible updates, or use `stable` to always run the most recent release. Note that `latest` follows the `main` branch rather than the latest release, so avoid it in production.
32 changes: 31 additions & 1 deletion docs/v2/reference/data_dictionary.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,39 @@ _Oura_ follows a Cardano chain and outputs events. Each event contains data abou

A consumer aggregating a sequence of multiple events will notice redundant / duplicated data. For example, the "block number" value will appear repeated in the context of every event of the same block. This behavior is intended, making each event a self-contained record is an architectural decision. We favor "consumption ergonomics" over "data normalization".

## Pipeline Data Model

Internally, every unit that flows through an Oura v2 pipeline is a `ChainEvent` that wraps a chain `point` and (except for resets) a `Record`. When a sink serializes an event to JSON, it uses the following envelope:

```json
{
"event": "apply",
"point": { "slot": 0, "hash": "" },
"record": { }
}
```

- `event`: the kind of chain event — one of:
- `apply`: a block/tx was applied to the chain.
- `undo`: a block/tx was rolled back.
- `reset`: the pipeline reset to the given `point` (carries no `record`).
- `point`: the chain position, with `slot` (number) and `hash` (hex string). The chain origin is represented as a null/empty point.
- `record`: the payload. Its shape depends on which filters ran before the sink (see below).

The `record` is one of the following variants, determined by the source and the filters in the pipeline:

| Record | Produced by | Shape |
| :----------- | :------------------------------------------- | :------------------------------------------------------------------- |
| CBOR block | sources, by default | `{ "hex": "<cbor-hex>" }` — the raw block CBOR. |
| CBOR tx | [Split Block](/oura/v2/filters/split_block) | `{ "hex": "<cbor-hex>" }` — the raw transaction CBOR. |
| Parsed block | [ParseCbor](/oura/v2/filters/parse_cbor) | a [UTxORPC](https://utxorpc.org/cardano) `Block`. |
| Parsed tx | [ParseCbor](/oura/v2/filters/parse_cbor) | a [UTxORPC](https://utxorpc.org/cardano) `Tx`. |
| Generic JSON | [Into JSON](/oura/v2/filters/into_json) | an arbitrary JSON value. |
| Legacy v1 | [LegacyV1](/oura/v2/filters/legacy_v1) | a v1-style event (see _Available Events (Legacy V1 Filter)_ below). |

## Available Events (ParseCbor Filter)

When the filter [ParseCbor](/oura/v2/filters/parse_cbor) is enabled, the events will use the [UTxORPC](https://utxorpc.org/cardano) data structures.
When the [ParseCbor](/oura/v2/filters/parse_cbor) filter is enabled, records use the [UTxORPC](https://utxorpc.org/cardano) data structures: a `CborBlock` becomes a parsed `Block` and a `CborTx` becomes a parsed `Tx`. Refer to the [UTxORPC Cardano spec](https://utxorpc.org/cardano) for the complete field-level schema of those structures.

## Available Events (Legacy V1 Filter)

Expand Down
4 changes: 2 additions & 2 deletions docs/v2/sinks/elasticsearch.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ A sink that outputs events into an Elasticsearch server. Each event is json-enco

```toml
[sink]
type = "Elastic"
type = "ElasticSearch"
url = "https://localhost:9200"
index = "oura.sink.v0.mainnet"

Expand All @@ -20,7 +20,7 @@ password = "my very secret stuff"

### Section: `sink`

- `type`: the literal value `Elastic`.
- `type`: the literal value `ElasticSearch`.
- `url`: the location of the Elasticsearch's API
- `index`: the name of the index (or data stream) to store the event documents
- `idempotency` (optional): flag that if enabled will force idempotent calls to ES (to avoid duplicates)
Expand Down
4 changes: 2 additions & 2 deletions docs/v2/sinks/gcp_cloudfunction.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ A sink that sends each event to a cloud function. Each event is json-encoded and
type = "GcpCloudFunction"
url = "https://REGION-PROJECT_ID.cloudfunctions.net/FUNCTION_NAME"
timeout = 30000
authorization = true
authentication = true

[sink.headers]
extra_header_1 = "abc"
Expand All @@ -25,7 +25,7 @@ extra_header_2 = "123"
- `type`: the literal value `GcpCloudFunction`
- `url`: Your function url
- `timeout` (optional): the timeout value for the HTTP response in milliseconds. Default value is `30000`.
- `authorization` (optional): value to add as the 'Authorization' HTTP header
- `authentication` (optional): a flag that, when set to `true`, attaches a GCP identity token as the 'Authorization' HTTP header (see _GCP Authentication_ below). Default value is `false`.
- `headers` (optional): key-value map of extra headers to pass in each HTTP call

### GCP Authentication
Expand Down
4 changes: 4 additions & 0 deletions docs/v2/sinks/kafka.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@ A sink that implements a _Kafka_ producer. Each event is json-encoded and sent a
type = "Kafka"
brokers = ["kafka-broker-0:9092"]
topic = "cardano-events"
ack_timeout_secs = 5
paritioning = "Random"
```

- `type`: the literal value `Kafka`.
- `brokers`: indicates the location of the _Kafka_ brokers within the network. Several hostname:port pairs can be added to the array for a "cluster" scenario.
- `topic` this field indicates which _Kafka_ topic to use to send the outbound messages.
- `ack_timeout_secs` (optional): the timeout, in seconds, to wait for the broker to acknowledge a produced message.
- `paritioning` (optional): the strategy used to assign each message to a partition. Available values are `ByBlock` (messages from the same block share a partition key) and `Random`. Default value is `Random`. (Note: the config key is spelled `paritioning` to match the field name in the codebase.)
4 changes: 3 additions & 1 deletion docs/v2/sinks/redis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,15 @@ Example configuration that sends all events into a single stream named `mystream
type = "Redis"
url = "redis://localhost:6379"
stream_name = "mystream"
stream_max_length = 100000
```

### Section: `sink`

- `type`: the literal value `Redis`.
- `url`: the redis server in the format `redis://[<username>][:<password>]@<hostname>[:port][/<db>]`
- `stream_name` : the name of the redis stream for StreamStrategy `None`, default is "oura" if not specified
- `stream_name` (optional): the name of the redis stream, default is `oura-sink` if not specified
- `stream_max_length` (optional): if set, caps the stream to this many entries (using `XADD ... MAXLEN`), discarding the oldest entries as new ones arrive. If omitted, the stream grows unbounded.

## Conventions

Expand Down
41 changes: 41 additions & 0 deletions docs/v2/sinks/sql_db.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: SQL Database
sidebar:
label: SQL Database
---

A sink that executes a SQL statement against a relational database for each event. It is built on top of [sqlx](https://github.com/launchbadge/sqlx)'s `Any` driver, so it can target SQLite, PostgreSQL or MySQL depending on the connection string.

For every event, the sink renders a [Handlebars](https://handlebarsjs.com/) template into a SQL statement and executes it. There are three independent templates, one for each kind of chain event:

- `apply`: rendered when a block/tx is applied to the chain.
- `undo`: rendered when a block/tx is rolled back.
- `reset`: rendered when the pipeline resets to a given point (no record is available, only the point).

This sink requires Oura to be built with the `sql` feature.

## Configuration

```toml
[sink]
type = "SqlDb"
connection = "sqlite::memory:"
apply_template = "INSERT INTO events (slot, data) VALUES ({{point.slot}}, '{{record}}')"
undo_template = "DELETE FROM events WHERE slot = {{point.slot}}"
reset_template = "DELETE FROM events WHERE slot > {{point.slot}}"
```

### Section: `sink`

- `type`: the literal value `SqlDb`.
- `connection`: the database connection string passed to sqlx (e.g. `sqlite::memory:`, `sqlite://./oura.db`, `postgres://user:pass@host/db`).
- `apply_template`: a Handlebars template rendered to a SQL statement for `apply` events.
- `undo_template`: a Handlebars template rendered to a SQL statement for `undo` events.
- `reset_template`: a Handlebars template rendered to a SQL statement for `reset` events.

## Template Data

Each template is rendered with the following data context:

- `point`: the chain point. For a specific point this is an object with `slot` (number) and `hash` (hex string). For the chain origin it is `null`.
- `record`: the event payload. For `reset` events this value is `null`. The shape for non-reset events depends on the record type produced by the preceding filters (e.g. raw CBOR hex, a parsed transaction, or a legacy v1 event).
1 change: 1 addition & 0 deletions docs/v2/sinks/terminal.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,4 @@ wrap = true
- `type` (required): the literal value `Terminal`.
- `throttle_min_span_millis` (optional, default = `500`): the amount of time (milliseconds) to wait between printing each event into the console. This is used to facilitate the reading for human following the output.
- `wrap` (optional, default = `false`): a true value indicates that long output text should break and continue in the following line. If false, lines will be truncated to fit in the available terminal width.
- `adahandle_policy` (optional): the minting policy id (hex) of [ADA Handle](https://adahandle.com/). When set, the sink resolves and displays handles found in outputs for a friendlier rendering. If omitted, handle resolution is disabled.
Loading
Loading