The future of asset freshness in Dagster

[anuthebananu]

In Dagster 1.11, we are introducing a preview of FreshnessPolicy, a new way to declaratively specify rules for asset freshness. Freshness policies are a first-class property of assets, offer a natural API for expressing freshness rules and applying them to individual assets and asset selections, and have dedicated treatment throughout the Dagster UI. These policies are automatically evaluated by Dagster and don’t require users to manually schedule evaluations.

Over the coming months, we’ll be adding capabilities and making improvements to freshness policies, with the goal of making them the primary way of representing asset freshness in Dagster.

As a result, we’re making some changes to the existing freshness APIs in Dagster, which are outlined below.

Renaming existing FreshnessPolicy to LegacyFreshnessPolicy

If you’re a long-time Dagster user, the name “freshness policy” might be familiar to you. This is because there is an existing API also named FreshnessPolicy . This API has been marked as deprecated since version 1.6 as we found it unsuitable and unintuitive for tackling many common freshness use cases.

In 1.11, FreshnessPolicy has been renamed to LegacyFreshnessPolicy to make room for the new freshness model. This is a breaking change, and will require updating imports in your Dagster code if you’re still using FreshnessPolicy. Existing imports of FreshnessPolicy will fail with an ImportError

To continue using LegacyFreshnessPolicy in 1.11, please refer to the 1.11 migration guide.

The new FreshnessPolicy

As of 1.11, the new FreshnessPolicy API is in preview. We encourage you to experiment with it, and give us feedback (in the comments of this discussion)! You should expect a few rough edges and potentially breaking changes in upcoming releases, so we don't yet recommend it for use in critical production environments.

More information about the different types of freshness policies available and their usage can be found in our docs:

https://docs.dagster.io/guides/labs/freshness/

What about freshness checks?

Freshness checks aren’t going anywhere for now - they will continue to be supported through 1.11 and beyond, and remain a viable option for specifying and evaluating asset freshness in Dagster.

In the future, we expect all freshness checks use cases will be better supported by freshness policies, and accordingly intend to supersede freshness checks with the new FreshnessPolicy API once the latter is more mature. Consequently, freshness checks won’t be receiving feature support or improvements moving forward.

If you want to work on the bleeding edge of upcoming features and can tolerate some instability for a few months, we recommend you give the new freshness policies a shot. If you are already using freshness checks or prefer to wait for more stable APIs, we recommend you use freshness checks. When freshness policies are stable, we’ll provide a migration guide to move to freshness policies.