Major changes to dg and Components leading up to RC in early July

[schrockn]

About two months ago we previewed dg and Components in a Github discussion, followed by a webinar last month. If you are new to the system and want to learn more we have docs in Labs for both dg and Components. We have been making great progress in concert with our early partners and are ready to move on to the next phase of the project: Release Candidate (RC)

dg changes leading up to RC

This week are pushing major changes to the dg CLI interface and project layout over the next two releases (1.10.18 and 1.10.19). This is in preparation for our reclassification of dg and Components both as RC the beginning of July. At that point, dg and Components will be integrated into our mainline documentation, used in our tutorials, and recommended for production use.

From here on out, we are focusing on stabilization, docs, and building Components-style integrations for the technologies that will most benefit from them, without any more major changes in vocabulary or project layout.

We're confident based on our and our partners' experiences that this is going to be a broadly used set of features increasingly at the core of the Dagster experience. It makes Dagster easier to use and more powerful at the same time. And it is opt-in, additive, and incrementally adoptable. In fact, we've changed our entire internal data platform to use this new stack in place and with incremental process, and we're thrilled with the process and results.

We thank all the early adopters and appreciate your patience dealing with the changes and bugs that are inevitable as part of a preview. It is an invaluable part of the process, thank you for being a part of it. Once the RC lands, there will be a step function improvement in stability.

You do not need to immediately migrate to these changes, and we have strived to maintain backwards compatibility for our existing partners. However, we will remove support for some of these APIs for the RC.

Structural changes in the dg cli architecture

dg is a traditional python CLI installed on a per-project basis

In our preview release dg was a tool that was installable using the uv tool system (and later brew, curl, and so forth). This was a constant source of confusion and introduced version skew issues. Instead, dg is installed on a per-virtual-environment basis, more in line with traditional Python CLIs.

create-dagster for project and workspace scaffolding

However we still want the ability to "bootstrap" a project without a pre-existing virtual environment. For this we have a new CLI, create-dagster, invocable via pipx or uvx, and installable via brew, curl, and as a uv tool. It is responsible for creating a new Dagster project on your local machine and provides a happy path for those users that do not have an opinionated virtual environment toolchain.

dg is package-manager-neutral

While we default to uv in create-dagster in our documentation nothing else ties the system to uv. You can use any Python environment management system as dg is installed in your local environment whether that be pip, poetry, or others.

CLI ontology

Namespaced scaffold commands

There is no longer a global namespace of scaffolds that spans projects, definitions, and components. Instead we are namespacing this command, which we believe is more intuitive and organized. There will be dg scaffold defs and dg scaffold component to start. dg scaffold project has been moved to the new dagster-create CLI.

dg scaffold defs

One of the new namespaces is defs, which means that you are scaffolding a set of definitions at a specific path in the defs hierarchy. This is inline with the move from component.yaml to defs.yaml detailed below.

No more "plugins"

Related, we are almost entirely hiding the term "plugin" as an external-facing noun. While it continues to live on in our internal architecture "A python package introspectable by dg" as a user you should not encounter the term aside from in diagnostics.

Project Layout Changes

No more lib

Previously we had a lib folder meant for autodiscovery of components and scaffolding, which also required the inclusion of a .gitignore file. The purpose of this folder was unclear to users and it implied that we are required all code outside of defs to reside there, which is not the case.

We still want an auto-discovery default location for scaffolded component classes (via dg scaffold component), so there is a components directory that components are scaffolded into and discoverable by the dg components list command.

definitions.py no longer required at project root

Prior to this release, a definitions.py file at the root of the project was required. This is no longer true and we support "autoloading" the defs from within the framework. This cleans up the project layout considerably.

No more special handling of component.py or definitions.py in the defs hierarchy

We have hardcoded special behavior when our autoloaded discovered definitions.py and components.py in the defs folder. For new projects this is no longer true. We have preserved the behavior for existing projects that called load_defs from the root definitions.py to avoid thrashing existing users.

Components changes

All component classes are imported at top-level dagster rather than dagster.components

Up until now our components classes (Component, ComponentLoadContext, etc) were in the dagster.components subpackage. They are now exported at the top-level of dagster

components.yaml —> defs.yaml

We have changed the name of the file that declares a component from components.yaml to defs.yaml.

We will support backwards compatibility for a few weeks, but will eliminate it once we push out the RC.

defs.yaml supports -- for multiple instances

Our usage by and our partners, it was common to want to instantiate multiple instances of components in a single yaml file. We now support this case with the --- sigil, similar to other familiar tools such as Kubernetes.

First-class support for "inline components"

We found that it was common to build custom components that were only used in a single folder in the defs hierarchy, and it felt disorganized to force this component in the global lib directory. As a result, we are more formally supporting the notion "inline components" in our scaffolding, whether the Python component class lives directly alongside the definitions that are created by it.

dg scaffold defs inline-component --typename ComponentName path/to/defs

Components Templating Frontend

More flexible mechanism for injecting variables into templates

Components comes with a yaml templating system (internally we call it Resolved) that allows for the injection of Python objects on a per-field basis. We’ve found that a killer feature of the system is the ability to inject functions as template variables and invoke them. (We colloquially refer to these as udfs). Right now the only way to do that is on a per-component-type basis and the only way to add additional variables and udfs is through entirely new custom componets.

We now allow a user to mixin new variables usable to in a template via a new @template_var decorator that can live in the defs hierarchy alongside where they are used.

Injectable by default

We have now made template fields “injectable” by default, meaning that you can use udfs and variables at any spot in the templating hierarchy. Previously one needed to “opt into” this capability at the per-field level, which was overly constraining and untenable when it came to constructing reusable libraries of schema.

Conclusion

We are releasing these changes as part of 1.10.18 to be released on Thursday.

[yuhan]

Heads-up for dg early adopters upgrading to Dagster ≥ 1.10.18

The dg CLI is now installed in each project's Python environment instead of as a global tool.
If you previously had dg installed globally and are upgrading, first uninstall the global dg tool and add dagster-dg-cli as a dependency to each of your projects. (Newly scaffolded projects will automatically include the correct dependency going forward.)

Migration steps:

1. Remove the old global CLI

   uv tool uninstall dagster-dg        # or, pip uninstall dagster-dg
   which dg                            # should NOT point to ~/.local/bin/dg

2. Add the new per-project CLI

   uv add dagster-dg-cli               # or, pip install dagster-dg-cli

Then, the mismatch warning disappears and dg --version will match the version bundled with your project.

Full details in the changelog → https://github.com/dagster-io/dagster/blob/master/CHANGES.md#11018-core--02618-libraries