Previewing dg CLI + Components: A New Era for Dagster 🚀

[schrockn]

Previewing dg CLI + Components: A New Era for Dagster 🚀

We’re excited to announce the preview of three significant additions to the open-source Dagster framework:

• dg, our new CLI : A new command-line frontend designed for the local development and setup experience.
• A new dg-driven project structure built with scaffolding and boilerplate elimination in mind, and stacked on uv to greatly ease Python environment management.
• "Components": A framework for building and using YAML-based DSLs , supported by high-quality tooling and automatic documentation

Why Now?

We’ve received consistent feedback that Dagster can be challenging to start with and isn’t always accessible to all users. Historically:

• We offered little guidance and tooling support on project structure, leaving users to figure it out independently.
• While our local development experience was strong with unit tests and the UI, we lacked a CLI-first workflow.
• Dagster’s Python APIs have been a barrier for those who do not want to learn a broad Python API or do not want to use Python when integrating with an orchestration. This expands the user base of users and stakeholders that can build with Dagster.

With dg and Components, we aim to address these concerns and make Dagster best-in-class when it comes to ease of use and accessibility while simultaneously making it more powerful. Put another way, these two projects together “lower the floor” by making it more accessible and “raise the ceiling” by improving the organization of large projects and providing more leverage to data platform teams.

Meet dg: A CLI and Project Management Solution

The dg CLI introduces a streamlined experience for managing Dagster projects:

• Tool-driven scaffolding and Project Organization: We have completely revamped our project organization and reoriented it around scaffolding. Our new project organization requires much less boilerplate, and you continue to use this scaffolding from project inception all the way to massive scale. Additionally this scaffolding is customizable by users on a per-integration basis, so you can generate exactly the right structure and configuration you need, regardless of target technology. Existing projects can be easily adapted to the format so existing users will not be left behind.
• Simplified Python Environments via uv: uv is the future of Python project management. It is so much faster and better than existing solutions; it not only makes project management better, but completely changes its nature. While we don’t require uv we strongly encourage it in the happy path of the project and dg is optimized for use with uv. The net result of this integration is that working with dg requires no manual virtual environment management, which has traditionally been a huge barrier to entry for less experienced Python programmers, and a massive source of pain even for the most advanced Python users.
• Automatic definition discovery: At scale projects in Dagster currently require lots of manually shuffling of imports and boilerplate(a process I refer to in a not-so-fondly way as the “import circus”). With dg, you can now set up your definitions in a hierarchical folder structure (built by scaffolding) and automatically load those definitions without writing any import statements.
• dg check: We now have a check command to easily and quickly check the validity of all your definitions prior to launching a tool or the web UI, and reports error messages in much more direct and clean fashion without framework code in the stack trace. This is great for local development and CI/CD.
• CLI-first local dev loop: Via commands like list , launch, check dg supports a much richer DX where you never have to leave your terminal.

Introducing Components: YAML-Powered Integrations and Definition Factories

In addition to dg we are previewing Components, a new framework that enhances integrations with a YAML DSL and robust tooling:

• YAML DSL: A declarative, accessible frontend with high-quality tooling—no deep Python knowledge required. High quality error messages, vscode integration and dynamic templating are all included.
• Automatic Documentation: Built-in docs and a generated site, driven entirely by metadata in code.
• Customizable: Components are deeply customizable, so you can build custom components and integrations for yourself and your stakeholders. With the assistance of the component tooling ecosystem, your integration feels like a full end-to-end product experience.
• Initial Integrations: Includes support for Python scripts, Sling, and dbt, with many more planned.

Incremental Adoption, No Disruption

These features are additive and there are no breaking changes.

• It is straightforward to make your project usable by dg.
• You can use existing code and definitions within Components.
• You can easily use Components in your existing projects with or without dg

Show me the Demo!

If you want to skip right to the demos here you go:

• dg introduction (video).
• Build an ELT pipeline with Sling and Dbt in 10 minutes (video)
• Building a custom Component (video)

dg New Project Layout and CLI

In the past, we’ve largely taken a hands off approach to project layout. While the existing dagster CLI did provide some default scaffolding it only was for initial setup, and even then, users (including us) had to completely reorganize their project, even at modest scale.

dg scaffold and Project Layout

To get started with dg you first run dg scaffold project <name>. This gives you a project structure like this:

 ├── <name>
 │   ├── __init__.py
 │   ├── defs
 │   ├── definitions.py
 │   └── lib
 │       └── __init__.py
 ├── <name>_tests
 │   └── __init__.py
 └── pyproject.toml

There are two new folders of note: defs and lib. lib contains types, decorators and functions used to create definitions. defs contains the definitions (e.g. assets, schedulers, sensors, etc) themselves.

The defs folder is where scaffold command creates target definitions. For example you can use the command dg scaffold dagster.asset hello.py create a hello.py file with a stub asset. As you will see with Components, this scaffolding is customizable and in the case of integrations can generate configuration files specific to that technology.

Automatic Definition Loading

Say goodbye to the import circus and heavyweight top-level Definitions construction.

In most dagster projects there is a top-level defintions.py file that looks something like this:

from dagster import dg
from my_project.assets.asset_one import asset_one
from my_project.assets.asset_two import asset_two
from my_project.resources import some_resource

defs = dg.Definitions(
  assets=[asset_one, asset_two], 
  resources={"some_resource": some_resource},
)

As a project scales this can become hundreds of lines of code. And it introduces friction when it comes to reorganizing your project.

In a dg project the top-level definitions.py looks like this

import new_project.defs
from dagster_components import load_defs

defs = load_defs(defs_root=new_project.defs)

and in most cases will never have to be touched again. load_defs performs an autoloading process which recursively discovers all definitions in a project.

Just move your existing definitions into the defs folder and start deleting those import statements.

CLI-driven DX

check → list → launch

• Easier to read errors with dg check defs

  Definitions in dagster are constructed programmatically and like within any programming, you introduce errors during the process. Previously dagster would produce long stack traces and you would have to “hunt” for the error that you introduced. This was annoying and time-consuming. We have improved the situation, with much shorter stack traces by default that typically point directly to the offending line of code right at the bottom.

Note: these are meant to be super small. The stack trace was so long it would dominate the post visually! There's just a much smaller wall of red.

Before:

After:

• dg list defs

  A frequent developer workflow is updating metadata about definitions and verifying that the what you thought was updated was actually updated. Previously this generally required launching the web UI locally and finding the definition in question. This was unnecessarily heavy and slow for many interactions. We now have a much lightweight approach in the form of dg list defs.

  > dg list defs
  Assets
  ┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓
  ┃ Key      ┃ Group   ┃ Deps ┃ Kinds ┃ Description ┃
  ┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩
  │ my_asset │ default │      │       │             │
  └──────────┴─────────┴──────┴───────┴─────────────┘

• dg launch --assets

  Similarly one frequently wants to launch a run to test an asset or selection of assets. Previously this would have required opening the web UI, selecting the subset you are interested in, and then clicking the “Materialize” button. Instead we offer a lightweight launch command that allows you launch a run directly in the terminal.

Multi-project layout via workspaces

Finally dg offers first class support for multi-project layouts in Dagster by creating an enclosing folder that we call a Workspace. One can create this via the dg init --workspace command. In the folder

This is where the uv integration shines. uv has sensible defaults for Python environment management where the virtual environment in a hidden .venv folder directly within the project.

Let’s say you the workspace with a pandas_project and then add another duckdb_project.

dg init --workspace my_workspace && dg scaffold project pandas_project && dg scaffold duckdb_project

pushd projects/pandas_project && uv add pandas && popd

pushd projects/duckdb_project && uv add duckdb && popd

Now if you run dg dev from the workspace root and go to the Deployment tab:

Each of these running code locations have their own dedicated virtual environment. It’s never been so easy.

Walkthrough of using Components: Setting up a dbt integration.

Look at the existing code

One of Dagster’s most popular integrations is dbt. Dagster, as an asset-oriented orchestrator, can natively understand and operate the underlying models and lineage of a dbt project like no other orchestrator.

However, setting up the integration is difficult. For some engineers, it takes too long; for others, it proves to be impossible.

The following is an example of setting up a fairly simple dbt integration. There is only one customization, adding a prefix to the default asset key

For illustrative purposes I have also added comments of an engineer just trying to get this thing up and running:

import dagster as dg
from dagster_dbt import DagsterDbtTranslator, DbtCliResource, DbtProject, dbt_assets

# This is overly complicated for doing something as simple adding a prefix
class CustomDagsterDbtTranslator(DagsterDbtTranslator):
    def get_asset_key(self, dbt_resource_props: dict) -> dg.AssetKey:
        return super().get_asset_key(dbt_resource_props).with_prefix("snowflake")

# For some reason I have to build this object and call prepare_if_dev,
# whatever that does.
dbt_project = DbtProject(project_dir="../../jaffle_shop")
dbt_project.prepare_if_dev()

# Wow I have to do a bunch of things now:
#   * I have to write a decorator that shuffles along the classes and objects
#     I created, which I don't understand.
#   * The function also has to take a resource # and some sort of "context"
#   * I then have to pass along to the dbt.cli call, which I don't get why
#     I have to do.
#   * All of this is just to call `dbt build`, which seems like it should
#     be the default behavior?
@dbt_assets(
    manifest=dbt_project.manifest_path,
    dagster_dbt_translator=CustomDagsterDbtTranslator(),
)
def jaffle_shot_dbt_assets(context: dg.AssetExecutionContext, dbt: DbtCliResource):
    yield from dbt.cli(["build"], context=context).stream()

# Now I have to do whatever this is because if I didn't Dagster complained
# about not getting a resource? Ok, whatever. It's working now.
defs = dg.Definitions(
    assets=[jaffle_shot_dbt_assets],
    resources={"dbt": dbt_project},
)

This is too complicated for what this code is doing.

With Component and dg:

dg scaffold component dagster_dbt.DbtProjectComponent jaffle_shop_assets --project-path jaffle_shop

With Components this user would write this, which is equivalent:

type: dagster_dbt.DbtProjectComponent

attributes:
  dbt:
    project_dir: "../../jaffle_shop"
  asset_attributes:
    key: "snowflake/{{ node.name }}"

This is much better. It expresses exactly what the user wants to customize, and no more.

What about tooling?

You may be thinking. “This looks great in theory, but I’ve gone down this road of YAML frontends before and while it feels good at first, with real usage you have terrible errors, no tooling support, no documentation, etc.”

This is all true, unless you have powerful tooling. We provide that. Let’s walk through the user experience of a user installing, discovering, using and debugging this integration.

First let’s scaffold a project and install dagster_dbt

dg scaffold project dbt_project && cd dbt_project && uv add dagster_dbt

Now we can run dg docs serve. We see embedded documentation that tells us a lot about how to create and use this component.

We can scaffold this and create the yaml file we showed above:

dg scaffold component dagster_components.dagster_dbt.DbtProjectComponent jaffle_shop_assets --project-path jaffle_shop

Note: dagster_components.dagster_dbt.DbtProjectComponent will become dagster_dbt.DbtProjectComponent once the project matures and is integrated into our main libraries

type: dagster_components.dagster_dbt.DbtProjectComponent

attributes:
  dbt:
    project_dir: ../../../jaffle_shop

We can verify the validity of the yaml file using two methods: dg check yaml and through IDE integrations.

Let’s imagine we add a key that does not exist:

type: dagster_components.dagster_dbt.DbtProjectComponent

attributes:
  dbt:
    project_dirz: ../../../jaffle_shop
  nope: "this does not exist"

dg check yaml prints out:

   2 |
   3 | attributes:
   4 |   dbt:
   5 |     project_dir: ../../../jaffle_shop
   6 |   nope: "this does not exist"
     |   ^ Additional properties are not allowed ('nope' was unexpected)
     |

Obvious and fast.

We can also install have VSCode and Cursor support (and any thing the supports the yaml-language-server) via dg utils configure-editor vscode

A component in the end creates definitions, which means that all the tooling in the first section also applies!

Building a custom Component

The Components system is extensible and powerful, and you can build your own components. By doing so can provide a complete end-to-end product experience for your stakeholders, from discovery to inspection, to authoring, to deployment

There are three pillars of any Component and we will go through each of them

• A definitions factory
• A YAML frontend
• Scaffolding

In our theoretical situation, we will be authoring custom components for downstream stakeholders that want to write raw SQL to a DuckDB database. These are not particularly technical users so we want them to only have to deal with YAML and SQL files.

Let’s take a tour of the component.

Definitions Factory

The core workhorse function of a component is build_defs

@dataclass
class DuckDbComponent(Component):
    """A component that allows you to write SQL and print it out without learning dbt or Dagster's concepts."""
    
    assets: Sequence[AssetSpec]
    sql_file: str

    def build_defs(self, load_context: ComponentLoadContext) -> dg.Definitions:
        specs = to_specs(load_context, self.assets)
        assert len(self.specs) >= 1, "Must have at least asset"
      
        @dg.multi_asset(name=f"run_{self.specs[0].key.to_user_string()}", specs=self.assets)
        def _asset(context: dg.AssetExecutionContext):
            df = duckdb.execute(open(self.sql_file).read()).fetchdf()
	          print(df.head().to_markdown(index=False))
            
        return dg.Definitions([_asset])

What this is in effect is a formally supporting factory pattern, which up until now in Dagster has been a convention.

YAML Frontend

Next we want to provide a YAML frontend for this component, so that it can be used without Python. As a component author you create Pydantic models. These model classes directly determine the YAML schema.

class DuckDbComponentModel(ResolvableModel):
    sql_file: str
    assets: Sequence[AssetSpecModel]

@dataclass
class DuckDbComponent(Component, ResolvedFrom[DuckDbComponentModel]):
    """A component that allows you to write SQL without learning dbt or Dagster's concepts."""

    assets: Sequence[ResolvedAssetSpec] # Annotated[AssetSpec]
    sql_file: st

Note: ResolveAssetSpec is just an annotated AssetSpec. This allows the component to have a Pythonic interface that can take AssetSpec and other python objects directly, while still providing the YAML schema via DuckDbComponentModel

type: my_project.DuckDbComponent
attributes:
   assets:
      - key: "my_asset"
   sql_file: "my_asset.sql"

SELECT * FROM `my_asset.csv`

From this point on all the tooling you have seen so far applies: dg docs, dg check yaml, and so on.

Custom Scaffolding

You may have noticed that even in this example there is boilerplate for your stakeholder to setup. They need to create a component.yaml file, fill out the asset key and the sql file, and then create the sql file itself.

On an opt-in basis, we can automate this process via custom scaffolding which can be attached to the component:

class DuckDbScaffolderParams(BaseModel):
    sql_file: Optional[str]
    asset_key: Optional[str]

class DuckDbComponentScaffolder(Scaffolder):
    def scaffold(self, request: ScaffoldRequest, params: DuckDbScaffolderParams) -> None:
        root_name = request.target_path.stem
        asset_key = params.asset_key or f"{root_name}"
        sql_file = params.sql_file or f"{root_name}.sql"

        Path(sql_file).touch()

        scaffold_component_yaml(
            request=request,
            attributes={"sql_file": sql_file, "assets": [{"key": asset_key}]},
        )
        

@scaffold_with(DuckDbComponentScaffolder)
@dataclass
class DuckDbComponent(Component, ResolvedFrom[DuckDbComponentModel]): ...

Implementing the scaffolder is essentially a plugin for the dg cli application, and you can now invoke the scaffolder via the following command

dg scaffold component your_project.DuckDbComponent --sql-file my_asset.sql --asset-key my_asset

Which ends up calling the scaffold method you defined. This command creates the component.yaml file we saw above.

Call to Action: Explore and Contribute

Both dg and Components are available in preview under the Labs tab in our docs. To get started:

• brew install uv && uv tool install dagster-dg
• dg init

While you are free to explore this toolkit on your own, if you have a strong desire to use this in production immediately, please reach out to us in the #dg-components channel. The ideal user is someone who has or is considering building their own DSL or scaffolding frameworks on top of Dagster already, and would rather invest in the system supported by Dagster Labs and shape its roadmap via feedback and usage.

We’re incredibly excited about this community preview and believe that these initiatives are a massive leap forward for Dagster and will reshape the entire project as we drive forward with our roadmap. It represents a step function change in ease of use and power.

[TimoVink]

Very cool, many features were already reluctantly home-growing (like autodiscovery) becoming native 🎉. Looking forward to having a play!

Minor note, the "Labs" link (https://dagster.io/docs/labs) is broken. It should probably be something like https://docs.dagster.io/guides/labs/.

[cmpadden]

Fixed, thank you!

[menzenski]

Will the "scaffold" command include support for scaffolding out e.g. a Kubernetes user deployment from the dagster/dagster-user-deployments Helm chart? That would be awesome - I'd love to have a way to support end-to-end project scaffolding all the way from "create project" to "deploy it".

[cmpadden]

That's a neat idea.

The scaffold function will be entirely customizable, so you, or a community member, would be able to create a custom scaffold template for deployments.

[danielgafni]

Looking at this example:

class DuckDbComponentModel(ResolvableModel):
    sql_file: str
    assets: Sequence[AssetSpecModel]

@dataclass
class DuckDbComponent(Component, ResolvedFrom[DuckDbComponentModel]):
    """A component that allows you to write SQL without learning dbt or Dagster's concepts."""

    assets: Sequence[ResolvedAssetSpec] # Annotated[AssetSpec]
    sql_file: st

This looks like quite some code duplication. Are there any plans to reduce it?

I guess you can always have a shared parent class for common parameters

[v1gnesh]

Still reading through this, a quick question - how do I handle having a separate venv for dagster webserver(s)?
Code locations are placed inside a workspace, okay. What about a venv for webserver?
EDIT: Please also treat Jetbrains IDEs (PyCharm / IntelliJ) as first class "citizens". Tired of being autopiloted into VS Code.

[danielgafni]

Please put a 👍 on #31346 😄

[thomaslaich]

I did. Thanks for directing me to the issue :)

Although I'm not sure this would tackle my problem exactly. After all, pixi environments are not venvs, are they?

[thomaslaich]

After looking into the PR here:
#32709

and the function here:

dagster/python_modules/libraries/dagster-dg-core/dagster_dg_core/utils/__init__.py

Line 107 in 21ff3be

else:

This might actually work if I just point it to .pixi/default. I will give this a try :)

Pretty sure it won't work on Windows though - I don't think pixi envs have a Script folder.

[cmpadden]

Thanks for looking at that PR @thomaslaich , let me see what I can do about that Scripts folder.

[danielgafni]

Let's continue the discussion in the PR. Thank you Colton for addressing this

[Bonnevie]

With regards to scaffolding and the "import circus", I have always had some pains from managing key prefixes and dependencies. I have my asset definitions spread over multiple files, and I find it convenient to be able to do something like:

from dagproject.other_assets import dependency_asset

PREFIX = ["module_prefix"]

@asset(key_prefix=PREFIX, ins={"dependency": AssetIn(key=dependency_asset.key))
def new_asset(dependency):
   ...

This has always clashed with auto-discovery a la load_assets_from_modules since it would detect dependency_asset in both other_assets and the module it is imported into, typically resulting in a duplicate key error. My solution so far is to do from dagproject import other_assets and then reference it as other_assets.dependency_asset, but that contributes to the circus. If I for some reason instead need to set the key as a literal, I need to import the PREFIX around to keep myself somewhat sane while writing out asset keys.

Does the new project scaffolding avoid the above auto-discovery issues, and does it help further with organizing assets or dependencies across multiple modules or only with simplifying the definitions file itself?

[danielgafni]

I am also having issues with this. Relevant discussion: #28422

[johnoscott]

This is so good and acknowledges the fact that many Orchestration workflows are repeatable patterns that are well expressed as YAML configs. I can't wait to see the 3rd party tooling support for this. Well done Dagster team.

[mattsmith123]

getting an error calling yarn when I dg docs serve after following the dbt example above:

subprocess.CalledProcessError: Command '['yarn', 'install']' returned non-zero exit status 1.

[mattsmith123]

Got it working. I had an old node.js environment configured.

[cmpadden]

Thanks for pointing this out, @mattsmith123 - we'll need to improve the documentation around environment setup and dependencies for docs.

[jairus-m]

Just an FYI for anyone else looking here!

I came across issues running dg docs serve as well. I don't have any experience with javascript and node.js but a simple global install of yarn did the trick:

On mac: brew install yarn (my installed version is 1.22.22).

[cmpadden]

Thank @jairus-m, we're also exploring adding it to the UI in dg dev so that it doesn't have to be served independently. Hopefully that simplifies things!

[matthias-Q]

Is there some documentation for dg? dgasks me to add configs into the pyproject.toml (e.g. tool.dg). I want to know what config options are there.
We have a quite old dagster project, which grew overtime and does not follow the suggested structure of having "defs" folder. How can I specify what my "defs" folder is. I tried setting tool.dg.project.root_module but that does not work

[mjclarke94]

Finding dg launch much better for quick iterative debugging of more complex assets/resources than launching things via the webUI.

(Not to mention having a one liner which you can pass to an AI agent as a "keep running this until the errors go away" workflow is terrifying and amusing in equal measure).

One thing I am really missing though is having it play nice with debugpy. I'm using the trick (cannot remember original author!) of having a launch.json containing:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Dagster",
            "type": "debugpy",
            "request": "launch",
            "module": "dagster",
            "args": [
                "dev",
            ],
            "subProcess": true,
            "serverReadyAction": {
                "pattern": "Serving dagster-webserver",
                "uriFormat": "http://127.0.0.1:3000",
                "action": "openExternally"
            },
        },
    ]
}

Which allows you to set breakpoints in VSCode anywhere whilst an asset is being materialised and really dig into whats going on. Bit of a pain having to spin up and tear down the webserver every five minutes but otherwise very helpful.

It would be great to have some option like:

dg launch --assets foo --partition bar --remote-debugger

To natively support this in the much more streamlined dg launch dev workflow.

[mentallabyrinth]

Been playing with dg over the weekend by converting the Dagster University project and absolutely love it. Much, much better experience than my first foray into Dagster; having the need to wrangle dependency/package management issue this time around are non-existent -- much more enjoyable experience.

However, I am having an issue with auto loading resources. Assets have no problem, but resources appear not found when using the auto definitions loading. When inspecting the load_defs block, the resources array is empty. Is there something additional to be done when loading resources automatically?

[jairus-m]

I've been playing around with the dagster_dlt integration and the new dg CLI and have also been having issues with loading resources!

I documented the behavior here.

It looks like resources configured in load_defs within definitions.py are not recognized by dg?

import dagster_project.defs
from dagster_components import load_defs

from dagster_project.defs.defs import resources

defs = load_defs(
    defs_root=dagster_project.defs,
    resources={
        "dlt": resources.dlt_resource # this does not seem to be recognized
    },    
)

How are you declaring the resource for auto-loading in your migration of your Dagster University project?

[schrockn]

Hey @jairus-m we'll load into the load_defs issue, but looking at your repo I think easiest fix would be to change your resources/__init__.py from

from dagster_dlt import DagsterDltResource

dlt_resource = DagsterDltResource()

to

import dagster as dg
from dagster_dlt import DagsterDltResource

defs = dg.Definitions(resources={"dlt_resource": DagsterDltResource()})

[lokofoko]

Hey, I am trying to understand how will this work with this action https://github.com/dagster-io/dagster-cloud-action/blob/main/actions/build_deploy_python_executable/action.yml , that is used in serverless deployment.

It requires setup.py, that is not used in here, do you have any ready to switch options for this matter?

[lokofoko]

Do I understand correctly @jairus-m, that using requirements.txt instead of setup.py, it will allow mentioned action to go through?

[jairus-m]

Yes, If I remember correctly from the GHA logs when I first encountered the issue, the Dagster serverless deployment will first default to looking for a setup.py file and if absent, will use a requirements.txt file.

[lokofoko]

Thanks, will try this approach. Much appreciated. Both you and @schrockn for such fast response.

[lokofoko]

hey, @jairus-m, does yours add pywin to the requirements? for some reason it automatically adds it

[jairus-m]

@lokofoko mine does not, but uv adds all the necessary dependencies based on the explicit packages you defined in the pyproject.toml. therefore, if one of the dependencies you outlined requires pywin, then i would expect it to be in your requirements.txt.

did the deployment work?

[AlexRichards77]

I'm getting an error when trying to scaffold an asset, it creates a directory instead of a py file.

PS C:\Repos\malinko> dg scaffold dagster.asset assets/hello.py  
Using C:\Repos\malinko\.venv\Scripts\dagster-components.exe
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Repos\malinko\.venv\Scripts\dagster-components.exe\__main__.py", line 8, in <module>
  File "C:\Repos\malinko\.venv\Lib\site-packages\dagster\components\cli\__init__.py", line 35, in main
    cli(auto_envvar_prefix=ENV_PREFIX)
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 1161, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 1082, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 1697, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 1443, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\click\core.py", line 788, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Repos\malinko\.venv\Lib\site-packages\dagster\components\cli\scaffold.py", line 46, in scaffold_object_command
    scaffold_object(path, obj, typename, scaffold_params, scaffold_format)
  File "C:\Repos\malinko\.venv\Lib\site-packages\dagster\components\component_scaffolding.py", line 78, in scaffold_object
    check.invariant(
  File "C:\Repos\malinko\.venv\Lib\site-packages\dagster_shared\check\functions.py", line 1681, in invariant
    raise CheckError(f"Invariant failed. Description: {desc}")
dagster_shared.check.functions.CheckError: Invariant failed. Description: scaffold must be either 'yaml' or 'python'. Got None.
An error occurred while executing a `dagster-components` command in the Python environment at C:\Repos\malinko\.venv.

`uv run dagster-components scaffold object dagster_components.dagster.asset 'C:\Repos\malinko\malinko\defs\assets\hello.py'` exited with code 1. Aborting.

[schrockn]

Can you post this question in the slack channel? #dg-components (link)

[danielgafni]

Looks like the Components system doesn't provide a way to set key_prefix for all assets/components in scope?
I was looking at AssetPostProcessor in particular.

Edit: according to @cmpadden that's in development

[matthias-Q]

Will there be a migration guide? It will be quite hard to migrate larger dagster projects into this structure

[cmpadden]

Hey @matthias-Q - we'll definitely put together a migration guide, that's a great idea.

As for the migration procedure, I just did that for our dagster-open-platform repository, and it was a surprisingly small lift. The commit may seem overwhelming, but it was essentials a git mv my_previous_definitions_location defs to use the new defs/ structure. Then, it was updating imports to use the new path. You can see the full commit here.

dagster-io/dagster-open-platform@cc06560

This project was already structured with multiple Definitions and a Definitions.merge in the root definitions.py, so that may contribute to the ease of migration. But you can see how the definitions.py changed to a single load_defs call in this screenshot.

Hope this helps, and if there are certain scenarios you hope to see in the migration guide, please share. Thank you.

[schrockn]

The generalized approach is just to move your existing files that contain Dagster definitions into the defs folder and let them be autoloaded. Just start moving files into defs and start deleting import statements.

[NellyWhads]

Hey guys! Awesome work here!

Where can we go to see the roadmap and feature progress for dagster-dg? Are there a set of tags on GitHub?

[Shrekouille]

I have tested the dagster components locally on my machine, and it's great !
What about the kubernetes deployment ? Is there a helm template providing this functionality ? https://artifacthub.io/packages/helm/dagster/dagster is code locations oriented from my understanding.

[EandrewJones]

Yes, it'd be great to see an e2e example repo for deploying a multi-project workspace that uses this new scaffolding similar to the existing CI/CD hybrid deployment repo: https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/tree/main

Nothing beats a full example repo.

[NellyWhads]

Waiting patiently to be able to use these flexible, ergonomic components in production!

[cmpadden]

Soon!

[schrockn]

Update is here: #30285 RC is coming soon!

[nsteins]

How does autoloading of definitions work with Asset factories and the like?

[schrockn]

Hey @nsteins. Components are designed to be the first class notion of asset factories. If you don't want to use that API, you can also write

@dg.definitions
def defs():
   return Definitions(assets=build_my_assets())

which does not evaluate on import automatically.

One of the major value propositions of the new layout is that we decouple definitions construction from the python import process, which has a ton of benefits.

[nsteins]

Thanks @schrockn, this seems like a very useful feature, I'll have to dig into it more and figure out how to translate our factories into components. I agree with some of the earlier suggestions that a migration guide would be very useful.