How can I use the cli correctly？

[imporoutco586]

Hi,

As the title asks, I created a project based on the documentation and ran it, and then I found the cli command in the dagster documentation API class. I tried to debug my project using the cli, but I found that using the cli (e.g. dagster job execute -m project1 --job test1_job) generated a completely new instance, I can't find these requests in the dagster websever which I have run.

What should I do?I want to run multiple jobs through cli commands and have them reflect on the dagster core or dagster ui that I have already run. I want to use this to investigate how dagster schedules multiple job requests.

Look forward to hearing from you. Any reply will be greatly appreciated.

[garethbrickman]

To ensure that job executions from the CLI are reflected in the Dagster UI, you need to make sure that both the CLI and the Dagster UI (webserver) are using the same Dagster instance. This is typically managed by setting the DAGSTER_HOME environment variable to point to the same directory for both processes.

1. Set the DAGSTER_HOME Environment Variable: Ensure that the DAGSTER_HOME environment variable is set to the same location in the environment where you're running the Dagster webserver and the CLI commands. This environment variable tells Dagster where to store instance-related data, including run history, event logs, and configuration.

2. Check the Instance Configuration: Verify that the dagster.yaml file in the DAGSTER_HOME directory is correctly configured and that both the CLI and webserver are indeed using the same instance configuration.

3. Use the Same Python Environment: Make sure that the Python environment from which you're running the CLI commands is the same environment where the Dagster webserver is running. This ensures that they have access to the same Dagster packages and configuration.

4. Check Network Configuration: If you're running the webserver and the CLI on different machines or containers, ensure that they can communicate over the network and that there are no firewall or network policy issues preventing them from accessing the same Dagster instance.

[garethbrickman]

Choose a directory on your filesystem where you want Dagster to store its instance-related data, such as run history and event logs. This directory will be your DAGSTER_HOME.

Set the DAGSTER_HOME environment variable to point to the chosen directory. The way to set environment variables depends on your operating system and shell:

bash (Linux/macOS):

export DAGSTER_HOME=/path/to/your/dagster_home

PowerShell (Windows):

$env:DAGSTER_HOME = "C:\path\to\your\dagster_home"

After setting the DAGSTER_HOME environment variable, you can start using Dagster tools like dagit and dagster-daemon, and they will use the directory you specified for storing instance data.

Remember to replace /path/to/your/dagster_home or C:\path\to\your\dagster_home with the actual path to the directory you have chosen for your Dagster instance. If the directory does not exist, you should create it before setting it as DAGSTER_HOME.

[imporoutco586]

Thank you very much!

I have set up DAGSTER_HOME, but the cli I am running on the command line is still not reflecting in the Dagster UI. Here are the results of my run:

dagster asset materialize -m my_project --select hackernews_top_story_ids
2024-03-13 16:07:56 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 26332 - RUN_START - Started execution of run for "__ASSET_JOB_0".
2024-03-13 16:07:56 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 26332 - ENGINE_EVENT - Executing steps using multiprocess executor: parent process (pid: 26332)
2024-03-13 16:07:56 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 26332 - hackernews_top_story_ids - STEP_WORKER_STARTING - Launching subprocess for "hackernews_top_story_ids".
2024-03-13 16:07:57 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - STEP_WORKER_STARTED - Executing step "hackernews_top_story_ids" in subprocess.
2024-03-13 16:07:58 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - RESOURCE_INIT_STARTED - Starting initialization of resources [io_manager].
2024-03-13 16:07:58 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - RESOURCE_INIT_SUCCESS - Finished initialization of resources [io_manager].
2024-03-13 16:07:58 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - LOGS_CAPTURED - Started capturing logs in process (pid: 25304).
2024-03-13 16:07:58 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - STEP_START - Started execution of step "hackernews_top_story_ids".
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - STEP_OUTPUT - Yielded output "result" of type "Any". (Type check passed).
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - hackernews_top_story_ids - Writing file at: data\hackernews_top_story_ids using PickledObjectFilesystemIOManager...
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - ASSET_MATERIALIZATION - Materialized value hackernews_top_story_ids.
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - HANDLED_OUTPUT - Handled output "result" using IO manager "io_manager"
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 25304 - hackernews_top_story_ids - STEP_SUCCESS - Finished execution of step "hackernews_top_story_ids" in 21.59s.
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 26332 - ENGINE_EVENT - Multiprocess executor: parent process exiting after 24.56s (pid: 26332)
2024-03-13 16:08:20 +0800 - dagster - DEBUG - __ASSET_JOB_0 - a020ac7d-7efc-4df9-a331-b86ec719befe - 26332 - RUN_SUCCESS - Finished execution of run for "__ASSET_JOB_0".

The output log does not show that it generated a temporary storage folder, but in fact it still ran the instance in a temporary folder. Is there a problem with my cli that is causing this? Welcome to point out!

Thank you very much for your answer anyway!

[imporoutco586]

Thank you very much for your patience, I have found the answer to this question!

There is also a small question, can I execute more than one job at a time through the cli? Use a command such as dagster job execute -m my_project --job job1,job2 My present format seems to be wrong, looking forward to your answer!

[garethbrickman]

Ah what was the solution?

For your follow-up question:

Yes, you can execute more than one job at a time through the CLI. Each job execution command you issue through the CLI will run in its own process. To run multiple jobs concurrently, you can open multiple terminal windows or tabs and issue a separate execution command for each job you want to run.

For example, to execute two jobs, you would open two separate terminal sessions and run the following commands, one in each terminal:

dagster job execute -f job_file_1.py

dagster job execute -f job_file_2.py

Each command will start the execution of the respective job, and they will run in parallel, assuming your system has the necessary resources to support concurrent execution.

[imporoutco586]

Ok, thank you for your answer, I all understand!

For the first question, your first answer in the second floor is right. I have been able to find feedback on my cli on the UI. The last time it didn't work I forgot to set DAGSTER_HOME on the terminal running the cli as well.

Anyway, thanks again for your patience!

[max-hassani]

Is it possible to configure the CLI to use a self-hosted dagster on K8s?
I am interested in triggering via CLI the execution/materializing of jobs.

[themightychris]

The CLI runs Dagster locally, all the commands to trigger things don't connect to a server but run the work directly within them. If you want to trigger a run on a hosted instance, you need to POST a GraphQL API request