Can you attach runtime metadata to jobs?

[dagsir[bot]]

is there a way to attach runtime metrics/metadata to jobs? Is this only possible at the op/asset level?

---

The question was originally asked in Dagster Slack.

[smackesey]

What exactly do you mean by "runtime metadata"? It is possible to attach metadata to the job definition at definition time:

@job(metadata={...})
def my_job():
    ...

[treilly1]

Meaning information not known at definition time (number of files ingested, file success / failure rate, etc.). We are somewhat in between asset and task (op) based workflows. For assets we attach information via Output metadata argument. This works great and is automatically graphed in UI. However, some of our older task based workflows involve multiple ops and we'd like to attach these runtime values to the job instead of having to click through the ops within the job to find values. Maybe another reason to migrate towards assets

[smackesey]

I think by "attach these runtime values to the job" you mean "to the run"-- jobs are inherently "definition-time"-- think of them like a function definition. A run represents an execution of a job.

Unfortunately we don't currently have any concept of "run-level" metadata, so probably your best bet is to migrate to assets. You can still attach metadata to on op outputs that you can see in the event log though.