Optimizing processing of monthly customer invoices as MultiPartitionsDefinition

[the4thamigo-uk]

Hi I would like to have an asset that represents an invoice pushed to the accounting system for a customer for a given month.

I think I should model this in dagster using MultiPartitionsDefinition, where the month dimension is a represented by an instance of MonthlyPartitionsDefinition , and the customer dimension is represented by an instance of DynamicPartitionsDefinition.

Each month I would like to materialize invoices for the previous month for all the customers in the customer DynamicPartitionsDefinition.

For performance reasons I would like to materialize all the customers in a single run, rather than launching separate runs for each customer, as the materialization is quick compared to the time taken to launch a pod for the run in kubernetes.

Is the MultiPartitionsDefinition the right model for this scenario? and how can I ensure that the process of materialization is efficient?

Update: it looks like you can kinda do this using a partition range, but Im not sure you can specify a particular set of partitions to execute in the run :

#17956

[sryza]

Hi @the4thamigo-uk - specifying a non-contiguous range of partitions to backfill in a single run isn't currently possible, although it seems like a reasonable thing to add support for. I recommend filing a Github issue for this if you're interested in it, although I can't promise how soon we'll be able to get to it.

[the4thamigo-uk]

Thanks @sryza, I'll make a request, but it seems there is a huge backlog of issues it is quite possible it will never see the light of day.

In the meantime, I'm planning to tackle this with an op job that calls materialze() for each of the partitions. Is there a better way to do this do you think?