Discontiguous Time Window Partitions

[maxfirman]

Problem

Dagster does not currently support discontiguous time window partitions. This limitation means that dealing with business day partitions or custom calendars is awkward. See this related issue.

For example, to express business days (Mon-Fri) using time window partitions one can do the following:

TimeWindowPartitionsDefinition(start=datetime(2016, 1, 5), fmt="%Y-%m-%d", cron_schedule="0 0 * * 1-5")

The problem with this approach is that the Friday partition actually spans the range [Friday 00:00:00, Monday 00:00:00). This does not accurately model the intention, which is that the Friday partitions should span the range [Friday 00:00:00, Saturday 00:00:00), and Saturday and Sunday should be void. The result is that the Friday partition does not appear until Monday morning.

A workaround to this issue is to set the end_offset to 1, as shown.

TimeWindowPartitionsDefinition(
    start=datetime(2016, 1, 5),
    fmt="%Y-%m-%d", 
    cron_schedule="0 0 * * 1-5", 
    end_offset=1
)

This will "fix" the problem, in so far as the Friday partition will now show up on Friday morning. However it introduces complexity with automation and freshness checks, as the latest partition will always be missing. It also does not address the root cause of the problem.

Potential Solution

One potential solution would be to support an end_cron_schedule parameter in the TimeWindowPartitionsDefinition, e.g:

TimeWindowPartitionsDefinition(
    start=datetime(2016, 1, 5), 
    fmt="%Y-%m-%d", 
    cron_schedule="0 0 * * 1-5", 
    end_cron_schedule="0 0 * * 2-6"
)

The partition windows would start on the cron_schedule ticks and end on the subsequent cron_schedule or end_cron_schedule tick, depending on which one is earlier.

Dynamic Time Window Partitions

Cron schedules do not fit all use cases. At our organisation we have several business day calendars, which exclude weekends, public holidays and other non-trading days. These calendars are distributed over APIs.

We can use DynamicPartitionsDefinition to create dynamic partitions as described here. However the partitions created by DynamicPartitionsDefinition are "categorical" rather than "time window" partitions, which means that the UI does not show the date range picker, and TimeWindowPartitionMapping will not work.

It would be great if there was a DynamicTimeWindowPartitionsDefinition where we add/remove time window partitions dynamically.

[rprechelt]

I have identical use cases as above and am struggling against this same limitation. Any solution, including the one above, would be well appreciated!

[reinholdsson]

At our organisation we have a similar need for business days logic. Our use case is that we get reports sent to us N business days after month's ending, hence we would like to create monthly partitions but filter out those partitions if there hasn't yet passed N business days until today's date.

I have tried to create a subclass of TimeWindowPartitionsDefinition and overwrite its method _get_current_timestamp(). It works for e.g. get_partition_keys(), but in the dagster webserver (UI) it still shows all partitions up until today (?).

Another suggestion would be to add an argument such as filter_fn to TimeWindowPartitionsDefinition, e.g.

TimeWindowPartitionsDefinition(
    start=datetime(2025, 8, 5),
    fmt="%Y-%m-%d", 
    cron_schedule="0 0 * * 1-5",
    filter_fn=lambda x: ...
)

This would serve as a solution to multiple use cases, not sure though how difficult it would be to implement?