change from winter to summer time not considered

[joemfe]

Describe the bug
I think you missed the shift from winter to summer time and vice versa. When comparing your SLP H0 implementation with one of our own for the year 2024, I found that up until the 30th of March the profile match almost perfectly and from 31st of March to 26th of October (included) they are off by one hour, which if I interpret this correctly indicates that you do not consider summer / winter time, right?

So basically, the SLP H0 is only applicable in countries where there is no change between summer and winter time. If applied in german context (as I did) this change should be adressed because otherwise the SLP H0 is off one hour during the summer time period.

I hope I did not miss any documentation on this, but if I did, please let me know!

To Reproduce

# %%

from demandlib import bdew
import pandas as pd
import holidays  # pip install holidays


pd.options.plotting.backend = "plotly"
demandlib = bdew.ElecSlp(
    year=2024,
    holidays=holidays.Germany(years=2024)
).create_dynamic_h0_profile() * 100000 # scale profileso we do not have such small values

# inspect the DateTimeIndex time zone information (there is none)
print(demandlib.index.tz)  # prints "None"

# plot demandlib SLP H0 from 2024-03-27 to 2024-04-03
fig1 = demandlib.loc['2024-03-27':'2024-04-03'].plot()
fig1.update_layout(
    title='DemandLib SLP H0 (original DateTimeIndex)',
    xaxis_title='Time',
    yaxis_title='Value',
    legend=dict(x=0.01, y=0.99)
)
fig1.show()

# get the peak time on 2024-03-27 (Wednesday in winter time) and 2024-04-02 (Wednesday in summer time)
peak_time_winter = demandlib.loc['2024-03-27'].idxmax()
print(f"Peak time on 2024-03-27: {peak_time_winter}")
# Prints: Peak time on 2024-04-02: 2024-04-02 19:45:00

peak_time_summer = demandlib.loc['2024-04-02'].idxmax()
print(f"Peak time on 2024-04-02: {peak_time_summer}")
# Prints: Peak time on 2024-04-02: 2024-04-02 19:45:00

# %%

# now we a apply a DateTimeIndex which is TZ-aware for Germany
dt_index = pd.date_range(start="2024-01-01", end="2025-01-01", freq="15min", tz="Europe/Berlin")
dt_index = dt_index[:len(demandlib)]

demandlib.index = dt_index
# inspect the DateTimeIndex time zone information (now we see the timezone)
print(demandlib.index.tz)  # prints "Europe/Berlin"

# plot demandlib SLP H0 from 2024-03-27 to 2024-04-03
fig1 = demandlib.loc['2024-03-27':'2024-04-03'].plot()
fig1.update_layout(
    title='DemandLib SLP H0 (original DateTimeIndex)',
    xaxis_title='Time',
    yaxis_title='Value',
    legend=dict(x=0.01, y=0.99)
)
fig1.show()

# get the peak time on 2024-03-27 (Wednesday in winter time) and 2024-04-02 (Wednesday in summer time)
peak_time_winter = demandlib.loc['2024-03-27'].idxmax()
print(f"Peak time on 2024-03-27: {peak_time_winter}")
# Prints: Peak time on 2024-03-27: 2024-03-27 19:45:00+01:00

peak_time_summer = demandlib.loc['2024-04-02'].idxmax()
print(f"Peak time on 2024-04-02: {peak_time_summer}")
# Prints: Peak time on 2024-03-27: 2024-04-02 20:45:00+02:00

# now we see that the peak time on 2024-04-02 changed from 19:45:00+01:00 to 20:45:00+02:00
# but we would expect it to be 19:45:00+02:00

Expected behavior
When comparing to non-holiday wednesdays between summer and winter time I would expect the evening peak to be at the same time (e.g. 19:45). Currently as the code example shows the peak in winter time is 19:45+01:00 and the peak in summer time is 20:45+02:00.

Screenshots

[p-snft]

Thanks for bringing that up. Looking at our implementation, we neglect time zones and just assume local time is given. And reading through the definitions of BDEW, I assume that, eg. the value for 12 pm is the value for 12 pm independent from the day of the year. I think, the proper way to work with this is switching to timezone-aware time stamps. This would also require setting a "target" time zone for the profiles. For example, it would make sense to give a time index in UTC but demand profiles for Germany (including DST).

At the same time, in your plot it does not look like the 31st of March really is one hour longer then the other days. (I might be mistaken, but look.)

In the end, I think both implementations have their problems: By neglecting DST, we do not have the extra hour in spring and the missing hour in fall. Your implementation shifts the whole profile by one hour in summer, which is also not intended (or only correct if you replace the time index before using it). In your plot having a continuous time axis (with constant time zone), I would rather expect the blue line be shifted to the right with respect to the red demandlib one.

PS: Technically, the profiles should only be relevant for Germany.

[joemfe]

Thank you for your quick reply!

As for you comment on the plot. The 31st of March actually is one hour shorter because the clock jumps from 1:59 to 3:00. I also double checked and found that the index is correct.

I agree that the timestamps are correct if you look at each day individually in a way that for example the peak on every sunday is at 12:00 noon throughout the whole year. But if you look at the annual profile as a whole it is in my opinion not correct (especially if it is only relevant for Germany) because the shift to and from DST should also shift the profile.

My solution for now is to create a proper tz-aware index and then clip the extra hour in March and insert it in October.

Maybe if there is time you could implement a optional parameter for create_dynamic_h0_profile() to return a Series indexed with tz-aware Timestamps in "Europe/Berlin" timezone where the shift is considered.

[p-snft]

Sorry, you are right. You get up one hour early in summer. What I wanted to say: in your plot, every day seems to have the same length.

Thinking about it, I now interpret your time axis as given in central European standard time, the axis of demandlib is in local time (switches from standard time to DST) but incorrectly adds/removes hours so that every day has exactly 24 hours. Probably, always returning a timezone-aware index would solve the problem you mention here. An optional parameter would be used to set a different time zone. ("Europe/Berlin" would be a meaningful standard for BDEW.)

At the same time, the new BDEW25Profiles, accept a pd.DatetimeIndex. Should we enforce timezone-aware time indexes (and possibly a parameter to offset the time series)? Seems like this was the only clean way to do it.

[joemfe]

I came up with a quick snippet illustrating my thoughts and also what you elaborated on in your last reply which should serve as a basis for a common understanding of my original issue.

from demandlib import bdew
import holidays  # pip install holidays
import pandas as pd

demandlib = bdew.ElecSlp(
    year=2024,
    holidays=holidays.Germany(years=2024)
).create_dynamic_h0_profile() * 100000 # scale profileso we do not have such small values

print(demandlib.index[:5])

# get the time of the maximum power demand every sunday throughout the year
sundays = demandlib[demandlib.index.weekday == 6]
time_of_max_value_per_sunday = sundays.groupby(sundays.index.date).idxmax()
number_of_timestamps_per_sunday = sundays.groupby(sundays.index.date).count()

# the peak is always at 12:00 (noon), which is correct for each day on its own
print('Distinct timestamps of max power demand on all sundays')
print(time_of_max_value_per_sunday.dt.time.value_counts())  # 12:00:00    52 times for 52 weeks


# here we should see 96 (regular sunday with 24h), 92 (shift to DST in March), 100 (shift from DST in October)
# the shift always happends on a sunday
print('Distinct Number of timestamps on all sundays')
print(number_of_timestamps_per_sunday.unique())  # [96]


# %%

# now let's replace the index with a tz-aware index for Europe/Berlin
dt_index = pd.date_range(start="2024-01-01", end="2024-12-31T23:45", freq="15min", tz="Europe/Berlin")
demandlib.index = dt_index

# now we see the index is tz-aware and time jumps from 01:59 to 03:00 on 31st of March
print(demandlib.index[:5])
print(demandlib.loc['2024-03-31'].head(12).index)

# get the time of the maximum power demand every sunday throughout the year
sundays = demandlib[demandlib.index.weekday == 6]
time_of_max_value_per_sunday = sundays.groupby(sundays.index.date).idxmax()
number_of_timestamps_per_sunday = sundays.groupby(sundays.index.date).count()

# now the peaks are at 12:00 (noon) and 13:00, which is incorrect for the DST period
print('Distinct timestamps of max power demand on all sundays')
print(time_of_max_value_per_sunday.dt.time.value_counts())  # 13:00:00 30 times (during DST) and 12:00:00 22 times (rest of the year)


# now we get the expected result mentioned above
print('Distinct Number of timestamps on all sundays')
print(number_of_timestamps_per_sunday.unique())  # [ 96  92 100]

And here is the output:

I would appreciate if you'd adress this issue in the future :)

[p-snft]

Overwriting the time axis of course will produce wrong results. The time series was calculated using a time zone naive index. The only way to address the issue is by properly converting the data to a timezone-aware index (and information about the location). This would include deleting points in time that do not exist in reality (as 2:30 at the last Sunday in March) and duplicating other values (as 2:30 at the last Sunday in October).

The question is whether we should do this internally or not. I just suggested this question for the upcoming developer meeting.

[joemfe]

Thanks a lot for passing on the issue 👍