List data gaps in metadata file

[davidjablonski]

Hi,

for our visualisations on klimadashboard.at, we use the klima-v2-1d dataset for a variety of stations. We’re only requesting stations that have a start date before 1900, but even then we’ll often find gaps in the data between the start and end date. As an example, station ID 6 has metadata suggesting operation from 1896 until today, but there are gaps in temperature data f.ex. from 1981-05-01 to 1985-04-30.

The only way to discover these gaps is actually downloading the data repeatedly and checking whether the gaps still exist. It would be great to have data gaps included (and potentially explained) in a more performant metadata file, either by setting an "has_complete_data" boolean value or by including an array of data gaps, each with a start point and end point. This would make it very easy to filter all stations that have complete datasets from date X to date Y.

I raise this issue also because we’re repeatedly encountering timeouts on API requests even when we’re well below the 1000000 limit provided in the documentation. So it’s often not possible to download data for one station from start to finish, but only in smaller chunks, which makes identifying data gaps even more cumbersome.

Any pointers would be much appreciated, thank you!

Greetings,
David

[dlang-geosphereat]

Hi David,

thank you for your request. Data gaps are unfortunately difficult to handle, as there are many reasons why data is not available: data not (yet) digitized, data not (yet) passing quality control checks, certain parameters never being measured, certain sensors being defect, etc. The latter two reasons also mean, that one parameter, e.g. temperature is not available, while others are.

That being said, we do have additional metadata about which parameters were (or should have been) observed, and we are working to make this information available in a user-friendly way. We'll also look into, if/how we can provide information about gaps.

As for timeouts/performance in the API request: I'll look into it.

Best,
Daniel