Feature request: xdas open behaviour

[yetinam]

Hi @atrabattoni ,
thanks for the great work with xdas! Using it for SeisBench, I've found a few things that might make user comfort about the reading function better, so I wanted to start a discussion.

• I was wondering, if it would be possible to automatically infer the engine for open_datarray? For users who know their data well, it's no problem specifying the correct engine, but for users who got data second-hand (like me), it's always a bit of a struggle to figure out. Looking at the different data types, it's probably pretty easy to develop a routine that identifies the engine based on the available attributes?
• Is there a reason why open_dataarray and open_mfdataarray are two separate functions? From a user perspective, I have the same intent when calling either of them. The change is just in the backend. Maybe they could be unified into an open function. The obspy read function works this way.
• It might be useful to allow the open function to work as a context manager. That would especially handle the file pointers more smoothly and fit closely with the way the underlying h5py library works. The lifecycle of the data array would then be clear. This could also be complimented with an xdas.read function, that does the same as open with the difference that it directly reads data into memory and is therefore not a context manager. The code could look something like this:

# Option with open
with xdas.open("mydasdata.nc") as da:
    # da is a virtual array
    picks = model.classify(da)  # Some processing call

# Option with read
da = xdas.read("mydasdata.nc")  # da is directly read into memory, no context manager required

As I said in the beginning, these are just suggestion that I think might make the file reading more smooth. I totally understand that their might be reasons against this (or that you might not actually have the capacity to implement this).

[atrabattoni]

Hi @yetinam,

Thanks for the great feedback! It's also great to hear that using Xdas as the back-end DAS of Seisbench is working out.

• Automatically inferring the engine is totally doable. I opened PR Automatic engine detection #62 to start working on it.
• The reason is that we tried to stick as closely as possible to the Xarray API. However, we could definitely create a open function that automatically switches between the different open_* functions. I opened PR Automatic open function #63 for that.
• This third point is less clear to me. I must confess that I usually don't care about closing virtual datasets in my scripts. Have you encountered situations where this was necessary? I'm not against adding a read(...) function that is essentially open(...).load(). Though it looks like good things to have, and it should not be too difficult to develop. Let's give it a try. I will open a PR when the two previous are done to avoid any conflict :).

Thanks again for taking the time to write down those potential improvements. It's really helpful!

[atrabattoni]

A went a bit through the code base, for the context manager part I need to add a close method to most Xdas objects. We will do that in a second step.

[yetinam]

Hi @atrabattoni ,
thanks for the quick (and detailed) reply! For your question on the context manager, I've indeed run into issues. I wrote a unit test that writes a file before immediately reading it back in and that sometimes failed on mac, so I had to wrap it with some failure logic. In that case, it's actually not the reading but the to_netcdf that's causing the trouble, but I think something similar could happen if the open file pointers stick around until the garbage collection picks them up.

[atrabattoni]

Roadmap:

• Create generic xdas.open function.
• Enable automatic engine detection.
• Improve lifecycle management of data arrays and collections.

[atrabattoni]

I've juste merged #63. We now have a generic xdas.open function in the dev branch.

[atrabattoni]

Hi @atrabattoni , thanks for the quick (and detailed) reply! For your question on the context manager, I've indeed run into issues. I wrote a unit test that writes a file before immediately reading it back in and that sometimes failed on mac, so I had to wrap it with some failure logic. In that case, it's actually not the reading but the to_netcdf that's causing the trouble, but I think something similar could happen if the open file pointers stick around until the garbage collection picks them up.

Yeah ok I see, I will delve more carefully into data lifecycle management.