-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Description
Right now we have a heavily modified fork of zarrs_tools
. A couple of things need to be upstreamed:
- Environment management.. There is no command for how to create an env, what packages are needed etc.
- System specifics API for swapping out commands for timing and cache busting (and anything else that might be system-specific)
- Small Datasets + CI The benchmarks probably should run on CI at least in a small case. I'm had a lot of trouble getting them to work on both Rocky Linux and my ubuntu. Rocky Linux did not like the
Popen
(I had to pass inshell=True
) and ubuntu was complaining about CLI downcasting - Parallelization Currently, the
zarrs-python
is parallelized by default so benchmark is a bit wonky and not really "fair." See this issue.