Make debugging easier

[mikapfl]

The motivation

As a data curator, I want to run notebooks as IPython notebooks or Python files, using the debugging features of my IDE.

The problem

In the standalone-producer-repo pattern, neither the jupytext notebook in src/ nor the rendered notebooks in dist/ (available after running a book via notebook run) can be run as-is because they won't find their data.

The proposed solution

Stop guessing around so much about where the data is. Relative file paths in yaml files should always be relative to the yaml file, not relative to some arbitrary other location.

Alternatives

Provide a notebook_dir parameter for download_file so that I can explicitly tell bookshelf to stop guessing by providing notebook_dir=".".

Additional context

Rant: papermill was a mistake, and bookshelf is therefore built on mistakes

Maybe I'm overreacting, but I think this points to the wider problem of "too much complexity", which I always suspect whenever people run ipython notebooks non-interactively. Ipython was developed for data vis and running things interactively, and it is marginally better than simple python at it. As soon as you want to run something in a pipeline, IMHO you should just turn it into python functions which can be passed around, debugged, sliced and diced. I spent the better part of a day debugging and setting up something which effectively reads a CSV and writes out a slightly mutated CSV together with some metadata which is read straight from a yaml file.

[mikapfl]

Possible test case:

$ pwd
/home/mikapfl/work/bookshelf-ndcs-cr/src
$ ls
data/  ndcs-cr.py  ndcs-cr.yaml
$ uv run python ndcs-cr.py
Traceback (most recent call last):
  File "/home/mikapfl/work/bookshelf-ndcs-cr/src/ndcs-cr.py", line 32, in <module>
    metadata = load_nb_metadata("ndcs-cr", version=version)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mikapfl/work/bookshelf-ndcs-cr/.venv/lib/python3.11/site-packages/bookshelf_producer/notebook.py", line 98, in load_nb_metadata
    config, raw_data = _load_nb_config(name, nb_directory)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mikapfl/work/bookshelf-ndcs-cr/.venv/lib/python3.11/site-packages/bookshelf_producer/notebook.py", line 54, in _load_nb_config
    raise FileNotFoundError(f"Could not find {metadata_fname}")
FileNotFoundError: Could not find /home/mikapfl/work/bookshelf-ndcs-cr/.venv/notebooks/ndcs-cr.yaml

[lewisjared]

I agree that relative data handling leaves a lot to be desired. I've only had to use it once or twice so never put much effort into supporting it.

Most of the time I work in my IDE and debug directly or jupyter notebook if I need to create/check figures. That works well if remote data is needed

[znichollscr]

I love the hot take. I basically agree. The way we solved this in https://github.com/climate-resource/CMIP-GHG-Concentration-Generation is complete (in the sense that you can't lose your data, and it's very obvious where the tool tried to look if things do go wrong) but definitely not simple. It basically boils down to writing a config file with full paths, before running anything else. This means you have an extra step of writing the config file with full paths, but that's trivial to automate.

There is some balance between sane defaults and making people realise that where they put their data does matter and is coupled to a whole bunch of configuration that expects data to be in a certain place i.e. follow the rules. For bookshelf, sane defaults are probably the way to go because it's such a simple thing. Mika's test case is a good one I think.

Looking at it, it also seems to me that looking for yaml files relative to the notebook dir is a mistake, especially if we know we're moving those notebooks around between src and dist. We could probably guess, but I think there are saner ways to guess and yes, I would not couple this to something that we know we are going to move.

Relative file paths in yaml files should always be relative to the yaml file

I think this would be a good convention.

IMHO you should just turn it into python functions which can be passed around, debugged, sliced and diced

For what it's worth, I really like the notebook-based flow. Having the figures right next to the output has really helped me on multiple occasions and being able to just drop into an interactive session for debugging (with all paths set correctly, admittedly) has been great for me. I agree though, if the paths weren't set correctly, I would go nuts.

[mikapfl]

drop into an interactive session for debugging

I guess that is the main thing where I disagree because I find debugging notebooks actually harder than debugging pure python scripts, mostly due to pycharm's debugger. But yeah, having the figures right there is of course the main advantage, and debugging in an ipython session is also acceptable. So if figures are a main thing in a system, I can see the reasons to like notebook workflows (back when I did lots of figures, I still converged on a python scripts workflow mostly, with one script per figure, the script my_awesome_figure.py then produces my_awesome_figure.pdf).

[znichollscr]

I guess that is the main thing where I disagree because I find debugging notebooks actually harder than debugging pure python scripts, mostly due to pycharm's debugger

If we save our notebooks with jupytext, the notebooks are python scripts no?

I still converged on a python scripts workflow mostly, with one script per figure, the script my_awesome_figure.py then produces my_awesome_figure.pdf

Interesting. I found this flow had way too long an iteration time for me, so would not want that to be the only option.

[znichollscr]

@mikapfl one other use case for notebooks: HTML rendering. For most things it doesn't matter, but for xarray objects it is very nice and I assume not something that is supported within a Python/IPython environment?

[mikapfl]

for xarray objects it is very nice and I assume not something that is supported within a Python/IPython environment?

The pycharm debugger does a nice job rendering python objects for introspection. xarray support in particular is not perfect, but serviceable.