Study of a regression occurring in the CI following the update to pycdfpp >= 0.9.0

[brenard-irap]

Non-regression tests show a change related to retrieving data for a Cluster parameter from CDAWeb.
You can see an example log here.

And more specifically, the following log line:

FAILED tests/test_cdaweb.py::SpecificNonRegression::test_get_cluster_fgm_data - AssertionError: <speasy.products.variable.SpeasyVariable object at 0x7fd2b0195640> is not None

After some testing, I realized that this appeared when moving from version 0.8.7 to version >= 0.9.0 of pycdfpp.

1. Behavior from speasy

This can be observed by forcing a downgrade of the pycdfpp library:

pip install --force-reinstall "pycdfpp==0.8.7"

After clearing the cache, the following speasy code:

import speasy as spz
result = spz.get_data(spz.inventories.data_tree.cda.Cluster.C1.FGM_SPIN.C1_CP_FGM_SPIN.B_vec_xyz_gse__C1_CP_FGM_SPIN, "2018-03-02", "2018-03-03", disable_proxy=True, disable_cache=True)
print(result)

Produces the following result:

None

And this is actually what the test tests/test_cdaweb.py::SpecificNonRegression::test_get_cluster_fgm_data expects.

Now, if I switch to version 1.9.0 of pycdfpp:

pip install --force-reinstall "pycdfpp==0.9.0"

I obtain a SpeasyVariable:

<speasy.products.variable.SpeasyVariable object at 0x7f95ebb43440>

This SpeasyVariable does not contain any data (which seems normal to me after checking the CDAWeb file):

• result.values => array([], shape=(0, 3), dtype=float32)
• result.time => array([], dtype='datetime64[ns]')

Note that if I switch back to version 0.8.7 of pycdfpp and rerun the same speasy code, I now get:

<speasy.products.variable.SpeasyVariable object at 0x7f2206d49100>

If I clear the cache and run the code again, I get "None" once more.

This is strange because I made the calls with "disable_proxy=True, disable_cache=True", so I wouldn't expect any data to end up in the cache... But that might be another issue.

2. Behavior from pyistp

To reproduce the same behavior from pyistp, I download the data file c1_cp_fgm_spin_20180302_v01.cdf, and run the following code:

import pyistp
istp_loader = pyistp.load(file="c1_cp_fgm_spin_20180302_v01.cdf")
data = istp_loader.data_variable('B_vec_xyz_gse__C1_CP_FGM_SPIN')
print(data.axes[0].values)

With pycdfpp v0.8.7:

array(['1753-08-29T22:43:41.128654848'], dtype='datetime64[ns]')

With pycdfpp v0.9.0:

array(['NaT'], dtype='datetime64[ns]')

When looking at the CDF file, what is returned by version 0.9.0 seems more logical, as the file's time corresponds to the "Pad value":

time_tags__C1_CP_FGM_SPIN (No: 0) (Recs: 1)
-------------------------
Data Type:           CDF_EPOCH
Dimensionality:      0:[]	(T/)  
Pad value:           0000-01-01T00:00:00.000
Written Records:     1/1(max)
Allocated Records:   1/1(max)
Blocking Factor:     1 (records)
Attribute Entries:
     PARAMETER_TYPE  (CDF_CHAR/12): "Support_Data"
     CATDESC         (CDF_CHAR/25): "Interval centred time tag"
     UNITS           (CDF_CHAR/2): "ms"
     SI_CONVERSION   (CDF_CHAR/8): "1.0e-3>s"
     SIGNIFICANT_DIGITS (CDF_DOUBLE/1): 24.0
     FILLVAL         (CDF_EPOCH/1): 9999-12-31T23:59:59.000
     FIELDNAM        (CDF_CHAR/14): "Universal Time"
     LABLAXIS        (CDF_CHAR/2): "UT"
     DELTA_PLUS      (CDF_CHAR/29): "half_interval__C1_CP_FGM_SPIN"
     DELTA_MINUS     (CDF_CHAR/29): "half_interval__C1_CP_FGM_SPIN"
Variable Data:
  Record # 1: 0000-01-01T00:00:00.000

3. Behavior from pycdfpp

By running the following code:

import pycdfpp
cdf = pycdfpp.load("c1_cp_fgm_spin_20180302_v01.cdf")
print(cdf["time_tags__C1_CP_FGM_SPIN"].values)
print(pycdfpp.to_datetime64(cdf["time_tags__C1_CP_FGM_SPIN"].values))

With pycdfpp v0.8.7:

array([(0.,)], dtype=[('value', '<f8')])
array(['1753-08-29T22:43:41.128654848'], dtype='datetime64[ns]')

With pycdfpp v0.9.0:

array([(0.,)], dtype=[('mseconds', '<f8')])
array(['NaT'], dtype='datetime64[ns]')

Between the two versions, there is therefore a difference in the dtype and a difference in the conversion to datetime64.

I stopped my analysis there because there have been quite a few changes in the code between these two versions...

4. Conclusion

The current behavior in Speasy since version 0.9.0 of pycdfpp seems to be better: the data file exists, it contains a "record" but it is equal to the "pad value" => it seems more logical to have an "empty" SpeasyVariable rather than None in my opinion.

However, in this case, the regression test tests/test_cdaweb.py::SpecificNonRegression::test_get_cluster_fgm_data should be updated.