[FEAT] Make NoData value of downloaded tiff available to python api

[SlowMo24]

What is the problem? Please describe.

In our workflow, we execute an eval_script via a SentinelHubRequest to calculate the NDVI. We then run get_data on that request to retrieve the NDVI data array.

By default the value -999 is used as NoDataValue but this is not clear without trying and we could not find a way to retrieve that value programmatically.

Here's the solution

Return a Numpy Masked Array in the get_data method that can hold that information.

Alternatives

Provide a function to retrieve the NoDataValue.

Additional context

We now set the NoDataValue explicitly in the eval_script and keep a constant in our code that represents that value. If one needed flexibility here, she would probably have to do string-replacement when reading in the JS eval_script because the interaction between python and JS is difficult.

[mlubej]

Hi @SlowMo24!

Can you show the evalscript you are using where you get these -999 default values?

Cheers!

[SlowMo24]

Hy @mlubej , I did some more investigation. It seems like I raised the issue prematurely/on an intermediate state of my code. The default is in fact nan and the get_data method returns just that .... for "FLOAT32"

But this is not the case for "UINT8" where a nan value does not exist. By default, sentinelhub will use 0 as nan and not set the nodataValue of the returned raster. Therefore we have to manually set the nodataValue (in our case 255 because 0 has a meaning). Yet, the python client does not respect the nodataValue exposed by Uint8 rasters (whereas e.g. QGis does for the exact same image).

I'm sorry for the imprecise issue.

Here is some code to reproduce the issue. A masked array may still be an option to investigate but might be an overkill...

import numpy as np
from sentinelhub import SentinelHubRequest, DataCollection, MimeType, SHConfig, BBox, CRS

bbox = BBox((
    9.280732, -0.263633, 9.316524, -0.241747
), crs=CRS("4326"))
config = SHConfig()

evalscript_true_color = """
    //VERSION=3

    function setup() {
        return {
            input: [{
                bands: ["B02"]
            }],
            output: {
                bands: 1
            }
        };
    }

    function evaluatePixel(sample) {
        return [undefined];
    }
"""
input_data = [
    SentinelHubRequest.input_data(
        data_collection=DataCollection.SENTINEL2_L1C,
        time_interval=("2020-06-12", "2020-06-13"),
    ),
]

responses = [SentinelHubRequest.output_response("default", MimeType.TIFF)]

request = SentinelHubRequest(
    evalscript=evalscript_true_color,
    input_data=input_data,
    responses=responses,
    bbox=bbox,
    config=config,
    data_folder='cache'
)

data = request.get_data(save_data=True,)[0]
print(data)
print(type(data))
print(np.unique(data))
print(np.isnan(data).any())

prints

[[0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 ...
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]
 [0 0 0 ... 0 0 0]]
<class 'numpy.ndarray'>
[0]
False

and writes the following file

dbf75d56da4136646a599522d74be1df.zip

Adding nodataValue: 255 to the eval script does not change anything (which may not be the desired output of sentinelhub in general?) but also changing return [undefined]; to return [255]; will at least create a valid geotiff:

5791dc7157b0faf8209310e145e875d7.zip

although sentinelhub-py does not read it correctly:

[[255 255 255 ... 255 255 255]
 [255 255 255 ... 255 255 255]
 [255 255 255 ... 255 255 255]
 ...
 [255 255 255 ... 255 255 255]
 [255 255 255 ... 255 255 255]
 [255 255 255 ... 255 255 255]]
<class 'numpy.ndarray'>
[255]
False

I hope I was able to provide the information required?

[mlubej]

Hi @SlowMo24

As the docs say:

nodataValue (optional) - sets the GDAL nodata metadata tag to the specified value. Only applicable for tiff files.

so you can't expect it to control numpy data.

However, you can set a global NO_DATA value in your evalscript, and also use it to update the array values there, e.g.:

//VERSION=3

var NODATA = 123;

function setup() {
    return {
        input: [{
            bands: ["B02"],
            units: "DN",
        }],
        output: {
            bands: 1,
            sampleType: "INT16",
        }
    };
}

function evaluatePixel(sample) {
    let x = undefined; // or sample.B02;
    return [x == undefined ? NODATA : sample.B02];
}

[SlowMo24]

so you can't expect it to control numpy data.

if that is the design choice the issue can be closed.

But my argument is that any geotiff file reader supports the nodata metadata tag so sentinelhub-py should as well.

[mlubej]

I do agree that we drop information if it's available in the tiff. the only way we could fix it is by improving the tiff data decoder and returning a masked numpy array.

in principle it's all numpy, and masked arrays have basically the same interface, but there are differences, and these differences might break existing applications. @zigaLuksic what do you think?

[mlubej]

Also, @SlowMo24, community-written PRs are appreciated!

[SlowMo24]

Thank you for the invite, I can try a PR in a few weeks. I will though have to get a better understanding of the code first (until now all my comments were purely based on my user experience ;-) ).

Can I assume the decision for the tifffile library in favour of a proper geotiff library like rasterio was on purpose (probably due to GDAL and the size)?

And by the way, is there any change to be expected due to the acquisition by Planet?