Support of Envi format despite the eventual presence of aux.xml files

[aloboa]

I fully agree with:

Despite of all drawbacks (in my opinion too many), the ENVI Header file keeps to be a standard in the Imaging Spectroscopy Community to describe image metadata. So we need to support it and be able to take into account information describe in https://www.l3harrisgeospatial.com/docs/enviheaderfiles.html

(jakimwb dixit on 20220910 in #31 (comment))

Envi format should be supported, at least, for reading and display (including all data exploration tools), as
well as for exporting via Translate.

Supporting the envi format means taking into consideration the raw and hdr files only:
aux.xml files are not compatible with the envi format because the user relies on the hdr.
Whenever an image in envi format is displayed in EnMapBox, the eventual aux.xml file should be ignored. If as a consequence of QGIS and/or GDAL oddities an aux.xml file is created for an image in envi format, that file should be automatically deleted when the image is removed or EnMapBox closed.

In case automatic deletion of aux.xml files is not possible or would create worse problems, at least warn the user at reading and writing images in Envi format that he/she must be aware of the eventual presence of aux.xml files that could interfere with the .hdr file.

[janzandr]

I don't like that.
@jakimowb do we need to discuss this?

[jakimowb]

Thanks for creating this issue @aloboa . For the Imaging Spectroscopy community the handling of *.hdr files is an important issue.
We clearly need to discuss and communicate the reational behind our current and future implementation of *.hdr metadata support in the EnMAP-Box. Therefore I transformed @aloboa issue into a discussion topic.

[jakimowb]

I don't like that.

It's always good to explain why.

[janzandr]

It's always good to explain why.

The proposed behaviour goes against GDAL/QGIS data model.

I'm open for supporting ENVI format, but not at the cost of violating the GDAL/QGIS data model.

@jakimowb , do you have already a concept, how you want to implement this?

[aloboa]

If the GDAL/QGIS data model implies giving priority to the aux.xml file over the hdr, then the GDAL/QGIS data model is not fully supporting the Envi format and you have to choose: being respectful of the GDAL/QGIS data model or fully supporting the Envi format.

[janzandr]

Why don't you check that using the ENVI installation on 141.20.141.12 server?

[jakimowb]

do you mean to tell you the actual pixel value being reported by ENVI on these cases?

@aloboa excactly, I like to know how different components of ENVI (API to read raster values, map canvases, cursor location values, ENVI classif vs. modern ENVI) make use of the information stored in the *.hdr.

In particular I like to know in which situations offsets and scale factors are applied (or not).

[jakimowb]

Hi @aloboa

I do not have access to ENVI but will find a colleague. Nevertheless, what do you specifically mean by checking how it "is interpreted when reading by (i) in IDL, (ii) IDL, using the ENVI API, and, using the map visualization in (iii) ENVI Classic and (iv) modern ENVI?" DO you mean to tell you the actual pixel value being reported by ENVI on these cases?

Actually this comes a bit as a surprise to me, probably I have misunderstood you. Based on our earlier discussions, e.g.

Envi format should be supported, at least, for reading and display (including all data exploration tools), as
well as for exporting via Translate.

I assumed that the aim of this ENVI header support discussion is to let the EnMAP-Box react in the same way as ENVI does.
For example, if a pixel value (on disk) is 5000, but ENVI apply a data gain value and shows 0.5 as pixel value, than the EnMAP-Box should do so as well. Therefore I would like to know first where ENVI and the underlying IDL API makes programmatically use of ENVI header file information.

[jakimowb]

Thanks @janzandr, I wasn't aware of that server. If my rather limited time for the EnMAP-Box allows, I'll check it.
(Please don't share company internal IPs / locations on public platforms like this)

[janzandr]

(Please don't share company internal IPs / locations on public platforms like this)

It's not really critical to share your companies IP adress. CMS gives away the 141.20. part in their VPN-FAQs and lots of random intrusion attacks will test your IP anyways.

[aloboa]

I do not have envi myself, so this will take a while. According to janzandr, it is actually easier for you to check.

[janzandr]

FYI @jakimowb , you can install the latest ENVI on your local system and connect to the HU license server via VPN. It's not complicated, just ask Stephan Schulz for the details.

[jakimowb]

FYI: scale & offset in ENVI hdr are soon available via GDAL data model OSGeo/gdal#6444

[Mesadova]

@jakimowb @aloboa @janzandr
Hi, I'm the colleague.

• First of all, ENVI only has the option of "Edit ENVI header" to edit the header file directly, there is not a header visualizer per se.
  For the example file "test.img", the values may be visualized editing the various options described in https://www.l3harrisgeospatial.com/docs/enviheaderfiles.html

i.e. the data offset values read from the header file (data offset values = {5.00000000e+000}) as seen in "Edit ENVI header".

All the values described in the test header file are correctly read in the "Edit ENVI header" visualizer.
The "Edit ENVI header" is found both in ENVI classic and "modern" ENVI.
The IDL opens the header file directly as a text file, as far as I know, there is not an edit header option.

• In the other hand, for the "test.img" image, ENVI classic reads its value of 2.

In "modern" ENVI, there is an error probably due to the image size.

I don't have much experience in the IDL environment, it seems that it cannot open *.img files directly.

Hope to have had helped you,
Cheers

P.S. all ENVI programs are version 5.0

[jakimowb]

Thank you @Mesadova.
That's what I wanted to know. This shows that the ENVI classic cursor location tool does not use the data gain and offset values:

data gain values = {10000}
data offset values = {5}

In contrast, the next QGIS raster layer legend and pixel location will apply them:

[aloboa]

@Mesadova many thanks.
@jakimowb

I do not have access to ENVI but will find a colleague. Nevertheless, what do you specifically mean by checking how it "is interpreted when reading by (i) in IDL, (ii) IDL, using the ENVI API, and, using the map visualization in (iii) ENVI Classic and (iv) modern ENVI?" DO you mean to tell you the actual pixel value being reported by ENVI on these cases?

Actually this comes a bit as a surprise to me, probably I have misunderstood you. Based on our earlier discussions, e.g.

Envi format should be supported, at least, for reading and display (including all data exploration tools), as
well as for exporting via Translate.

I assumed that the aim of this ENVI header support discussion is to let the EnMAP-Box react in the same way as ENVI does.
For example, if a pixel value (on disk) is 5000, but ENVI apply a data gain value and shows 0.5 as pixel value, than the EnMAP-Box should do so as well. Therefore I would like to know first where ENVI and the underlying IDL API makes programmatically use of ENVI header file information.

The ENVI format is used well beyond ENVI itself in many other software packages for HSI processing, as it is the only format able to deal with hyperspectral metadata. It is used, for example, by many hyperspectral cameras. If you come up with a better format for EnMapBox, use it, but create a good import application.
Regarding these particular tests, it is unlikely that another person is going to solve your doubts better than you, thus if you have easy access to ENVI, it will be a lot more efficient that you test it yourself. Otherwise, I think you will end up having to invest more time trying to make you understand and clarifying misunderstandings.

[jakimowb]

Hi @aloboa , no doubt, for historic reasons, the ENVI format is the de-facto standard, supported by various HSI application and we therefore need to support it. @janzandr and me just have to find a good balance between file-format specific support, flexibility and resulting exta hours for development and code-maintenance.

[jakimowb]

However, to my very subjective technical point of view - please correct my where I am wrong- the ENVI header format is meanwhile a very outdated way to store metadata:

1. It does not follow any "industry standard" like XML or JSON, for which each modern programming language provides an I/O library. Instead, everyone has to write its own I/O routine or import an external library. Text editors do not provide special support or syntax highlighting for. This makes manual edits error-prone.
2. Everything that you find in the ENVI Header can be stored in the GDAL metadata model as well. That does not mean that applications make special use of it, but in terms of flexibility, the GDAL model is much more powerful.
3. The ENVI options to describe the physically structure of binary data are rather limited. GeoTIFF, for example, offers tiles and allows to modify the number of bits that is used per pixel. If I have a classificaton with 3 classes only, I can store them in pixels with a size of NBITS=2. This greatly reduces filesizes in big data applications.

[jakimowb]

Another ENVI feature now supported by GDAL: OSGeo/gdal#3876