[Bug]: Partial backscatter returned at Antimeridian (Mask bug)

[abradley60]

Checked for duplicates

Yes - I've already checked

Describe the bug

Summary

• There appears to be a bug where partial backscatter data is returned for bursts that intersect the antimeridian / dateline (180 degrees).
• From my investigation, The bug is likely associated with the generation of the shadow layover mask.
• The bug impacts sentinel-1 scenes. I am unsure if this bug would also impact the processing of Nisar scenes over the antimeridian and therefore be high on your priority list to address.

Details

The bursts I have tested are at high latitude near the Antarctic coastline. I have tested a couple different sentinel-1 SLC’s in the region, finding similar results. The following is the produced backscatter for two bursts t028_059508_iw1 and t028_059507_iw2 corresponding to S1A_IW_SLC__1SSH_20220122T153615_20220122T153645_041575_04F1EF_9254. The image below shows the resulting backscatter plotted in QGIS using 3031 polar stereographic projection. Artifacts either side of the antimeridian appear for both bursts.

Figure 1 – HH Backscatter for t028_059508_iw1 and t028_059507_iw2 for scene S1A_IW_SLC__1SSH_20220122T153615_20220122T153645_041575_04F1EF_9254 over the antimeridian.

Further analysis suggests to me that this may be an issue with the generation of the shadow and layover mask. Below shows the masks plotted in QGIS. As can be seen, there are some interesting striping artifacts showing layover, and large blocks either side of the antimeridian appear to be impacted by both shadow and layover. From what I can see, the generation of the other ancillary layers is normal.

Figure 2 – Data masks for bursts showing artifacts.

When we set apply_shadow_masking : False, complete data is returned for the bursts as expected. I have not spent time diving into the isce3 code, but I suspect the culprit is associated with compute_layover_shadow_mask, and potentially the Ellipsoid?

Figure 3 – Backscatter bursts with NO masking applied

There is full DEM coverage for the bursts that have been tested. Given the high latitude of the test areas, the DEM is passed in using EPSG:3031 Polar Stereographic Coordinates. The bursts are actually over water, so ellipsoid heights are used.

Files

• Backscatter, masks, runconfigs – https://deant-data-public-dev.s3.ap-southeast-2.amazonaws.com/index.html?prefix=persistent/rtc-bugfix/antimeridian/s1_rtc_c1/
• Static / Ancillary layers – https://deant-data-public-dev.s3.ap-southeast-2.amazonaws.com/index.html?prefix=persistent/rtc-bugfix/antimeridian/s1_rtc_static_c1/
• DEM - https://deant-data-public-dev.s3.ap-southeast-2.amazonaws.com/persistent/rtc-bugfix/antimeridian/S1A_IW_SLC__1SSH_20220122T153615_20220122T153645_041575_04F1EF_9254_dem.tif
• Backscatter without mask applied - https://deant-data-public-dev.s3.ap-southeast-2.amazonaws.com/index.html?prefix=persistent/rtc-bugfix-nomask/antimeridian/s1_rtc_c1/

What did you expect?

Expected nominal processing of the bursts.

Reproducible steps

- Data was produced using `rtc_s1.py OPERA-RTC_runconfig.yaml`. 
- The config and the DEM file for the run can be found in the link above.
- We use a burst-db which was built from https://github.com/opera-adt/burst_db
- Aside from that, the SLC was downloaded from the ASF and the orbit files from ESA’s CDSE

Environment

- Linux environment set up as suggested by the RTC Dockerfile 
- Opera-adt/RTC version = 1.0.2
- s1-reader version = 0.2.4
- isce3==0.15.0

[gshiroma]

Thank you so much, @abradley60 , for reporting this issue and providing the data we need to debug it. We'll keep you posted.

[gshiroma]

@abradley60 , thank you again for reporting this issue and sharing the files. I was able to replicate the error and I can confirm that the layover/shadow mask generated by ISCE3 Topo module in slant-range and before geocoding contain these antimeridian artifacts. What’s surprising is that this bug occurs with a DEM in polar stereographic coordinates. I tried processing it with a different DEM also in polar stereo (EPSG 3031) and I got the same error.

@hfattahi and @gmgunter , this bug probably affects the generation of the layover shadow mask for NISAR as well.

[gshiroma]

@abradley60 , PR #96 was merged which closed this issue. But I'd like to confirm with you that the problem was solved succesfully. The source of the bug is actually in the Topo module of ISCE3, which I'm also fixing. But in the meanwhile, the PR #96 should solve the problem by ensuring that the antimeridian crossing is detected and correctly handled by the Topo module in ISCE3.

[abradley60]

Thanks @gshiroma , I'll rerun my test bursts and confirm #96 solves the problem in the meantime.

[gshiroma]

Awesome. Thank you, @abradley60 !

[abradley60]

Thanks again @gshiroma. I have run some tests, the results are looking good for bursts t028_059508_iw1. The mask is seamless and there is no layover / shadow as expected over the water.

However, I seem to be running into an issue with burstt028_059507_iw2. The processing seems to be hanging on the creation of the mask. I have left it for an hour or two but it does not seem to be progressing past this point:

journal:
 -- Actual DEM bounds used:
 -- Top Left: -12937.5 -2.18728e+06
 -- Bottom Right: 85059.3 -2.28938e+06 
 -- Spacing: 31.0903 -31.0903
 -- Dimensions: 3153 3285
Topo progress (block 2/2): 100%
Layover/shadow mask progress (block 2/2): 96%

I am using the same DEM and scene as I linked in the start of this thread. Note I can see in the logs (linked below) there is a warning journal: East limit may be insufficient for global height range, however on visual inspection the DEM more than covers the burst:

Files

Burst runconfig -> https://deant-data-public-dev.s3.ap-southeast-2.amazonaws.com/persistent/rtc-bugfix/antimeridian-fixed-v1-hanging-burst/OPERA-RTC_runconfig_t028_059507_iw2.yaml
Process Logs -> https://ap-southeast-2.console.aws.amazon.com/s3/object/deant-data-public-dev?region=ap-southeast-2&bucketType=general&prefix=persistent/rtc-bugfix/antimeridian-fixed-v1-hanging-burst/t028_059707_iw2_logs.txt

Environment

Lockfile -> https://github.com/opera-adt/RTC/blob/c5cc7403af16bb67fd14f43bf2aec15c6a422297/Docker/lockfile.lock
RTC (commit with the fix) -> https://github.com/opera-adt/RTC/tree/c5cc7403af16bb67fd14f43bf2aec15c6a422297
s1-reader == 0.2.5
isce3 == 0.15.0 

[gshiroma]

Thank you, @abradley60 , I'm reopening this issue and will investigate what happened.

[gshiroma]

Hi @abradley60 , I'm having a hard time replicating the issue processing the burst t028_059507_iw2. I used the RTC software (main branch), the DEM you provided, and ISCE3 versions 0.15.0 and 0.24.4, and my processing has been successful for t028_059507_iw2 (see plots below). Is it possible that you ran out of memory in your instance?

As for the warning message "journal: East limit may be insufficient for global height range" this is likely related to the additional margin requested by the module loading the DEM (e.g., Topo). If your DEM fully covers the scene, this warning shouldn't affect the outcome of your processing.

[abradley60]

Thanks @gshiroma. Let me change to a larger instance and try again. I'll let you know if that works - sorry for the inconvenience.

[gshiroma]

Thank you, @abradley60 , and no problem at all. It's possible I'm unable to replicate the error because I'm running the software in a different environment.