Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sen2VM Release history

## Merged in main (1.3.0.rc2)
## Merged in main

### Main feature

Expand All @@ -10,6 +10,28 @@

*

* Feature:

*

* Doc:

*

## 1.3.0 (2026-07-17)

### Main features

* Allow to have mosaic DEM and not only cut in square degrees
* Notebooks improvments
* GIPP handling more robust

### Updates

* Fix:

* Update GIPP handling (#70)

* Feature:

* Handle mosaic DEM (multiple square degrees): #62
Expand All @@ -23,6 +45,9 @@
* Creation of a Notebook for Inverse location (#54)
* Allow using OTB instead of gdal in direct location Notebook (through a new notebook)

* Change input description related to DEM and GIPP
* Addition of Unit test for GIPP handling (#51, #70)


## 1.2.0 (2026-05-04)

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM ghcr.io/sen2vm/sen2vm-build-env:latest AS launcher

ENV SEN2VM_VERSION=1.3.0.rc2
ENV SEN2VM_VERSION=1.3.0

WORKDIR /Sen2vm

Expand Down
2 changes: 1 addition & 1 deletion documentation/Input/DEM_CDSE_Download.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@

Digital Elevation Models (DEM) are essential for geolocation and orthorectification in Sen2VM. Users can use different types of DEM (cf [Inputs Description](../Input/input_description.md)).

Sen2VM requires DEM data **organized per square degree** (see §[DEM format requirements](../Input/input_description.md#131-dem)) but can now also handle mosaic of square degrees. The recommended way to obtain Copernicus DEM in the correct format is to use the **[CDSE-Copernicus-DEM-downloader](https://github.com/senbox-org/CDSE-Copernicus-DEM-downloader)** tool, which downloads individual 1°×1° geocells from the Copernicus Data Space Ecosystem.
Sen2VM requires DEM data **organized per square degree** (see §[DEM format requirements](../Input/input_description.md#131-dem)) but can now also handle **mosaic of square degrees**. The recommended way to obtain Copernicus DEM in the correct format is to use the **[CDSE-Copernicus-DEM-downloader](https://github.com/senbox-org/CDSE-Copernicus-DEM-downloader)** tool, which downloads individual 1°×1° geocells from the Copernicus Data Space Ecosystem.

---

Expand Down
3 changes: 0 additions & 3 deletions documentation/Input/input_description.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,9 +146,6 @@ The GIPP folder does not require a specific structure; the system searches throu

If only .tar or .tar.gz archives of the GIPPs are available, the archives are extracted.

> [!CAUTION]
> Without automatic GIPP selection, the GIPP folder should contain only the required GIPP files. No archive extraction is performed in this mode.

The GIPP required are the following ones:
* **GIP_VIEDIR**: contains Viewing Direction required by Rugged to create viewing model based on TAN_PSI_X/Y_LIST tags. There is one GIP_VIEDIR file **per band** and each file contains information per **detector** (in the following tags: _[DATA/VIEWING_DIRECTIONS_LIST/VIEWING_DIRECTIONS/TAN_PSI_X_LIST]_ and _[DATA/VIEWING_DIRECTIONS_LIST/VIEWING_DIRECTIONS/TAN_PSI_Y_LIST]_)
* **GIP_SPAMOD**: contains transformations to apply to viewing direction from tags, available in the _[DATA]_ field:
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<groupId>esa.sen2vm</groupId>
<artifactId>sen2vm-core</artifactId>
<!-- NOTE: remind to change also the SEN2VM_VERSION in the Dokerfile-->
<version>1.3.0.rc2</version>
<version>1.3.0</version>

<name>sen2vm-core</name>
<url>https://github.com/sen2vm/sen2vm-core</url>
Expand Down
11 changes: 7 additions & 4 deletions sen2vm-notebook/src/notebook-direct-grid-otb.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"# /!\\/!\\/!\\ If you do not have your own GIPP folder, you may use the inputs-download-notebook to download it, then you may indicate the path were you \n",
"# /!\\/!\\/!\\ downloaded it here in PATH_GIPP\n",
"# /!\\/!\\/!\\ If you have your own GIPP folder, please note that this current notebook will search for a subfolder with mission S2[A/B/C] inside the GIPP folder\n",
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"\n",
"# Path to DEM directory \n",
Expand Down Expand Up @@ -275,7 +274,12 @@
"\n",
"docker_l1b = \"/data/L1B\"\n",
"docker_dem = \"/data/DEM\"\n",
"docker_gipp = f\"/data/GIPP/{mission}\"\n",
"\n",
"if os.path.isdir(os.path.join(PATH_GIPP, mission)):\n",
" docker_gipp = f\"/data/GIPP/{mission}\" \n",
"else:\n",
" docker_gipp = f\"/data/GIPP/\"\n",
"\n",
"# =====================================================\n",
"# 3. Geoid management\n",
"# =====================================================\n",
Expand All @@ -294,7 +298,6 @@
"\n",
"if os.path.isfile(PATH_IERS):\n",
" iers_host = PATH_IERS\n",
" PATH_IERS = os.path.dirname(PATH_IERS)\n",
"elif os.path.isdir(PATH_IERS):\n",
" for f in os.listdir(PATH_IERS):\n",
" if f.startswith(\"bulletin\"):\n",
Expand Down Expand Up @@ -484,7 +487,7 @@
" \"-v\", f\"{PATH_DEM}:/data/DEM\",\n",
" \"-v\", f\"{PATH_GIPP}:/data/GIPP\",\n",
" \"-v\", f\"{PATH_GEOID}:/data/GEOID\",\n",
" \"-v\", f\"{PATH_IERS}:/data/IERS\",\n",
" \"-v\", f\"{iers_host}:{docker_iers}\",\n",
" \"-v\", f\"{WORKDIR}:/workspace\",\n",
" \"sen2vm\",\n",
" \"-c\", config_inside,\n",
Expand Down
11 changes: 7 additions & 4 deletions sen2vm-notebook/src/notebook-direct-grid.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"# /!\\/!\\/!\\ If you do not have your own GIPP folder, you may use the inputs-download-notebook to download it, then you may indicate the path were you \n",
"# /!\\/!\\/!\\ downloaded it here in PATH_GIPP\n",
"# /!\\/!\\/!\\ If you have your own GIPP folder, please note that this current notebook will search for a subfolder with mission S2[A/B/C] inside the GIPP folder\n",
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"\n",
"# Path to DEM directory \n",
Expand Down Expand Up @@ -275,7 +274,12 @@
"\n",
"docker_l1b = \"/data/L1B\"\n",
"docker_dem = \"/data/DEM\"\n",
"docker_gipp = f\"/data/GIPP/{mission}\"\n",
"\n",
"if os.path.isdir(os.path.join(PATH_GIPP, mission)):\n",
" docker_gipp = f\"/data/GIPP/{mission}\" \n",
"else:\n",
" docker_gipp = f\"/data/GIPP/\"\n",
"\n",
"# =====================================================\n",
"# 3. Geoid management\n",
"# =====================================================\n",
Expand All @@ -294,7 +298,6 @@
"\n",
"if os.path.isfile(PATH_IERS):\n",
" iers_host = PATH_IERS\n",
" PATH_IERS = os.path.dirname(PATH_IERS)\n",
"elif os.path.isdir(PATH_IERS):\n",
" for f in os.listdir(PATH_IERS):\n",
" if f.startswith(\"bulletin\"):\n",
Expand Down Expand Up @@ -487,7 +490,7 @@
" \"-v\", f\"{PATH_DEM}:/data/DEM\",\n",
" \"-v\", f\"{PATH_GIPP}:/data/GIPP\",\n",
" \"-v\", f\"{PATH_GEOID}:/data/GEOID\",\n",
" \"-v\", f\"{PATH_IERS}:/data/IERS\",\n",
" \"-v\", f\"{iers_host}:{docker_iers}\",\n",
" \"-v\", f\"{WORKDIR}:/workspace\",\n",
" \"sen2vm\",\n",
" \"-c\", config_inside,\n",
Expand Down
11 changes: 7 additions & 4 deletions sen2vm-notebook/src/notebook-inverse-grid.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,6 @@
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"# /!\\/!\\/!\\ If you do not have your own GIPP folder, you may use the inputs-download-notebook to download it, then you may indicate the path were you \n",
"# /!\\/!\\/!\\ downloaded it here in PATH_GIPP\n",
"# /!\\/!\\/!\\ If you have your own GIPP folder, please note that this current notebook will search for a subfolder with mission S2[A/B/C] inside the GIPP folder\n",
"# /!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\/!\\\n",
"\n",
"# Path to DEM directory \n",
Expand Down Expand Up @@ -274,7 +273,12 @@
"\n",
"docker_l1b = \"/data/L1B\"\n",
"docker_dem = \"/data/DEM\"\n",
"docker_gipp = f\"/data/GIPP/{mission}\"\n",
"\n",
"if os.path.isdir(os.path.join(PATH_GIPP, mission)):\n",
" docker_gipp = f\"/data/GIPP/{mission}\" \n",
"else:\n",
" docker_gipp = f\"/data/GIPP/\"\n",
"\n",
"# =====================================================\n",
"# 3. Geoid management\n",
"# =====================================================\n",
Expand All @@ -293,7 +297,6 @@
"\n",
"if os.path.isfile(PATH_IERS):\n",
" iers_host = PATH_IERS\n",
" PATH_IERS = os.path.dirname(PATH_IERS)\n",
"elif os.path.isdir(PATH_IERS):\n",
" for f in os.listdir(PATH_IERS):\n",
" if f.startswith(\"bulletin\"):\n",
Expand Down Expand Up @@ -501,7 +504,7 @@
" \"-v\", f\"{PATH_DEM}:/data/DEM\",\n",
" \"-v\", f\"{PATH_GIPP}:/data/GIPP\",\n",
" \"-v\", f\"{PATH_GEOID}:/data/GEOID\",\n",
" \"-v\", f\"{PATH_IERS}:/data/IERS\",\n",
" \"-v\", f\"{iers_host}:{docker_iers}\",\n",
" \"-v\", f\"{WORKDIR}:/workspace\",\n",
" \"-v\", f\"{OUTPUT_FOLDER}:/output\",\n",
" \"sen2vm\",\n",
Expand Down
39 changes: 35 additions & 4 deletions src/main/java/esa/sen2vm/input/gipp/GIPPFileManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ public static List<File> searchGIPFilesFromRegex(Path root, String dirNameRegex,
final Pattern dirPattern = Pattern.compile(dirNameRegex);
final Pattern filePattern = Pattern.compile(fileNameRegex);
final List<File> results = new ArrayList<>();
final List<String> tarExtension = Arrays.asList("TGZ", "tar.gz","tgz");
// Stack indicating whether we are currently in a qualified subtree
final Deque<Boolean> qualifiedStack = new ArrayDeque<>();
Files.walkFileTree(root, new SimpleFileVisitor<Path>() {
Expand Down Expand Up @@ -129,7 +130,30 @@ public FileVisitResult visitFile(Path filePath, BasicFileAttributes attrs) {
if (inQualifiedSubtree || filePattern.matcher(fileName).matches()) {
File file = filePath.toFile();
String extension = getFileExtension(file);
if(validExtensions.stream().anyMatch(item -> item.contains(extension)))

if(tarExtension.stream().anyMatch(item -> item.contains(extension)))
{
try
{
List<Path> listPath = UntarGIPP.untarGz(file.toPath(), Paths.get(file.getParent()));
for(Path untarPath:listPath)
{
File untarFile = untarPath.toFile();
String untarFileExtension = getFileExtension(untarFile);
if(validExtensions.stream().anyMatch(item -> item.contains(untarFileExtension)))
{
results.add(untarFile);
}
}
LOGGER.info("Untar GIPP: "+file.toString());
}
catch(IOException e)
{
LOGGER.warning("The targz extraction of GIPP has failed: "+file.toString());
e.printStackTrace();
}
}
else if(validExtensions.stream().anyMatch(item -> item.contains(extension)))
{
results.add(file);
}
Expand Down Expand Up @@ -265,9 +289,16 @@ public static List<String> typedGIPPList(List<String> gippList, String gippType)
*/
public static File findGippFile(Path root, String dirNameRegex, List<String> gippList, String fileNameRegex, List<String> validExtensions) throws IOException, Sen2VMException {
final List<File> results = findGippFiles(root, dirNameRegex, gippList, fileNameRegex, validExtensions);
if(results.size()==0)
if(results.isEmpty())
{
throw new Sen2VMException("The directory must be contains keyword: "+fileNameRegex);
if (gippList.isEmpty())
{
throw new Sen2VMException("GIPP directory(ies) must contains at list one GIPP of type: " + fileNameRegex);
}
else
{
throw new Sen2VMException("GIPP directory(ies) for GIPP type " + fileNameRegex + " must match GIPP list from Datastrip Metadata: " + Arrays.toString(gippList.toArray()));
}
}
else if(results.size()>1)
{
Expand All @@ -276,7 +307,7 @@ else if(results.size()>1)
.map(File::getAbsolutePath)
.collect(Collectors.joining(", "));

LOGGER.info("GIPP "+dirNameRegex+" : "+message);
LOGGER.info("GIPP " + dirNameRegex + " : " + message);
throw new Sen2VMException("Duplicate GIPP file type found");
}
else
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/esa/sen2vm/input/gipp/GIPPManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ protected void loadAllGIPP(List<BandInfo> bands) throws Sen2VMException
}
catch (Exception e)
{
throw new Sen2VMException("Error when reading the blind pixel GIPP file: " + fileBlindPixel, e);
throw new Sen2VMException("Error when reading the blind pixel (GIP_BLINDP) GIPP file: " + fileBlindPixel + ": " + e.getMessage(), e);
}

// Load spacecraft model gipp
Expand All @@ -179,7 +179,7 @@ protected void loadAllGIPP(List<BandInfo> bands) throws Sen2VMException
}
catch (Exception e)
{
throw new Sen2VMException("Error when reading spacecraft model GIPP file: " + fileSpaMod, e);
throw new Sen2VMException("Error when reading spacecraft model (GIP_SPAMOD) GIPP file: " + fileSpaMod + ": " + e.getMessage(), e);
}

// Load viewing directions gipp
Expand All @@ -191,7 +191,7 @@ protected void loadAllGIPP(List<BandInfo> bands) throws Sen2VMException
File file = gippFilePathFromIndexBand(bands.get(i), gippFilePathList);
if (file == null)
{
throw new Sen2VMException("Viewing directions GIPP file missing for band "+ bands.get(i));
throw new Sen2VMException("Viewing directions (GIP_VIEDIR) GIPP file missing for band "+ bands.get(i));
}

// Load GIPP DATA
Expand All @@ -211,7 +211,7 @@ protected void loadAllGIPP(List<BandInfo> bands) throws Sen2VMException
}
catch (Exception e)
{
throw new Sen2VMException("Error when reading viewing directions GIPP files from", e);
throw new Sen2VMException("Error when reading viewing directions GIPP (GIP_VIEDIR) files: " + e.getMessage(), e);
}

// Load prdloc model gipp, only for RAW mode
Expand Down
Loading
Loading