Error in sentinel1_sar_interferogram

[mattiacallegari]

Attempting to run sentinel1_sar_interferogram to generate approximately 200 interferograms within a single job frequently results in errors. Distributing the workload across multiple jobs, e.g., processing 20 interferograms per job and executing them simultaneously improves overall stability. However, errors still occur intermittently in some cases, even with this approach, e.g.:

[EmileSonneveld]

@mattiacallegari Can you share a few job-ids of failed jobs? Did this also happen with coherence for example?

[jdries]

I'm wondering if the concurrent job limit is the issue here. Jobs seem to be submitted in a short timeframe...

[mattiacallegari]

@mattiacallegari Can you share a few job-ids of failed jobs? Did this also happen with coherence for example?

j-260423143414431f8d4b401fbc56b206

[EmileSonneveld]

I made 3 fixes and tested the failed jobs, they passed.

• increase memory (some error messages where unclear, what is typically caused by OOM)
• Added a retry to the catalog API query
• Added a retry for the whole gpt process. As DEM and orbit file requests might fail

Script used to re-run the failed process graphs

import json
import os
from pathlib import Path

import openeo
import pandas as pd
from openeo import BatchJob
from openeo.extra.job_management._manager import MultiBackendJobManager
from openeo.extra.job_management._thread_worker import Task


def start_job(
      row: pd.Series, connection: openeo.Connection, **kwargs
) -> openeo.BatchJob:
  print(row)
  file_path: str = row["file_path"]
  datacube = connection.datacube_from_flat_graph(
      json.loads(Path(file_path).read_text())
  )

  return datacube.create_job(
      title=file_path,
      description="Started from " + __file__,
  )


def prepare_jobs_db() -> pd.DataFrame:
  """Prepare a database using pandas, containing job configurations for benchmarking."""
  # This script should be sitting next to a few json files containing process graphs that need to run.
  # For example: `j-260423143414431f8d4b401fbc56b206.json`
  files = [
      filename
      for filename in os.listdir(".")
      if filename.startswith("j-") and filename.endswith(".json")
  ]

  jobs = []
  for file_path in files:
      jobs.append(
          {
              "file_path": file_path,
          }
      )

  return pd.DataFrame(jobs)


jobs_database = prepare_jobs_db()
manager = MultiBackendJobManager()
connection = openeo.connect(
  url="https://openeo.dataspace.copernicus.eu"
).authenticate_oidc()
manager.add_backend("cdse", connection=connection, parallel_jobs=3)

job_db_path = "job_tracker.parquet"
if os.path.exists(job_db_path):
  db = pd.read_parquet(job_db_path)
  # Retry failed jobs
  db.loc[db["status"] == "error", "status"] = "not_started"
  db.loc[db["status"] == "start_failed", "status"] = "not_started"
  db.to_parquet(job_db_path, index=False)
manager.run_jobs(df=jobs_database, start_job=start_job, job_db=job_db_path)

[jdries]

@EmileSonneveld I'm not so keen on random memory increases, I mean, we finetuned costs and also presented it.
It would be better if you could actually post those unclear messages.

[EmileSonneveld]

Yes, blindly increasing is not a great idea. I am re-running the jobs with the original ram now.
The job of j-260423143414431f8d4b401fbc56b206 has a lot of error messages in STAC validation that can be ignored, but no real error message.

[EmileSonneveld]

It also works with the original amount of ram, so I reverted.