Scenario Failure: max_ndvi_composite

[github-actions]

Benchmark Failure: max_ndvi_composite

Scenario ID: max_ndvi_composite
Backend System: openeofed.dataspace.copernicus.eu
Failure Count: 1
Timestamp: 2025-05-19 06:57:30

Links:

• Workflow Run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15106294007
• Scenario Definition: https://github.com/ESA-APEx/apex_algorithms/blob/7b42386622e1a33965771d6bb9bdd883fb23d0db/algorithm_catalog/vito/max_ndvi_composite/benchmark_scenarios/max_ndvi_composite.json
• Artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15106294007#artifacts

---

Contact Information

Point of Contact:

Name	Organization	Contact
Jeroen Dries	VITO	Contact via VITO (VITO Website, GitHub)

---

Process Graph

{
  "maxndvi1": {
    "process_id": "max_ndvi_composite",
    "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/refs/heads/main/algorithm_catalog/vito/max_ndvi_composite/openeo_udp/max_ndvi_composite.json",
    "arguments": {
      "spatial_extent": {
        "west": 5.07,
        "east": 5.09,
        "south": 51.21,
        "north": 51.23
      },
      "temporal_extent": [
        "2023-08-01",
        "2023-09-30"
      ],
      "bands": [
        "B04"
      ]
    },
    "result": true
  }
}

---

Error Logs

scenario = BenchmarkScenario(id='max_ndvi_composite', description='max_ndvi example', backend='openeofed.dataspace.copernicus.eu'...14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif'}, reference_options={})
connection_factory = <function connection_factory.<locals>.get_connection at 0x7f5f079a91c0>
tmp_path = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0')
track_metric = <function track_metric.<locals>.append at 0x7f5f079a94e0>
upload_assets_on_fail = <function upload_assets_on_fail.<locals>.collect at 0x7f5f079a9260>
request = <FixtureRequest for <Function test_run_benchmark[max_ndvi_composite]>>

    @pytest.mark.parametrize(
        "scenario",
        [
            # Use scenario id as parameterization id to give nicer test names.
            pytest.param(uc, id=uc.id)
            for uc in get_benchmark_scenarios()
        ],
    )
    def test_run_benchmark(
        scenario: BenchmarkScenario,
        connection_factory,
        tmp_path: Path,
        track_metric,
        upload_assets_on_fail,
        request
    ):
        track_metric("scenario_id", scenario.id)
        # Check if a backend override has been provided via cli options.
        override_backend = request.config.getoption("--override-backend")
        backend_filter = request.config.getoption("--backend-filter")
        if backend_filter and not re.match(backend_filter, scenario.backend):
            #TODO apply filter during scenario retrieval, but seems to be hard to retrieve cli param
            pytest.skip(f"skipping scenario {scenario.id} because backend {scenario.backend} does not match filter {backend_filter!r}")
        backend = scenario.backend
        if override_backend:
            _log.info(f"Overriding backend URL with {override_backend!r}")
            backend = override_backend
    
        connection: openeo.Connection = connection_factory(url=backend)
    
        # TODO #14 scenario option to use synchronous instead of batch job mode?
        job = connection.create_job(
            process_graph=scenario.process_graph,
            title=f"APEx benchmark {scenario.id}",
            additional=scenario.job_options,
        )
        track_metric("job_id", job.job_id)
    
        # TODO: monitor timing and progress
        # TODO: abort excessively long batch jobs? https://github.com/Open-EO/openeo-python-client/issues/589
        job.start_and_wait()
    
        collect_metrics_from_job_metadata(job, track_metric=track_metric)
    
        results = job.get_results()
        collect_metrics_from_results_metadata(results, track_metric=track_metric)
    
        # Download actual results
        actual_dir = tmp_path / "actual"
        paths = results.download_files(target=actual_dir, include_stac_metadata=True)
        # Upload assets on failure
        upload_assets_on_fail(*paths)
    
        # Compare actual results with reference data
        reference_dir = download_reference_data(
            scenario=scenario, reference_dir=tmp_path / "reference"
        )
    
>       assert_job_results_allclose(
            actual=actual_dir,
            expected=reference_dir,
            tmp_path=tmp_path,
            rtol=scenario.reference_options.get("rtol", 1e-6),
            atol=scenario.reference_options.get("atol", 1e-6),
        )

tests/test_benchmarks.py:79: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

actual = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual')
expected = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference')

    def assert_job_results_allclose(
        actual: Union[BatchJob, JobResults, str, Path],
        expected: Union[BatchJob, JobResults, str, Path],
        *,
        rtol: float = _DEFAULT_RTOL,
        atol: float = _DEFAULT_ATOL,
        tmp_path: Optional[Path] = None,
    ):
        """
        Assert that two job results sets are equal (with tolerance).
    
        :param actual: actual job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param expected: expected job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param rtol: relative tolerance
        :param atol: absolute tolerance
        :param tmp_path: root temp path to download results if needed.
            It's recommended to pass pytest's `tmp_path` fixture here
        :raises AssertionError: if not equal within the given tolerance
    
        .. versionadded:: 0.31.0
    
        .. warning::
            This function is experimental and subject to change.
        """
        issues = _compare_job_results(actual, expected, rtol=rtol, atol=atol, tmp_path=tmp_path)
        if issues:
>           raise AssertionError("\n".join(issues))
E           AssertionError: Issues for metadata file 'job-results.json':
E           Differing 'derived_from' links (12 common, 1 only in actual, 1 only in expected):
E             only in actual: {'/eodata/Sentinel-2/MSI/L2A_N0500/2023/09/24/S2B_MSIL2A_20230924T103659_N0510_R008_T31UFS_20241101T040059.SAFE'}
E             only in expected: {'/eodata/Sentinel-2/MSI/L2A/2023/09/24/S2B_MSIL2A_20230924T103659_N0509_R008_T31UFS_20230924T132849.SAFE'}.
E           Issues for file 'openEO.tif':
E           Left and right DataArray objects are not close
E           Differing values:
E           L
E               array([[[ 885, 1118, ...,  388,  295],
E                       [2238, 2752, ...,  457,  229],
E                       ...,
E                       [ 187,  243, ...,  235,  283],
E                       [ 210,  212, ...,  221,  301]]], shape=(1, 227, 147), dtype=int16)
E           R
E               array([[[ 885, 1118, ...,  388,  295],
E                       [2238, 2752, ...,  457,  229],
E                       ...,
E                       [ 187,  243, ...,  235,  283],
E                       [ 210,  212, ...,  221,  301]]], shape=(1, 227, 147), dtype=int16)
E           band 1: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0, max: 1568.0, mean: 97.77, var: 20333.7
E           band 1: differing pixels: 3260/33369 (9.8%), bbox ((644525.0, 5675215.0), (645985.0, 5677475.0)) - 100.0% of the area

/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/openeo/testing/results.py:488: AssertionError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': send 'start'
0:00:14 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': created (progress 0%)
0:00:19 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': created (progress 0%)
0:00:26 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': created (progress 0%)
0:00:34 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': created (progress 0%)
0:00:44 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:00:56 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:01:12 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:01:31 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:01:56 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:02:26 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:03:04 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': running (progress N/A)
0:03:51 Job 'cdse-j-250519065327406ab8a102992e2b5cfb': finished (progress 100%)
------------------------------ Captured log call -------------------------------
INFO     conftest:conftest.py:131 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO     openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO     conftest:conftest.py:144 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO     conftest:conftest.py:148 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO     openeo.rest.connection:connection.py:232 Found OIDC providers: ['CDSE']
INFO     openeo.rest.auth.oidc:oidc.py:404 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:329 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.job:job.py:401 Downloading Job result asset 'openEO.tif' from https://openeo.dataspace.copernicus.eu/openeo/1.1/jobs/j-250519065327406ab8a102992e2b5cfb/results/assets/NmE3N2ZjZDEtOWMwOC00NmU5LWI4NzUtNTRmYjk5OWFiMjAw/0dc683eb7c722b0a5e0de58dc8ca8cb9/openEO.tif?expires=1748242640 to /home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual/openEO.tif
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading reference data for scenario.id='max_ndvi_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference'): start 2025-05-19 06:57:22.207041
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/job-results.json'): start 2025-05-19 06:57:22.207363
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/job-results.json'): end 2025-05-19 06:57:22.896074, elapsed 0:00:00.688711
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/openEO.tif'): start 2025-05-19 06:57:22.896408
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/openEO.tif'): end 2025-05-19 06:57:23.740332, elapsed 0:00:00.843924
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading reference data for scenario.id='max_ndvi_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference'): end 2025-05-19 06:57:23.740514, elapsed 0:00:01.533473
INFO     openeo.testing.results:results.py:400 Comparing job results: PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual') vs PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference')
WARNING  openeo.testing.results:results.py:169 Difference (ascii art) for band 1:
┌──────────────────────────────────────────────────────────────────────────┐
│W  n                d '".,   d,^^,^.'...''...`'.......`.          ^`^^```,│
│  'z'  1          ?     "        ...  ^'`^'....'`.'."             "... ..'│
│ .....             .....         '`.^'.''''`^"",^,"`               '.''..`│
│   ......                       ` `.`"  ^"`^`. .  `";                "^^'.│
│                                         >,":,`'"",;;:^'!!l}?^:       `'^`│
│                                         I+~l\ Ii   .   I".'+I:!       :'.│
│                I<+<`' I            l`       >;>  >^ >i![!:,i~lI'"       ^│
│                                     I     ""         <   ~_:}:l!         │
│                                                            '!i^          │
│                                                                          │
│                              .'               .` ''>f1\t_,^ ,`^          │
│                                 .'..  ..      ..`'^I)              ' "   │
│          .  .   ' ``           '.'^. .."`'^`.'``~;i          '^.`:l, l^,'│
│          '   ' .             .`'`''.'"`^     ",l'l       `.'`            │
│^               .''':^`;`I,;"'`"'`"""^i    ',.^,^^,:",l                   │
│`.^```^''```'``^.''^'.        ..'. " .'.^`'`'.`.'.'"..^    !     !       _│
│`^:`I    ..      '......    . ''.  ..  ..... ...'.'`_>!!                  │
│      ..             .....       .'','..  .                ..'_[.         │
│.            `.       ....        :"'....      .     ...  .. :"...'..     │
│.'.....       '      .... .  .'..  .'   ^^' .     .    .^'-1i"^.... ......│
│.   .  .       '  .`......  ..         ..''''`.`'` .   .'"`';i'.    .     │
│ .     '.     .  '`.`.. .................'...'..'..    I:;,Ih    .     .'.│
│.  .. ..      .  ..  . .....      .    .. ...";. .'. `.  .   . <,"'..'....│
│. ....... . ` ....     .   .        : '     ...`...  '..'...  .``^,.  <!  │
│ ......  .  ... . ..  .                ...'...'`..    .  '...    "   i^   │
│         . . .                        .     ''`  .                        │
└──────────────────────────────────────────────────────────────────────────┘

[github-actions]

Benchmark Failure: max_ndvi_composite

Scenario ID: max_ndvi_composite
Backend: openeofed.dataspace.copernicus.eu
Timestamp: 2025-06-10 13:23:42 UTC

Links:

• Workflow Run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15560627772
• Scenario Definition: https://github.com/None/blob/c4699bad320f312f67438e2605b5d5373fc17a66/algorithm_catalog/vito/max_ndvi_composite/benchmark_scenarios/max_ndvi_composite.json
• Artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15560627772#artifacts

---

Contact Information

Point of Contact:

Name	Organization	Contact
Jeroen Dries	VITO	Contact via VITO (VITO Website, GitHub)

---

Process Graph

{
  "maxndvi1": {
    "process_id": "max_ndvi_composite",
    "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/refs/heads/main/algorithm_catalog/vito/max_ndvi_composite/openeo_udp/max_ndvi_composite.json",
    "arguments": {
      "spatial_extent": {
        "west": 5.07,
        "east": 5.09,
        "south": 51.21,
        "north": 51.23
      },
      "temporal_extent": [
        "2023-08-01",
        "2023-09-30"
      ],
      "bands": [
        "B04"
      ]
    },
    "result": true
  }
}

---

Error Logs

scenario = BenchmarkScenario(id='max_ndvi_composite', description='max_ndvi example', backend='openeofed.dataspace.copernicus.eu'...14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif'}, reference_options={})
connection_factory = <function connection_factory.<locals>.get_connection at 0x7fe239c5ade0>
tmp_path = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0')
track_metric = <function track_metric.<locals>.track at 0x7fe239c5b060>
upload_assets_on_fail = <function upload_assets_on_fail.<locals>.collect at 0x7fe239c5ac00>
request = <FixtureRequest for <Function test_run_benchmark[max_ndvi_composite]>>

    @pytest.mark.parametrize(
        "scenario",
        [
            # Use scenario id as parameterization id to give nicer test names.
            pytest.param(uc, id=uc.id)
            for uc in get_benchmark_scenarios()
            if uc.id == "max_ndvi_composite" #TODO remove it, after verifying fix
        ],
    )
    def test_run_benchmark(
        scenario: BenchmarkScenario,
        connection_factory,
        tmp_path: Path,
        track_metric,
        upload_assets_on_fail,
        request,
    ):
        track_metric("scenario_id", scenario.id)
        # Check if a backend override has been provided via cli options.
        override_backend = request.config.getoption("--override-backend")
        backend_filter = request.config.getoption("--backend-filter")
        if backend_filter and not re.match(backend_filter, scenario.backend):
            # TODO apply filter during scenario retrieval, but seems to be hard to retrieve cli param
            pytest.skip(
                f"skipping scenario {scenario.id} because backend {scenario.backend} does not match filter {backend_filter!r}"
            )
        backend = scenario.backend
        if override_backend:
            _log.info(f"Overriding backend URL with {override_backend!r}")
            backend = override_backend
    
        connection: openeo.Connection = connection_factory(url=backend)
        track_metric("milestone", "connected", update=True)
    
        # TODO #14 scenario option to use synchronous instead of batch job mode?
        job = connection.create_job(
            process_graph=scenario.process_graph,
            title=f"APEx benchmark {scenario.id}",
            additional=scenario.job_options,
        )
        track_metric("job_id", job.job_id)
        track_metric("milestone", "job created", update=True)
    
        # TODO: monitor timing and progress
        # TODO: abort excessively long batch jobs? https://github.com/Open-EO/openeo-python-client/issues/589
        job.start_and_wait()
        # TODO: "job started" milestone
        track_metric("milestone", "job finished", update=True)
    
        collect_metrics_from_job_metadata(job, track_metric=track_metric)
        track_metric("milestone", "job metadata obtained", update=True)
    
        results = job.get_results()
        track_metric("milestone", "job results metadata obtained", update=True)
        collect_metrics_from_results_metadata(results, track_metric=track_metric)
    
        # Download actual results
        actual_dir = tmp_path / "actual"
        paths = results.download_files(target=actual_dir, include_stac_metadata=True)
        track_metric("milestone", "job results downloaded", update=True)
    
        # Upload assets on failure
        upload_assets_on_fail(*paths)
    
        # Compare actual results with reference data
        reference_dir = download_reference_data(
            scenario=scenario, reference_dir=tmp_path / "reference"
        )
        track_metric("milestone", "benchmark reference data downloaded", update=True)
    
>       assert_job_results_allclose(
            actual=actual_dir,
            expected=reference_dir,
            tmp_path=tmp_path,
            rtol=scenario.reference_options.get("rtol", 1e-6),
            atol=scenario.reference_options.get("atol", 1e-6),
        )

tests/test_benchmarks.py:91: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

actual = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual')
expected = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference')

    def assert_job_results_allclose(
        actual: Union[BatchJob, JobResults, str, Path],
        expected: Union[BatchJob, JobResults, str, Path],
        *,
        rtol: float = _DEFAULT_RTOL,
        atol: float = _DEFAULT_ATOL,
        pixel_tolerance: float = _DEFAULT_PIXELTOL,
        tmp_path: Optional[Path] = None,
    ):
        """
        Assert that two job results sets are equal (with tolerance).
    
        :param actual: actual job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param expected: expected job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param rtol: relative tolerance
        :param atol: absolute tolerance
        :param pixel_tolerance: maximum fraction of pixels (in percent)
            that is allowed to be significantly different (considering ``atol`` and ``rtol``)
        :param tmp_path: root temp path to download results if needed.
            It's recommended to pass pytest's `tmp_path` fixture here
        :raises AssertionError: if not equal within the given tolerance
    
        .. versionadded:: 0.31.0
    
        .. warning::
            This function is experimental and subject to change.
        """
        issues = _compare_job_results(
            actual, expected, rtol=rtol, atol=atol, pixel_tolerance=pixel_tolerance, tmp_path=tmp_path
        )
        if issues:
>           raise AssertionError("\n".join(issues))
E           AssertionError: Issues for metadata file 'job-results.json':
E           Differing 'derived_from' links (12 common, 1 only in actual, 1 only in expected):
E             only in actual: {'/eodata/Sentinel-2/MSI/L2A_N0500/2023/09/24/S2B_MSIL2A_20230924T103659_N0510_R008_T31UFS_20241101T040059.SAFE'}
E             only in expected: {'/eodata/Sentinel-2/MSI/L2A/2023/09/24/S2B_MSIL2A_20230924T103659_N0509_R008_T31UFS_20230924T132849.SAFE'}.
E           Issues for file 'openEO.tif':
E           Left and right DataArray objects are not close
E           Differing values:
E           L
E               array([[[ 885, 1118, ...,  388,  295],
E                       [2238, 2752, ...,  457,  229],
E                       ...,
E                       [ 187,  243, ...,  235,  283],
E                       [ 210,  212, ...,  221,  301]]], shape=(1, 227, 147), dtype=int16)
E           R
E               array([[[ 885, 1118, ...,  388,  295],
E                       [2238, 2752, ...,  457,  229],
E                       ...,
E                       [ 187,  243, ...,  235,  283],
E                       [ 210,  212, ...,  221,  301]]], shape=(1, 227, 147), dtype=int16)
E           band 1: value difference exceeds tolerance (rtol 1e-06, atol 1e-06), min:1.0, max: 1568.0, mean: 97.77, var: 20333.7
E           band 1: differing pixels: 3260/33369 (9.8%), bbox ((644525.0, 5675215.0), (645985.0, 5677475.0)) - 100.0% of the area

/opt/hostedtoolcache/Python/3.12.10/x64/lib/python3.12/site-packages/openeo/testing/results.py:515: AssertionError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': send 'start'
0:00:14 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': created (progress 0%)
0:00:20 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': created (progress 0%)
0:00:27 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': created (progress 0%)
0:00:35 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:00:45 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:00:57 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:01:13 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:01:32 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:01:57 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:02:27 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:03:04 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': running (progress N/A)
0:03:51 Job 'cdse-j-2506101319424f2fb5491834d5ade98e': finished (progress 100%)
------------------------------ Captured log call -------------------------------
INFO     conftest:conftest.py:131 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO     openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO     conftest:conftest.py:144 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO     conftest:conftest.py:148 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO     openeo.rest.connection:connection.py:250 Found OIDC providers: ['CDSE']
INFO     openeo.rest.auth.oidc:oidc.py:404 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:347 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.job:job.py:429 Downloading Job result asset 'openEO.tif' from https://openeo.dataspace.copernicus.eu/openeo/1.1/jobs/j-2506101319424f2fb5491834d5ade98e/results/assets/NmE3N2ZjZDEtOWMwOC00NmU5LWI4NzUtNTRmYjk5OWFiMjAw/b78e6b3b462397196776fac6727e6802/openEO.tif?expires=1750166615 to /home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual/openEO.tif
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading reference data for scenario.id='max_ndvi_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference'): start 2025-06-10 13:23:38.372500
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/job-results.json'): start 2025-06-10 13:23:38.372802
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/job-results.json'): end 2025-06-10 13:23:39.061681, elapsed 0:00:00.688879
INFO     apex_algorithm_qa_tools.scenarios:util.py:346 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/openEO.tif'): start 2025-06-10 13:23:39.062068
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14544662950!tests_test_benchmarks.py__test_run_benchmark_max_ndvi_composite_!actual/openEO.tif' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference/openEO.tif'): end 2025-06-10 13:23:39.568461, elapsed 0:00:00.506393
INFO     apex_algorithm_qa_tools.scenarios:util.py:352 Downloading reference data for scenario.id='max_ndvi_composite' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference'): end 2025-06-10 13:23:39.568619, elapsed 0:00:01.196119
INFO     openeo.testing.results:results.py:418 Comparing job results: PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/actual') vs PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_max_ndvi_co0/reference')
WARNING  openeo.testing.results:results.py:171 Difference (ascii art) for band 1:
┌──────────────────────────────────────────────────────────────────────────┐
│W  x                d '".\   p\^^\^.'...''...`'.......`.          ^`^"```\│
│  'c'  {          -     "        ...  ^'`^'....'`.'."             "... ..'│
│ .....             .....         '`.^'.''''`^""\","`               '.''..`│
│   ......                       ` `.`\  ""`^^. .  `":                \^^'.│
│                                         i\",\`'""\:;,^'llI[-^,       `'^`│
│                                         ;~<l\ I!   .   ;".'~;,l       ,'.│
│                I>~>`' ;            I^       i;i  i" ii![l,\!<l;'"       ^│
│                                     I     "\         >   <+,}:Il         │
│                                                            'li^          │
│                                                                          │
│                              .'               .` ''>f1\/+\^ \`^          │
│                                 .'..  ..      ..`'^;1              ' "   │
│          .  .   ' ``           `.'^. .."``^`.'``~:i          '^.`:I\ I^,'│
│          '   ' .             .`'`''.'"`^     \\I'I       `.``            │
│^               .''',^`:^;\;"'`"'`"""^!    '\."\^^\,"\I                   │
│`.^```^''```'``^.''^'.        ..`. " .'.^`'`'.`.'.`"..^    l     l       _│
│`^,`;    ..      '......    . ''.  ..  ..... ...'.'`+>ll                  │
│      ..             .....       .''\'..  .                ..'+].         │
│.            `.       ....        ,"'....      .     ...  .. ,"...'..     │
│.'.....       '      .... .  .'..  ''   ^^' .     .    .^'_{!"^.... ......│
│.   .  .       '  .`......  ..         ..''''`.`'` .   .'"`';!'.    .     │
│ .     '.     .  ``.`.. .................'...'..'..    ;,:\;h    .     .'.│
│.  .. ..      .  ..  . .....      .    .. ...":. .'. `.  .   . >\"'..'....│
│. ....... . ` ....     .   .        , '     ...`...  '..'...  .``^\.  <l  │
│ ......  .  ... . ..  .                ...'...'`..    '  '...    "   !^   │
│         . . .                        .     ''`  .                        │
└──────────────────────────────────────────────────────────────────────────┘

[github-actions]

Status: Failure

Benchmark: max_ndvi_composite

Scenario ID: max_ndvi_composite
Backend: openeofed.dataspace.copernicus.eu
Timestamp: 2025-06-12 11:39:17 UTC

Links:

• Workflow Run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15609335934
• Scenario Definition: https://github.com/None/blob/d4c56b805d746c7a13b2a7e98f8272002da2911d/algorithm_catalog/vito/max_ndvi_composite/benchmark_scenarios/max_ndvi_composite.json
• Artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/15609335934#artifacts

[HansVRP]

testing purposes