ORAS images pushed with 4.x fail to pull with 3.9 due to manifest mediaType

[GabeGEL]

Before you report an issue...

• Have you searched for a duplicate report? Nothing found in the singularity repo, but a likely associated bug report in apptainer
• Have you replicated the problem on the latest release? No, I don't have access to a more recent version
• Have you checked the latest documentation?
• Are you using an up-to-date system?

Only issues that can be replicated on the latest release, or development branch, of SingularityCE will be investigated and fixed. The open source project does not maintain long-term stable branches or fix bugs in prior versions. If you require LTS support then please see the Sylabs website.

Version of Singularity

For pulling images, I'm using:

$ singularity --version
singularity-ce version 3.9.5

For pushing images, I'm told that 4.2.1-r4 is being used.

Describe the bug

The seqeralabs wave service can be used to build singularity sif images and push them into an oras registry.
Until recently (~1 month), it was using v3.11.4-slim for this. At this time, I was able to pull images from the ORAS registry using singularity-ce version 3.9.5.

In a recent update to wave, the version of singularity used to create and push sif images was updated to 4.2.1-r4.
Since then, I've been unable to pull images. Attempting to do so results in the following error:

$ singularity pull oras://community.wave.seqera.io/library/pip_seaborn:eb0eaeddb41e95ed
FATAL:   While pulling image from oci registry: error fetching image to cache: failed to get checksum for oras://community.wave.seqera.io/library/pip_seaborn:eb0eaeddb41e95ed: could not get image manifest, received mediaType: application/vnd.docker.distribution.manifest.v2+json

Under the hood, this difference is due to the mediaType in the manifest json:

$ oras manifest fetch  --pretty community.wave.seqera.io/library/pip_seaborn:eb0eaeddb41e95ed

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",  ## <----
  "config": {
...

On older manifests, the mediaType was omitted which defaults to "application/vnd.oci.image.manifest.v1+json".

I believe the reciprocal issue was reported and fixed on apptainer: apptainer/apptainer#2178

To Reproduce

Steps to reproduce the behavior:

• Create a sif image with the latest version of singularity and push to an ORAS registry
• Attempt to pull the created image from the ORAS registry with singularity

(Note that I've been unable to confirm whether the issue with pulling presents with the current version of singularity)

Expected behavior

• Sif images pushed to an ORAS registry with newer versions of singularity can still be pulled.

OS / Linux Distribution

Which Linux distribution are you using? Is it up-to-date?

$ cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="8.10"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Oracle Linux Server 8.10"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:8:10:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 8"
ORACLE_BUGZILLA_PRODUCT_VERSION=8.10
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=8.10

Installation Method

Write here how you installed SingularityCE. Eg. RPM from GitHub / EPEL, compiled from source.

I'm afraid I don't have this information.

Additional context

I originally logged this issue against seqeralabs/wave at: seqeralabs/wave#835

[GabeGEL]

I have confirmed that newer versions of singularity are able to pull images with the docker manifest mediaType.

$ singularity --version
singularity-ce version 4.1.3

$ oras manifest fetch  --pretty community.wave.seqera.io/library/pandas_requests_tabulate:56f3ca6a6380a887 | head -n 4
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {

$ singularity pull oras://community.wave.seqera.io/library/pip_seaborn:eb0eaeddb41e95ed
INFO:    Downloading oras image
243.1MiB / 243.1MiB [============================================================================================================================================] 100 % 28.0 MiB/s 0s

I'm not sure whether you consider the broken backwards compatibility to be a bug or not.

[dtrudg]

Hi @GabeGEL

The root cause here is arguably not a Singularity 4.x bug, but the fact that 3.9 is unneccesarily strict in its handling of manifest mediaTypes for oras:// images.

An image can be pushed successfully with Singularity 4.x and pulled with 4.x or the oras tool - confirming that it is a valid ORAS image:

$ singularity push alpine_latest.sif oras://registry.localhost/oras:test
INFO:    Upload complete

$ ./oras pull registry.localhost/oras:test
✓ Pulled      alpine_latest.sif                                                                3.4/3.4 MB 100.00%   23ms
  └─ sha256:3a64b40c1afe5e25ae727da0daf1180003b846db9634df4f9c4d8b7c85c59db0
✓ Pulled      application/vnd.docker.distribution.manifest.v2+json                             479/479  B 100.00%     0s
  └─ sha256:dac32e2863f3f7fdabef476d1596324b229dddd6be992f42d5551faf8e88318f
Pulled [registry] registry.localhost/oras:test
Digest: sha256:dac32e2863f3f7fdabef476d1596324b229dddd6be992f42d5551faf8e88318f

I'm not sure whether you consider the broken backwards compatibility to be a bug or not.

This is a forwards compatibility issue, rather than backwards. You are attempting to use an image pushed with a newer major version (4.x), from an older version (3.x). Backwards compatibility is instead concerned with newer versions being able to consume and execute images created with older versions.

We generally consider forward compatibility to be less critical than backwards compatibility, and coupled with the fact that the issue arises only when using a version of SingularityCE that is explicitly unsupported, I'm afraid it's not a high priority to address, as mentioned in the issue template:

Only issues that can be replicated on the latest release, or development branch, of SingularityCE will be investigated and fixed. The open source project does not maintain long-term stable branches or fix bugs in prior versions. If you require LTS support then please see the Sylabs website.

Having said that, we likely could switch to the OCI manifest mediaType in 4.x. The docker manifest mediaType was only used as it was the default in the upstream dependency (google/go-containerregistry) that 4.x switched to when the OCI/ORAS handling code was re-written.

TLDR - this is likely to be fixed in future, but it's a somewhat low priority as SingularityCE 3.9 is not supported in any way, and is subject to security issues etc. You should not use SingularityCE 3.9 in production.