Try unauthenticated OCI registry interactions if expired/invalid credentials are set

[myoung3]

Version of Singularity
3.8.1

Describe the bug
If I associated a username/token pair for an oras repository with singularity using singularity remote login, then cancel that token (ie via github), I can't pull from public repositories.

To Reproduce

• step 0:
  • verify anonymous pulls are possible through oras/github. in a singularity installation that doesn't have any ghcr.io tokens stored, try singularity pull oras://ghcr.io/singularityhub/github-ci:latest
  • now clear your image cache singularity cache clean
• step 2: generate github PAT https://github.com/settings/tokens
• step 3: echo <PAT> | singularity remote login --username <gh username> --password-stdin oras://ghcr.io
• step 4:
  • test that you can pull with a valid key: singularity pull oras://ghcr.io/singularityhub/github-ci:latest
  • clear your cache singularity cache clean
• step 5 cancel your PAT here https://github.com/settings/tokens
• step 6 try to pull again singularity pull oras://ghcr.io/singularityhub/github-ci:latest

ingularity shell oras://ghcr.io/singularityhub/github-ci:latest
INFO[0000] trying next host                              error="failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden" host=ghcr.io
FATAL:   Unable to handle oras://ghcr.io/singularityhub/github-ci:latest uri: failed to get checksum for oras://ghcr.io/singularityhub/github-ci:latest: while resolving reference: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Expected behavior
print a warning that authentication failed, retry anonymously.

OS / Linux Distribution
Which Linux distribution are you using?

ingularity shell oras://ghcr.io/singularityhub/github-ci:latest
INFO[0000] trying next host                              error="failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden" host=ghcr.io
FATAL:   Unable to handle oras://ghcr.io/singularityhub/github-ci:latest uri: failed to get checksum for oras://ghcr.io/singularityhub/github-ci:latest: while resolving reference: failed to authorize: failed to fetch oauth token: unexpected status: 403 Forbidden

Installation Method
source

Additional context
it would also be good if there was a way to remove authenticated logins

[dtrudg]

Expected behavior
print a warning that authentication failed, retry anonymously.

Not sure that we want to do this. I am of the opinion that if credentials have been set they should always be used, and an auth failure should fail hard. Any thoughts @EmmEff ?

Additional context
it would also be good if there was a way to remove authenticated logins

This should definitely be provided.

[vsoch]

I would take an approach that most APIs taken with respect to auth - first try without it, and then get back a 403 to indicate that auth is required, and respond to that. For this case it would mean not using the expired token by default and the public image would pull! And then if auth is required, that would make sense to try the token and report failure to the user.

[dtrudg]

@vsoch - this is not the case for docker, podman etc. though - which I believe are the best targets to emulate here w.r.t. interaction with OCI registries.

With docker, pulling a public image from Docker Hub, if I attempt to pull with an invalid token set it will not fall back and allow a public pull:

$ docker pull ubuntu
Using default tag: latest
Error response from daemon: Head https://registry-1.docker.io/v2/library/ubuntu/manifests/latest: unauthorized: please use personal access token to login

Ditto podman:

$ podman pull ubuntu
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/ubuntu:latest...
Error: Error initializing source docker://ubuntu:latest: unable to retrieve auth token: invalid username/password: unauthorized: please use personal access token to login

One reason for this is that we now have things like rate limiting on OCI registries such as Docker Hub. I always want to pull authenticated, even public images. This is so that pulls count against my account limits, and do not eat up the low non-authenticated limits that apply for my IP address for anyone who is not logged in.

[vsoch]

Sorry it's the other way around - you try without any token first, and only add the token to try on that failure.

[dtrudg]

Sorry it's the other way around - you try without any token first, and only add the token to try on that failure.

Right - that's what we don't want to do because...

... we now have things like rate limiting on OCI registries such as Docker Hub. I always want to pull authenticated, even public images. This is so that pulls count against my account limits, and do not eat up the low non-authenticated limits that apply for my IP address for anyone who is not logged in.

[vsoch]

Hmm, well I think there are different opinions on this, because I'd prefer the first - I don't pull frequently enough either authenticated or not to worry about that (but I see the point that this could go badly in some automated CI). What if you default to what would be safest for CI, but then allow users to customize a config to say "please try without authenticated first?" There is definitely something to say for trying the simplest approaches before just blindly throwing credentials at something.

[dtrudg]

We are calling out to other 3rd party packages to actually do our operations against OCI registries, and that might be where an always auth first if configured vs noauth -> auth if needed configurable behavior might be better placed, but I'm not sure they would consider it.

Implementation would need some careful work in multiple places in Singularity which can interact with OCI registries, and I think this is something where I'd be happy to see it contributed as a PR from the community, but in the reverse direction - we should match docker and podman by default. I think this is vital as with shared IPs etc. it's easy to block someone else by consuming the unauthenticated Docker hub limits when you intend to be auth'd. Also, I don't think this would be a high Sylabs priority for development.

Adam is out at the moment, but I'll ask for his input too when he is back, and I'll talk to @EmmEff as well.

[EmmEff]

I agree with both @vsoch and @dtrudg :)

It makes sense not to burn through unauthorized Docker Hub limits, though I know at least some modules do default to an unauth-then-auth flow if a 403 is returned from the unauthorized request. I would have to investigate what others are doing, but as @dtrudg mentions, aren't Docker and podman doing auth always?

[dtrudg]

@myoung3

Additional context
it would also be good if there was a way to remove authenticated logins

Try singularity remote logout .... - I forgout about that one for a minute :-)