User error question: Failure to get valid OIDC token with id with GitLab

[matthewfeickert]

👋 Hi. This is a user error (as in I'm doing this wrong) question, so if there is a prefered different place to ask them please let me know and I'll move this.

Now that PyPI Trusted Publishers should be able to work with GitLab CI/CD @kratsg and myself are interested in using them for publication of tools being built on CERN's EE GitLab instance (https://gitlab.cern.ch/) (currently v16.9.4-ee and so should support id_tokens) as we've been using them with GitHub Actions and enjoying them.

However, though @kratsg has setup the PyPI package https://pypi.org/project/itkdb/ to support trusted publishers coming from GitLab CI/CD attempting to get a debug version of a .gitlab-ci.yml workflow to just get a OIDC token from id and then exchange it for a PyPI API token fails as

stages:
  - deploy

variables:
  # see https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip
    - venv/

image: python:3.11-bullseye
before_script:
  # want to set up a virtualenv to cache
  - apt-get update
  - apt-get install -y --no-install-recommends git jq
  - python -V
  - git config --global credential.helper 'cache'
  - python -m venv venv
  - source venv/bin/activate
  - python -m pip install -U pip pipx id
  - python -m pipx ensurepath
  - python -m pip freeze --local

publish-job:
  stage: deploy
  id_tokens:
    PYPI_ID_TOKEN:
      aud: pypi
  variables:
    OIDC_MINT_TOKEN_URL: "https://pypi.org/_/oidc/mint-token"
  script:
    # Retrieve the OIDC token from GitLab CI/CD and exchange it for a PyPI API token
    - oidc_token=$(python -m id PYPI)
    - echo "OIDC MINT TOKEN URL ${OIDC_MINT_TOKEN_URL}"
    - response=$(curl -X POST "${OIDC_MINT_TOKEN_URL}" -d "{\"token\":\"${oidc_token}\"}")
    - echo "RESPONSE ${response}"
    - api_token=$(jq --raw-output '.token' <<< "${response}")

    - echo "TEST TOKEN ${api_token}"

results in an invalid-payload error.

$ echo "RESPONSE ${response}"
RESPONSE {"errors":[{"code":"invalid-payload","description":"unknown trusted publishing issuer"}],"message":"Token request failed"}

As this error is indicating that python -m id PYPI is failing to retrieve a valid OIDC token in the current setup on CERN's GitLab do you have debugging advice, related to id?

Namespace aside (possibly main problem)

The project lives as an internal project at https://gitlab.cern.ch/atlas-itk/sw/db/itkdb/ so @kratsg selected a namespace of atlas-itk

as trying with the full namespace of atlas-itk/sw/db failed with

Invalid GitLab username of group/subgroup name

---

cc @henryiii as also somewhat related to scientific-python/cookie#411

[facutuesca]

Hi @matthewfeickert , thanks for the bug report! The namespace issue is definitely a bug on PyPI's side, I'll create a fix for it.

As for the general problem though, the issue is that PyPI does not currently support self-hosted GitLab instances. That is, we only validate OIDC tokens where the issuer in the claims is https://gitlab.com. That is why you're getting the unknown trusted publishing issuer error.

Looking at our docs, it looks like we didn't make this explicit, so that's definitely something we can fix. But for now, I'm afraid the only way to use Trusted Publishing with GitLab is when using the official gitlab.com instance.

(and just for clarification, this is all on PyPI's side: id is correctly retrieving a valid OIDC token from your GitLab instance, the problem is that PyPI does not accept it due to not having originated from gitlab.com)

[facutuesca]

Linking pypi/warehouse#15836, which tracks the namespace bug on PyPI's side

[matthewfeickert]

Thanks @facutuesca. As this isn't a id issue I'll close this. I'll look for Issues on self-hosting on https://github.com/pypi/warehouse/issues/ and I've subscribed to pypi/warehouse#15836.

[di]

I don't believe there is one, but feel free to create one!

[di]

@matthewfeickert Just curious here, if you install this library and run python -m id throwaway --decode from within GitLab CI, do you get a valid JWT?

(You can share the results here, the signature should be redacted by that command).

[matthewfeickert]

if you install this library and run python -m id throwaway --decode from within GitLab CI, do you get a valid JWT?

@di Yes. For the following (non-minimal because I'm doing a lazy edit) .gitlab-ci.yml

YAML:

stages:
  - deploy

variables:
  # see https://docs.gitlab.com/ee/ci/caching/#cache-python-dependencies
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  paths:
    - .cache/pip
    - venv/

image: python:3.11-bullseye
before_script:
  # want to set up a virtualenv to cache
  - apt-get update
  - apt-get install -y --no-install-recommends git jq
  - python -V
  - git config --global credential.helper 'cache'
  - python -m venv venv
  - source venv/bin/activate
  - python -m pip install -U pip pipx id
  - python -m pipx ensurepath
  - python -m pip freeze --local

publish-job:
  stage: deploy
  id_tokens:
    THROWAWAY_ID_TOKEN:
      aud: pypi
  script:
    - python -m id throwaway --decode

I get

$ python -m id throwaway --decode
{"kid":"Yi4Qlfo9GsiLCsaxm96WMTz1BxCGsKtiQ8wqEpQTIyM","typ":"JWT","alg":"RS256"}
{"namespace_id":"17511","namespace_path":"atlas-itk/sw/db","project_id":"62567","project_path":"atlas-itk/sw/db/itkdb","user_id":"6121","user_login":"feickert","user_email":"matthew.feickert@cern.ch","user_access_level":"developer","pipeline_id":"7274672","pipeline_source":"push","job_id":"38298873","ref":"debug/test-id-package","ref_type":"branch","ref_path":"refs/heads/debug/test-id-package","ref_protected":"false","runner_id":34772,"runner_environment":"self-hosted","sha":"6fa49a6a13d4772c712a208f0e15deebecae53c2","project_visibility":"public","ci_config_ref_uri":"gitlab.cern.ch/atlas-itk/sw/db/itkdb//.gitlab-ci.yml@refs/heads/debug/test-id-package","ci_config_sha":"6fa49a6a13d4772c712a208f0e15deebecae53c2","jti":"590c0fd4-95a7-485c-85a8-1d0647d6a10a","iss":"https://gitlab.cern.ch/","iat":1713888962,"nbf":1713888957,"exp":1713892562,"sub":"project_path:atlas-itk/sw/db/itkdb:ref_type:branch:ref:debug/test-id-package","aud":"pypi"}

(Sorry about the first response I deleted. I forgot to set the THROWAWAY_ID_TOKEN when I first posted.)

[di]

OK, seems like this should work fine here then. Thanks!