Description
Describe your question
Hello, sorry for the post here, between the azurerm provider for terraform, keda, and aks, it seems here is the most appropriate location.
I did setup a scalable agent arch using Keda on AKS. Each set of agents use their own namespace and federated identity to poll and register when when there's jobs queued for their respective agent pools.
So far, perfect.
But the blocker i'm hitting now, is when i want to authenticate using the Service Connection created on Azure DevOps side, and linked to the pipeline which jobs triggered the scaled agent to spin up.
Like i said, this agent relies on a Kubernetes job which is using a workload identity which only enables the pool job queue polling and the registration of the agent. For the pipeline purpose, i would really like to rely on the service connection and its permissions to respect least privilege principle. So if my pipeline is meant to configure DNS, i want it to use the service connection that has the needed set of permissions of the relevant scope to do so.
The problem i'm having, is that it seems i cant get terraform to authenticate properly with the service connection which are of type "workload identity federation with openid connect".
I tried to override env vars when the jobs starts but hit the following :
│ Error: mismatch between supplied Client ID and that provided by AKS Workload Identity - please remove, ensure they match, or disable use_aks_workload_identity
It seems the only way would be to use service principal authentication, with the help of a cert or secret , which would break the identity federation purpose of the service connection used here.
Ok let me recap, the issue is in the following lines
For a while the expected setup on the AAD side was something like :
- issuer: https://vstoken.dev.azure.com/azdo-tenant-id
- subject: sc://org-name/project-name/endpoint-name
Since a few days here , it seems this changed to :
- issuer: https://login.microsoftonline.com/organization-id/v2.0
- subject: /eid1/c/pub/t/...
The only token i manage to generate from the pipeline execution environment is the following :
- https://vstoken.dev.azure.com/azdo-tenant-id
- p://Sorg-name/project-name/pipeline-name
This last "oidcToken" is retrieved from SYSTEM_OIDCREQUESTURI, using the SYSTEM_ACCESSTOKEN as a Bearer.
Of course, if i edit the app registration to align with my token it works.
My interrogations are now, how can we generate the idToken with the correct issuer/subject ?
Do we need to recreate all of our existing service connections for the sake of consistency ?
Versions
AKS Server Version: v1.31.7
Keda 2.17.0 (latest)
Agent version 2.250.1 (latest, tried today)
Running on alpine image v3.21.3 (last stable)
Terraform v1.11.4 (last version)
AzureRM 4.27.0 (last version)
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Operation system
Alpine version of agent
Version controll system
No response
Azure DevOps Server Version (if applicable)
No response