Support for custom labels and annotations for resources deployed via karmada-operator (Kind: Karmada)

[spolspol]

Feature Request: Propagate Labels Defined in Karmada Resource to All Operator-Managed Resources

What would you like to be added:

Currently, the labels block defined within the Kind: Karmada resource only applies labels to Deployment resources. This means that when specifying custom labels within the karmadaAPIServer configuration, such as:

karmadaAPIServer:
  imagePullPolicy: IfNotPresent
  imageRepository: registry.k8s.io/kube-apiserver
  imageTag: v1.31.3
  labels:
    argocd.argoproj.io/instance: karmada

Only the Deployment for the Karmada API server will receive the argocd.argoproj.io/instance: karmada label. Other resources deployed by the Karmada operator, including (but not limited to) RBAC resources, Services, Endpoints, and Secrets, do not inherit these labels. This is due to a limitation in the Karmada CRD where label propagation is not implemented for these other managed resources.

This feature request proposes extending the functionality to allow the specified labels in the Kind: Karmada resource to be applied to all resources managed by the Karmada operator.

Why is this needed:

The ability to apply custom labels consistently across all resources managed by the Karmada operator is crucial for several use cases. The core requirement is to enable users to label and annotate all Kubernetes resources without arbitrary restrictions.

A primary use case is the integration with tools like ArgoCD. By applying specific labels (and potentially annotations) to all resources managed by the Karmada operator, users can accurately track these resources within the ArgoCD application's resource tree. This ensures proper visualization, monitoring, and management of the entire Karmada ecosystem within ArgoCD, streamlining operations and improving observability.

[zhzhuang-zju]

@spolspol There was a similar request before, which was to customize annotations for the Karmada API Server services. The API is as follows:

karmada/operator/pkg/apis/operator/v1alpha1/type.go

Lines 315 to 318 in 13bbcaa

	// ServiceAnnotations is an extra set of annotations for service of karmada apiserver.
	// more info: https://github.com/karmada-io/karmada/issues/4634
	// +optional
	ServiceAnnotations map[string]string `json:"serviceAnnotations,omitempty"`

You want to be able to apply the labels from the Kind: Karmada resource to all resources managed by the Karmada operator. Do you have any ideas regarding the implementation?

[RainbowMango]

A primary use case is the integration with tools like ArgoCD. By applying specific labels (and potentially annotations) to all resources managed by the Karmada operator, users can accurately track these resources within the ArgoCD application's resource tree. This ensures proper visualization, monitoring, and management of the entire Karmada ecosystem within ArgoCD, streamlining operations and improving observability.

I'm not similar to the resource tree here. Any documents from ArgoCD or would you like to show it at the Karmada community meeting?

[spolspol]

@RainbowMango While ArgoCD served as a motivating example for this post, labels and annotations have far broader applications. In ArgoCD, they are used to collet and track objects that are deployed via Kind: Application.

[spolspol]

Another use case that came up while I am testing: annotations on Service for LoadBalancer type to configure a load balancer. While ArgoCD use case is probably less common, use of Service annotations for LoadBalancers is very widespread. Lack of ability to set these makes operator unless for provisioning Karmada in any other configuration than ClusterIP/NodePort.

[spolspol]

Btw. should this be a bug instead of feature request?

[RainbowMango]

While ArgoCD use case is probably less common,

I don't know as I don't get how you use the ArgoCD with the operator yet.

Another use case that came up while I am testing: annotations on Service for LoadBalancer type to configure a load balancer.

I think the operator supports the use case, like the ServiceAnnotations for KarmadaAPIServer:

karmada/operator/pkg/apis/operator/v1alpha1/type.go

Lines 326 to 329 in a29335e

	// ServiceAnnotations is an extra set of annotations for service of karmada apiserver.
	// more info: https://github.com/karmada-io/karmada/issues/4634
	// +optional
	ServiceAnnotations map[string]string `json:"serviceAnnotations,omitempty"`

My concern about your use is exactly this use case. Some of the annotations might only make sense to Service which is intended for the load balancers, those should not be on other resources like Secret, Deployments, etc.

[spolspol]

Apologies as I have missed ServiceAnnotations for API server above.

However this still seems restrictive. The ArgoCD labels for example are needed if you want to trace ownership of resources in argo tree, so I would want to be able to set it on ALL resources created by operator, not just Deployment or API service. There are probably many other custom labels or annotations that different setups use, so why not make it available via Karmada CRD? Additionally this restriction is not present in Karmada chart, so why the inconsistency?

The problem here is that this implementation dictates the user where labels/annotations can be set, making it is very opinionated, rather than provide flexibility for making this decision to the user. Additionally, there is inconsistency between different methods of installation, this restricts ability to move from chart to operator, etc. because if you happen to set a label on every service for whatever valid or invalid reason, you can't do that in operator. This is anti-pattern.

TLDR; there should be an option to add custom labels and annotations to all resources created via the operator, why restrict it?

[RainbowMango]

Yes, it makes sense to me to have a field (I mean in the .spec.) to hold the annotations/labels which could be added to all resources created by the operator.

By the way, are you able to show us a demo about the argo tree thing? Maybe at the community meeting?

[spolspol]

ArgoCD UI draws a tree of resources deployed via its CRs like Project, ApplicationSet and Application, example screenshot (stolen) below:

Additionally this also plays a role in deleting Argo CRs and related resources they deploy, so if I delete a Application, all linked resources will also be deleted.

When Karmada operator creates Karmada stack, it lacks labels that make this UI graphing possible and Kaarmada resources do not show up anywhere in Argo.

ArgoCD provides facility to customise this tracking via labels and annotations: https://argo-cd.readthedocs.io/en/stable/user-guide/resource_tracking/

[RainbowMango]

Thank you~

Additionally this also plays a role in deleting Argo CRs and related resources they deploy, so if I delete a Application, all linked resources will also be deleted.
When Karmada operator creates Karmada stack, it lacks labels that make this UI graphing possible and Kaarmada resources do not show up anywhere in Argo.

For the Karmada operator, I think the feature you are requesting can be used for Argo to draw the map, but the linked resource should be cleaned by the Karmada operator, right? Is there a way to tell Argo not to delete the linked resource? by the way.

[RainbowMango]

ArgoCD provides facility to customise this tracking via labels and annotations: https://argo-cd.readthedocs.io/en/stable/user-guide/resource_tracking/

Thank you for sharing this, maybe this is the feature I've been waiting for years!
I showed the Karmada + Argo Demo within the ArgoCD community (maybe at year 2021 or 2022, not quite sure), where we need a way to draw the full map, currently, only ResourceBinding on this map, but Work is missing because ArgoCD doesn't support build the relationship cross namespaces.

[spolspol]

For the Karmada operator, I think the feature you are requesting can be used for Argo to draw the map, but the linked resource should be cleaned by the Karmada operator, right? Is there a way to tell Argo not to delete the linked resource? by the way.

Yes, operator does clean up resources.

I guess my ArgoCD use-case may be on the narrow side, however annotations and labels have wide range of uses - like tracking ownership of resources across large scale multi-team kubernetes clusters, they can be used for billing and tagging, etc, etc. so my thinking was why not leave it up to the user?