What happened:
Getting plenty of errors like:

2022/01/21 00:39:53 http: TLS handshake error from 10.97.36.76:42820: remote error: tls: bad certificate

In my cluster, this IP relates to Prometheus server. Looking at the service configuration I found that the Prometheus scraper annotations were pointing at the webhook TLS port instead of the metrics port.

What you expected to happen:
No errors in the logs

How to reproduce it (as minimally and precisely as possible):
Install Prometheus in the cluster where the webhook is running with scrape enabled for service endpoints.

Anything else we need to know?:

Environment:

• AWS Region: us-east-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks4
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.21
• Webhook Version: master branch at a65cc3d9c61cf6fc43f0f985818c474e0867d786