Support scraping metrics from target running with TLS #1190

pierDipi · 2025-07-18T09:21:29Z

vLLM server can run with TLS and metrics scraping doesn't work currently in that case.

Support setting --model-server-metrics-scheme=http|https and --model-server-metrics-https-insecure-skip-verify=false|true.

The change is backward compatible.

Fixes #1189

netlify · 2025-07-18T09:21:34Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`d26fb0d`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/687fcfc701594e00087c0dac
😎 Deploy Preview	https://deploy-preview-1190--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

k8s-ci-robot · 2025-07-18T09:21:38Z

Welcome @pierDipi!

It looks like this is your first PR to kubernetes-sigs/gateway-api-inference-extension 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/gateway-api-inference-extension has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-18T09:21:39Z

Hi @pierDipi. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

pierDipi · 2025-07-18T09:26:41Z

/cc @nirrozenbaum @ahg-g @kfswain

nirrozenbaum · 2025-07-18T10:19:29Z

/ok-to-test

vLLM server can run with TLS and metrics scraping doesn't work currently in that case. Signed-off-by: Pierangelo Di Pilato <[email protected]>

kfswain · 2025-07-18T21:37:05Z

cmd/epp/runner/runner.go

@@ -136,7 +138,9 @@ var (

 	modelServerMetricsPort = flag.Int("model-server-metrics-port", 0, "Port to scrape metrics from pods. "+
 		"Default value will be set to InferencePool.Spec.TargetPortNumber if not set.")
-	modelServerMetricsPath = flag.String("model-server-metrics-path", "/metrics", "Path to scrape metrics from pods")
+	modelServerMetricsPath                    = flag.String("model-server-metrics-path", "/metrics", "Path to scrape metrics from pods")


Looks great! Do you mind adding these flags (with the same defaults) to the helm chart? Example:

gateway-api-inference-extension/config/charts/inferencepool/templates/epp-deployment.yaml

Line 41 in 6cf8f31

- "--enable-pprof={{ .Values.inferenceExtension.enablePprof }}"

Thanks!

Thanks for the feedback Kellen, done!

kfswain · 2025-07-18T21:37:39Z

Looks great! Just adding a request to update helm also. Thanks!

Signed-off-by: Pierangelo Di Pilato <[email protected]>

elevran

Code wise - LGTM.
Not clear how the CA for the metrics endpoint certificate is added to the HTTPS client configuration in GIE.

elevran · 2025-07-22T11:26:15Z

cmd/epp/runner/runner.go

+		metricsHttpClient = &http.Client{
+			Transport: &http.Transport{
+				TLSClientConfig: &tls.Config{
+					InsecureSkipVerify: *modelServerMetricsHttpsInsecureSkipVerify,


how is the CA added to the TLS context? Is it intended to be added to the default certificate roots in the image?
Without adding the CA to the allowed list - won't you be forced to always use insecure-skip-verify set to true?

I agree, we should actually remove the flag for now and set this to true

@elevran agreed that this is suboptimal. Unfortunately this seems to be blocking downstream & is being pushed to be included in a v0.5.1 patch. WE will move forward with this, but would like @pierDipi to create an issue to track CA validation to help prevent MITM vulnerabilites.

cmd/epp/runner/runner.go

config/charts/inferencepool/values.yaml

Co-authored-by: Abdullah Gharaibeh <[email protected]>

kfswain · 2025-07-22T18:07:14Z

/lgtm
/approve

k8s-ci-robot · 2025-07-22T18:07:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 18, 2025

k8s-ci-robot requested review from danehans and robscott July 18, 2025 09:21

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 18, 2025

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 18, 2025

k8s-ci-robot requested review from ahg-g, kfswain and nirrozenbaum July 18, 2025 09:26

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 18, 2025

pierDipi force-pushed the https-scraping branch 2 times, most recently from 84a7c4d to 5d5825c Compare July 18, 2025 10:48

Support scraping metrics from target running with TLS

bb8696c

vLLM server can run with TLS and metrics scraping doesn't work currently in that case. Signed-off-by: Pierangelo Di Pilato <[email protected]>

pierDipi force-pushed the https-scraping branch from 5d5825c to bb8696c Compare July 18, 2025 10:51

kfswain reviewed Jul 18, 2025

View reviewed changes

Add '--model-server-metrics-*' flags to Helm values and EPP template

5110a95

Signed-off-by: Pierangelo Di Pilato <[email protected]>

pierDipi requested a review from kfswain July 21, 2025 07:39

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

Merge branch 'main' into https-scraping

40e19ae

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

elevran reviewed Jul 22, 2025

View reviewed changes

ahg-g reviewed Jul 22, 2025

View reviewed changes

cmd/epp/runner/runner.go Outdated Show resolved Hide resolved

ahg-g reviewed Jul 22, 2025

View reviewed changes

config/charts/inferencepool/values.yaml Outdated Show resolved Hide resolved

Update config/charts/inferencepool/values.yaml

54dcf0d

Co-authored-by: Abdullah Gharaibeh <[email protected]>

Update cmd/epp/runner/runner.go

d26fb0d

Co-authored-by: Abdullah Gharaibeh <[email protected]>

k8s-ci-robot assigned kfswain Jul 22, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 22, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 22, 2025

k8s-ci-robot merged commit 7f5ccbf into kubernetes-sigs:main Jul 22, 2025
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support scraping metrics from target running with TLS #1190

Support scraping metrics from target running with TLS #1190

pierDipi commented Jul 18, 2025

Uh oh!

netlify bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jul 18, 2025

Uh oh!

k8s-ci-robot commented Jul 18, 2025

Uh oh!

pierDipi commented Jul 18, 2025

Uh oh!

nirrozenbaum commented Jul 18, 2025

Uh oh!

kfswain Jul 18, 2025

Uh oh!

pierDipi Jul 21, 2025

Uh oh!

kfswain commented Jul 18, 2025

Uh oh!

elevran left a comment

Uh oh!

elevran Jul 22, 2025

Uh oh!

ahg-g Jul 22, 2025 •

edited

Loading

Uh oh!

kfswain Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

kfswain commented Jul 22, 2025

Uh oh!

k8s-ci-robot commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Support scraping metrics from target running with TLS #1190

Support scraping metrics from target running with TLS #1190

Conversation

pierDipi commented Jul 18, 2025

Uh oh!

netlify bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

k8s-ci-robot commented Jul 18, 2025

Uh oh!

k8s-ci-robot commented Jul 18, 2025

Uh oh!

pierDipi commented Jul 18, 2025

Uh oh!

nirrozenbaum commented Jul 18, 2025

Uh oh!

kfswain Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

pierDipi Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

kfswain commented Jul 18, 2025

Uh oh!

elevran left a comment

Choose a reason for hiding this comment

Uh oh!

elevran Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

ahg-g Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfswain Jul 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kfswain commented Jul 22, 2025

Uh oh!

k8s-ci-robot commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

netlify bot commented Jul 18, 2025 •

edited

Loading

ahg-g Jul 22, 2025 •

edited

Loading