Implement PrepareDataPlugin for prefix cache match plugin #1942

rahulgurnani · 2025-12-04T00:52:32Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements PrepareDataPlugin hook for prefix cache match plugin. The goal is to split the prefix cache match plugin into match plugin and scorer.
This is an intermediate step towards migration of the plugin. In a following PR, scorer would be added and the existing prefix cache match plugin would be deprecated.

Did some initial scale testing using vllm:

On this version

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  83.04     
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              12.04     
Output token throughput (tok/s):         24663.03  
Peak output token throughput (tok/s):    52286.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          26192.43  
---------------Time to First Token----------------
Mean TTFT (ms):                          1875.19   
Median TTFT (ms):                        1877.09   
P99 TTFT (ms):                           2368.76   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          31.62     
Median TPOT (ms):                        29.59     
P99 TPOT (ms):                           39.16     
---------------Inter-token Latency----------------
Mean ITL (ms):                           31.62     
Median ITL (ms):                         27.45     
P99 ITL (ms):                            92.53     
==================================================

On head (latest release)

============ Serving Benchmark Result ============
Successful requests:                     1000      
Failed requests:                         0         
Benchmark duration (s):                  101.31    
Total input tokens:                      127000    
Total generated tokens:                  2048000   
Request throughput (req/s):              9.87      
Output token throughput (tok/s):         20215.75  
Peak output token throughput (tok/s):    52972.00  
Peak concurrent requests:                1000.00   
Total Token throughput (tok/s):          21469.36  
---------------Time to First Token----------------
Mean TTFT (ms):                          1885.78   
Median TTFT (ms):                        1851.98   
P99 TTFT (ms):                           2482.30   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          33.57     
Median TPOT (ms):                        32.10     
P99 TPOT (ms):                           48.02     
---------------Inter-token Latency----------------
Mean ITL (ms):                           33.57     
Median ITL (ms):                         27.89     
P99 ITL (ms):                            93.49

@kaushikmitr / @BenjaminBraunDev FYI

Which issue(s) this PR fixes:

Addresses #1924

Does this PR introduce a user-facing change?:

yes. this PR adds a new feature flag "prepareDataPlugins"

k8s-ci-robot · 2025-12-04T00:52:34Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2025-12-04T00:52:37Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`ab82246`
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/693a372fb2aa7c0008df86b0
😎 Deploy Preview	https://deploy-preview-1942--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

liu-cong · 2025-12-04T21:26:30Z

pkg/epp/datalayer/plugins/data_types.go

+	PrefixCacheMatchPrecentKey = "PrefixCacheMatchPercentKey"
+)
+
+type PrefixCacheMatchPercent struct {


I suggest making this something more generic like PrefixCacheInfo so we can easily extend. One of things we will likely need to add is the cache tier info.

Instead (or in addition) of providing the percentage, let's provide the match length. The idea is to provide more "raw" data for consumers. Percentage is just something the scorer cares about, the latency predictor may work better with the length.

I renamed it to PrefixCacheInfo to make it easy for extension. Predictor is presently only trained to depend on prefix cache match percent. I think updating this to length may not be suitable in the short term. @BenjaminBraunDev keep me honest here. Thanks!

I think @liu-cong 's opinion carries the most weight here.

The reccomendations here seem reasonable, aggregating data for easier ergonomics of an unproven, experimental plugin, at the expense of overall usability does not seem a worth tradeoff.

I'm not sure why this comment was resolved

Updated the info to contain prefix cache related info. Thanks

rahulgurnani · 2025-12-08T15:08:35Z

/assign @kfswain

cmd/epp/runner/runner.go

pkg/epp/datalayer/factory.go

pkg/epp/datalayer/plugins/approximateprefix/data_types.go

kfswain · 2025-12-08T22:17:46Z

pkg/epp/requestcontrol/request_control_config.go

 // If a cycle is detected, it returns an error.
-func (c *Config) PrepareDataPluginGraph() error {
+func (c *Config) PrepareDataPluginGraph(enablePrepareDataPlugins bool) error {
+	if !enablePrepareDataPlugins {


Related to the other comment, lets remove this check from here.

Done, thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin_test.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

kfswain · 2025-12-08T22:22:24Z

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer_test.go

+	return keys
+}
+
+func TestPrefixCacheScorer_Score(t *testing.T) {


Can we have a unit test that also creates the legacy prefix scorer, and test to make sure the created scores have identical output? We should be able to validate the behavior is the same in these unit tests

I removed the scorer from this PR since I am not using it yet. I plan to add the scorer back in a separate PR. Scoping this PR to just adding prepare data step so that predictor can use it. Thanks!

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

pkg/epp/scheduling/framework/plugins/scorer/prefix_cache_match_scorer.go

pkg/epp/scheduling/framework/plugins/multi/prefix/plugin.go

cmd/epp/runner/runner.go

kfswain · 2025-12-10T18:48:59Z

pkg/epp/requestcontrol/request_control_config.go


 // PrepareDataPluginGraph creates data dependency graph and sorts the plugins in topological order.
 // If a cycle is detected, it returns an error.
 func (c *Config) PrepareDataPluginGraph() error {


The DAG should be built, completely agnostic of if any PrepareData plugins are in use. Will add suggestions to help clarify what I mean

kfswain · 2025-12-10T18:52:05Z

pkg/epp/requestcontrol/request_control_config.go

+	if len(c.prepareDataPlugins) == 0 {
+		return nil
+	}
 	dag := buildDAG(c.prepareDataPlugins)
 	plugins, err := sortPlugins(dag, c.prepareDataPlugins)
 	if err != nil {


Suggested change

if len(c.prepareDataPlugins) == 0 {

return nil

}

dag := buildDAG(c.prepareDataPlugins)

plugins, err := sortPlugins(dag, c.prepareDataPlugins)

if err != nil {

dag := buildDAG(c.allPlugins)

plugins, err := sortPlugins(dag, c.allPlugins)

if err != nil {

This is semi psuedocode. But I hope this kind of illustrates what I mean. We should always create the DAG, entirely agnostic of which plugins are in use.

Ohh. I understand. Added a TODO for it and address it as a fast follow in the next PR. Let me know if that's reasonable. Thanks!

… comments

rahulgurnani · 2025-12-10T20:23:45Z

/retest

kfswain · 2025-12-11T16:22:05Z

/lgtm
/approve

k8s-ci-robot · 2025-12-11T16:22:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, rahulgurnani

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Dec 4, 2025

k8s-ci-robot requested review from ahg-g and liu-cong December 4, 2025 00:52

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 4, 2025

rahulgurnani changed the title ~~Implement PreapreDataPlugin for prefix cache match plugin~~ Implement PrepareDataPlugin for prefix cache match plugin Dec 4, 2025

rahulgurnani force-pushed the pc-m branch from b8c71a8 to d6556ed Compare December 4, 2025 18:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2025

liu-cong reviewed Dec 4, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from c67979f to 5752e58 Compare December 6, 2025 05:15

rahulgurnani requested a review from liu-cong December 6, 2025 05:16

rahulgurnani marked this pull request as ready for review December 6, 2025 05:16

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2025

k8s-ci-robot requested review from elevran and robscott December 6, 2025 05:16

k8s-ci-robot assigned kfswain Dec 8, 2025

rahulgurnani requested a review from kfswain December 8, 2025 18:51

rahulgurnani force-pushed the pc-m branch from c330833 to ee9a486 Compare December 8, 2025 19:46

rahulgurnani mentioned this pull request Dec 8, 2025

Deprecate existing prefix cache match plugin in favor of new datalayer #1970

Open

kfswain reviewed Dec 8, 2025

View reviewed changes

zetxqx reviewed Dec 8, 2025

View reviewed changes

rahulgurnani force-pushed the pc-m branch from ee9a486 to 598b4b2 Compare December 9, 2025 01:23

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 9, 2025

rahulgurnani force-pushed the pc-m branch from c673022 to 74cf6a5 Compare December 9, 2025 02:03

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 9, 2025

rahulgurnani requested review from kfswain and zetxqx December 10, 2025 01:14

kfswain reviewed Dec 10, 2025

View reviewed changes

rahulgurnani added 14 commits December 10, 2025 19:53

Add PrepareRequestData method for the prefix cache plugin

6187ace

Add prefix cache match scorer

178632e

Ensure prefix cache plugin implements all preprae data plugin methods

b619766

Enable prepare data plugins behind a feature flag

e100653

Add static check to ensure plugin implements PrepareDataPlugin interface

4bd3d6b

Rename prefix cache match

e7e7eae

Add feature gate for prepare data plugin

0fce568

Update prefix cache match to have total and match length

963d164

Add more tests, register prefix cache scorer and address other review…

1145aea

… comments

Move prefix cache scorer out of this PR

c10dd35

Fix rebase errors

80e07e0

Update directory structure

4a2c02e

Update TODOs

b198efd

Validate type as well in dag validation

05d7565

rahulgurnani force-pushed the pc-m branch from 59e9016 to f4142a8 Compare December 10, 2025 19:54

rahulgurnani requested a review from kfswain December 10, 2025 20:11

rahulgurnani force-pushed the pc-m branch 3 times, most recently from 672ca5e to 1a0c28c Compare December 11, 2025 03:10

Fix rebase errors, add TODO

ab82246

rahulgurnani force-pushed the pc-m branch from 1a0c28c to ab82246 Compare December 11, 2025 03:14

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 11, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 11, 2025

k8s-ci-robot merged commit db2b7ce into kubernetes-sigs:main Dec 11, 2025
12 checks passed

Implement PrepareDataPlugin for prefix cache match plugin #1942

Implement PrepareDataPlugin for prefix cache match plugin #1942

Conversation

rahulgurnani commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Dec 4, 2025

Uh oh!

netlify bot commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for gateway-api-inference-extension ready!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rahulgurnani commented Dec 10, 2025

Uh oh!

kfswain commented Dec 11, 2025

Uh oh!

k8s-ci-robot commented Dec 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rahulgurnani commented Dec 4, 2025 •

edited

Loading

netlify bot commented Dec 4, 2025 •

edited

Loading

rahulgurnani Dec 9, 2025 •

edited

Loading

rahulgurnani Dec 10, 2025 •

edited

Loading