pluggable BBR framework proposal

fixes typos and minor reformatting issues

shortens description: omits the sequence diagram

minor

Author: davidbreitgand

docs/proposals/XXXX-pluggable-bbr-framework/README.md

@@ -0,0 +1,163 @@
+# Pluggable Body-Based Routing (BBR) Framework 
+
+Author(s): @davidbreitgand @srampal
+
+## Proposal Status
+
+***Draft***
+
+## Summary
+
+The Gateway API Inference Extension (v1.2.1) includes an initial implementation of Body-Based Routing (BBR). Currently, BBR provides a single capability: it extracts the model name from the request body and adds it to the `X-Gateway-Model-Name` header. This header is then used to route the request to the appropriate InferencePool and its associated Endpoint Picker Extension (EPP) instances.
+
+The current BBR implementation is limited and lacks extensibility. Similar to the [pluggability introduced in the scheduling subsystem](../0845-scheduler-architecture-proposal/README.md), BBR should support custom extensions without requiring modifications to the GIE code base.
+
+This proposal introduces a plugin architecture for BBR that allows developers to implement custom logic. Plugins could be organized into a chain or DAG for ordered and concurrent execution.
+
+See [this document](https://docs.google.com/document/d/1So9uRjZrLUHf7Rjv13xy_ip3_5HSI1cn1stS3EsXLWg/edit?tab=t.0#heading=h.55jwocr94axs) for additional context amd reference.
+
+## Goals
+
+The pluggable BBR Framework aims at addressing the following goals
+
+- Avoid monolithic architecture
+- Mimic pluggability and configurability of the scheduling subsystem without coupling between the two
+- Enable organizing plugins into a topology for ordered and concurrent execution
+- Avoid redundant recurrent body parsing across plugins in a topology for the sake of performance
+- Limit changes to the BBR feature to avoid any changes in the rest of the code base
+- Follow best practices and experience from the Scheduling subsystem
+  pluggability effort. For example, extending the system to support the above
+  should be through implementing well defined `Plugin` interfaces and registering
+  them in the BBR subsystem; any configuration would be done in the
+  same way (e.g., code and/or configuration file)
+- Reuse common code from EPP, such as `TypedName`, wherever make sense, but avoid reusing specialized code with non-BBR functionality to avoid abuse
+- Enable extensible collection and registration of metrics using lessons from the pluggable scheduling sub-system
+- Provide reference plugin implementations.
+
+## Non-Goals
+
+- Modify existing GIE abstractions
+- Fully align plugins, registries, and factories across BBR and EPP
+- Dynamically reconfigure plugins and plugin topologies at runtime
+
+## Proposal
+
+### Overview
+
+There is an embedded `BBRPlugin` interface building on the `Plugin` interface adopted from EPP. This interface should be implemented by any BBR plugin. Each pluigin is identified by its `TypedName` (adopted from EPP), where `TypedName().Type` gives the string representing the type of the plugin and `TypedName().Name()` returns the string representing the plugins implementation. BBR is refactored to implement the registered factory pattern. To that end, a `PluginRegistry` interface and its implementation are added to register `BBRPlugin` factories and concrete implementations created by the factories.
+In addition, a `PluginsChain` interface is defined to define an order of plugin executions. In the future, `PluginsChain` will be replaced by `PluginsDAG` to allow for more complex topological order and concurrency.
+
+`PluginsChain` only contains ordered `BBRPlugin` types registered in the `PluginRegistry`. `RequestPluginsChain` and `ResponsePluginsChain` are optionally configured for handling requests and responses respectively. If no configuration is provided, default `PluginsChain` instances will be configured automatically.
+
+Depending on a `BBRPlugin` functionality and implementation, the plugin might require full or selective body parsing. To save the parsing overhead, if there is at least one `BBRPlugin` in the `PluginsChain` that requires full body parsing, the parsing is performed only once into a shared official appropriate `openai-go` struct (either `openai.CompletionNewParams`  or `openai.ChatCompletionNewParams` depending on the request endpoint). This struct is shared for read-only to all plugins in the chain. Each `BBRplugin` receives the shared struct by value. If a plugin needs to mutate the body, in the initial implementation, it MUST work on its own copy, and the a mutated body is returned separately by each plugiin.
+
+### Suggested Components
+
+The sketch of the proposed framework is shown in the figure below.
+<img src="./images/pluggable-framework-architecture-sketch.png" alt="Components of the proposed framework" width="1000" />
+
+### Suggested BBR Pluggable Framework Interfaces
+
+```go
+//Base BBRPlugin interface
+type BBRPlugin interface {
+    plugins.Plugin             //imported from "sigs.k8s.io/gateway-api-inference-extension/pkg/epp/plugins"
+    RequiresFullParsing() bool //specifies whether a full parsing of the body is required (to facilitate efficient memory sharing across plugins in a plugins chain)
+}
+
+// MetadataExtractor is an example of embedded interface for extending BBR functionality.
+// Plugins implementing this interface extract metadata from an inbound OpenAI message by keys specified.
+// Plugins implementing the MetadataExtractor interface should be read-only.
+// One specific implementation of this interface is Model extraction from the body and returning 
+// it in the headers map by the custom key X-Gateway-Model-Name. 
+// The current BBR implementation will be rewritten accordingly to implement this interface.
+// NOTE: the chain runner will safely merge the headers map from all the plugins in the plugins chain 
+type MetadataExtractor interface {
+    BBRPlugin
+    Extract(ctx context.Context,
+        requestBodyBytes []byte,
+        metaDataKeys []string,
+        sharedMemory interface{}) (headers map[string]string, err error) //valid shared memory MUST be either openai.ChatCompletionNewParams or openai.CompletionNewParams struct
+}                                                                        
+//NOTE: in the initial PR, sharedMemory struct will be omitted for simplicity
+
+// ModelSelector is an example embedded interface for extending BBR fubctionality.
+// It defines an interface for selecting a model based on the body of the inbound request
+// model is the model selected 
+// The model might be different from the original model requested.
+// In this case, the Model in the body is replaced by the chosen model name
+// The headers map contains are the HTTP headers added by Select(...) on the inbound request message before passing the message back to the gateway provider (e.g., Istio/Envoy)
+// X-Gateway-Model-Name is MUSTbe set always by any ModelSelector implementation 
+// Plugins implementing the ModelSelector interface should always return mutatedBodyBytes even if the body is not mutated: in this case, the original bodyBytes MUST be returned 
+//Typically, but not neceassarily an implementation of ModelSelector will require a full body parsing to get exposed to the full body (e.g., for semantic routing)
+
+type ModelSelector interface {
+     BBRPlugin
+     Select(ctx context.Context, requestBodyBytes []byte, sharedMemory interface{}) (
+            headers map[string]string,
+            mutatedBodyBytes []byte, err error)
+}
+
+// placeholder for BBRPlugin constructors
+type PluginFactoryFunc func() bbrplugins.BBRPlugin //concrete constructors are assigned to this type
+
+// PluginRegistry defines operations for managing plugin factories and plugin instances
+type PluginRegistry interface {
+    RegisterFactory(typeKey string, factory PluginFactoryFunc) error //constructors
+    RegisterPlugin(plugin bbrplugins.BBRPlugin) error //registers a plugin instance (the instance MUST be created via the factory first)
+    GetFactory(typeKey string) (PluginFactoryFunc, error)
+    GetPlugin(typeKey string) (bbrplugins.BBRPlugin, error)
+    GetFactories() map[string]PluginFactoryFunc
+    GetPlugins() map[string]bbrplugins.BBRPlugin
+    ListPlugins() []string
+    ListFactories() []string
+    CreatePlugin(typeKey string) (bbrplugins.BBRPlugin, error)
+    ContainsFactory(typeKey string) bool
+    ContainsPlugin(typeKey string) bool
+    String() string    //human readable string for logging
+}
+
+// PluginsChain is used to define a specific order of execution of the BBRPlugin instances stored in the registry
+// The BBRPlugin instances 
+type PluginsChain interface {
+    AddPlugin(typeKey string, registry PluginRegistry) error                    //to be added to the chain the plugin should be registered in the registry first
+    AddPluginAtInd(typeKey string, i int, r PluginRegistry) error               //only affects the instance of the plugin chain
+    GetPlugin(index int, registry PluginRegistry) (bbrplugins.BBRPlugin, error) //retrieves i-th plugin as defined in the chain from the registry
+    Length() int
+    ParseChatCompletion(data []byte) (openai.ChatCompletionNewParams, error)    //parses the bytes slice into an appropriate openai-go struct
+    ParseCompletion(data []byte) (openai.CompletionNewParams, error)            //likewise
+    GetSharedMemory(which string) interface{}  //returns an appropriate shared open-ai struct dependent on whether which 
+                                               //corresponds to Completion or ChatCompletion endpoint requested in the body 
+    String() string
+}
+//NOTE: for simplicity, in the initial PR, PluginsChain instance will be defined for request for request only
+
+// ------------------------- Supported plugin interfaces -----------------------------------------------------------
+// The key in the SuppotedInterfaces map is the BBRPlugin type and the value is a slice of supported implementations
+// For this BBRPlugin type.
+// Edit SupportedInterfaces map when new interfaces and/or implementations are added/removed
+var SupportedInterfaces = map[string][]string{
+    "MetadataExtractor": {"simple-model-extractor"}, 
+    "ModelSelector":     {"semantic-model-selector-uniroute", "semantic-model-selector-knn"}, //future
+    "GuardRail":         {"obscenity-blocker", "pid-disclosure-blocker"}, //future
+}
+```
+
+### Implementation Phases
+
+The pluggable framework will be implemented iteratively over several phases.
+
+1. Introduce `BBRPlugin` `MetadataExtractor`, interface, registry, plugins chain, sample plugin implementation (`SimpleModelExtraction`) and its factory. Plugin configuration will be implemented via command line flags
+1. Introduce a second plugin interface, `ModelSelector` and sample plugin implementation
+1. Introduce shared struct (shared among the plugins of a plugins chain)
+1. Introduce an interface for guardrail plugin, introduce simple reference implementation, experiment with plugins chains on request and response messages
+1. Refactor metrics as needed to work with the new pluggable framework
+1. Implement configuration via manifests similar to those in EPP
+1. Implement `PluginsDAG` to allow for more complex topological order and concurrency.
+1. Continously learn lessons from this implementation and scheduling framework to improve the implementation
+1. Aim at aligning and cross-polination with the [AI GW WG]("https://github.com/kubernetes-sigs/wg-ai-gateway").
+
+## Open Questions
+
+1. More elaborate shared memory architecture for the best performance
+1. TBA