[Feature]: Add prometheus metrics for tracking request queue depth #17764
Description
The Feature
Add prometheus metrics for tracking the number of queued/running requests at the router level.
Motivation, pitch
In high-concurrency scenarios, our team is considering applying rate limiting to control request concurrency at the router layer. We’d like to see metrics that expose both the number of actively running requests and the number of queued requests. These metrics will help us analyze bottlenecks, tune concurrency limits, and better understand overall system behavior under load.
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
No
Twitter / LinkedIn details
No response