[Feature] metrics support #3534
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Conflicts: lmdeploy/messages.py lmdeploy/pytorch/engine/engine.py lmdeploy/pytorch/engine/engine_instance.py lmdeploy/pytorch/messages.py lmdeploy/pytorch/paging/scheduler.py
Conflicts: lmdeploy/serve/openai/api_server.py
grimoire
reviewed
Jun 30, 2025
Closed
RunningLeon
reviewed
Jul 7, 2025
RunningLeon
reviewed
Jul 7, 2025
RunningLeon
reviewed
Jul 8, 2025
| - job_name: lmdeploy | ||
| static_configs: | ||
| - targets: | ||
| - '$host_ip:$api_server_port1' # <= Modify this |
Collaborator
There was a problem hiding this comment.
can we config all dp server urls in here and show data in grafana board?
|
I see this has been merged but --enable-metrics is still not working? |
Collaborator
|
@voycey Hi, this feature only works for backend=pytorch, yet the default backend is turbomind. Metrics for turbomind backend will be added in another PR. |
|
Docs dont mention anything about this being limited to PyTorch backend :( Any ETA on Turbomind metrics? Its running incredibly fast for me and I would like to see what the token / second is on this - is there any other way? |
Collaborator
Author
|
@voycey |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Objective
Align with vLLM v1 metrics system and beyond. Here are several key alignments
-- Uses
time.perf_counter()for interval calculations (avoids clock drift issues).-- Gauges: Active requests, cache usage, etc
-- Counters: Token totals, request success / failure counts, etc
-- Histograms: TTFT (Time-To-First-Token), TPOT (Inter-Token Latency), end-to-end latency, etc
-- CLI logging
-- Prometheus & Grafana
We only record critical timestamps and events inside the engine process without further processing. Heavy-weight metrics calculations or publishing are separated from the main loop to minimize overhead.
For convenient Grafana visualization and usage, we align with SGLang.
TODO
prometheus_client, only install / import when neededFeature: Expert information collections (deferred in another PR)generate()and engine_async_loop_main()time.perf_counter()Usage
Start the server with
--enable-metricsMetrics Publishing - Logging
With
--enable-metrics, key metrics (e.g., finished / unfinished / running / waiting requests, token throughputs, cache usage) are printed to the terminal every 10 seconds.Metrics Publishing - Prometheus & Grafana
-- Raw Metrics
Access the raw Prometheus metrics via http://localhost:23333/metrics/ .
You can also curl the metrics endpoint
curl http:///localhost:23333/metrics/to view raw Prometheus results. No extra setups are required for this step.-- Prometheus Panel
Access the Prometheus panel via http://localhost:9090 (
9090is the current default port for the Prometheus panel). You need extra setups to access the Prometheus panel; please check the user guide for details.-- Grafana Panel
Access the Grafana panel via http://localhost:3000 (
3000is the current default port for the Grafana panel). You need extra setups to access the Grafana panel; please check the user guide for details.Request Timeline
The following diagram depicts how we define and calculate time intervals during the request lifecycle, which adheres to vLLM.
Performance Impacts
Tested with Qwen2.5-0.5B / Qwen2.5-7B / Qwen2.5-32B, no obvious performance impacts. (Requires #3627)
Check the following tables for output throughput details. We conducted tests using 1,000 prompts, with input length 1k and output length 1k. Each model was tested three times to reduce the impact of performance fluctuations.
Related Issues & PR