Any methods to support params update?

[Auraithm]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

Do Lmdeploy has any methods to update the model params or the model instances like vllm?

Reproduction

none

Environment

none

Error traceback

none

[lvhan028]

LMDeploy has already supported it.
Weights can be updated through api_server's /update_weights endpoint

[Auraithm]

Is there any reference document?

[Auraithm]

In addition to updating via the API, does the pipeline also support updating weights? Is there any documentation available?

[lvhan028]

Sorry, we haven't prepared the user guide yet.
Here’s how api_server updates the model weights:

@router.post('/update_weights', dependencies=[Depends(check_api_key)])
def update_params(request: UpdateParamsRequest, raw_request: Request = None):
    """Update weights for the model."""
    VariableInterface.async_engine.engine.update_params(request)
    return JSONResponse(content=None)

Note that pipeline is an alias for async_engine, so in theory, the pipeline also supports this feature.
However, users are responsible for constructing the request object — which is not a trivial task

@RunningLeon Could you help writing the user guide?

[Auraithm]

How should this request be constructed? And, if it's offline inference, is it possible to update directly from the model instead of converting it to base64 or something similar?

[RunningLeon]

@Auraithm Now we only support update through http api. Here is a simple example.

from lmdeploy.utils import serialize_state_dict
import requests
BASE_URL = 'http://0.0.0.0:24545'
api_key = 'sk-xxx'

headers = {
                "Content-Type": "application/json",
                "Authorization": f"Bearer {api_key}",
            }
segmented_state_dict: List[Dict[str, torch.Tensor]] = ...
num_segment = len(segmented_state_dict)
for seg_idx in range(num_segment):
    serialized_data = serialize_state_dict(segmented_state_dict[seg_idx])
    data = dict(serialized_named_tensors=serialized_data, finished=seg_idx == num_segment-1)
    response = requests.post(f"{BASE_URL}/update_weights", headers=headers, json=data)
    assert response.status_code == 200, f"response.status_code = {response.status_code}"

[Auraithm]

What is the version of lmdeploy? I used the version 0.10.1.

    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 123, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 109, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 389, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 290, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/starlette/concurrency.py", line 38, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/anyio/_backends/_asyncio.py", line 962, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/serve/openai/api_server.py", line 990, in update_params
    VariableInterface.async_engine.engine.update_params(request)
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/mp_engine/base.py", line 54, in update_params
    return self._collective_rpc('update_params', request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/mp_engine/zmq_engine.py", line 137, in _collective_rpc
    return self.rpc_client.call(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/mp_engine/zmq_rpc.py", line 249, in call
    raise Exception(reply['error'])
Exception: Not Implemented.

[RunningLeon]

@Auraithm hi, sorry for the trouble. You have to change source code to set distributed_executor_backend='ray' for pytorch backend engine before start the server.
You can change the source code for a workaround since this argument is not added in lmdeploy serve api_server cli command.
A PR will fix this later.

[Auraithm]

export LMDEPLOY_EXECUTOR_BACKEND=ray
lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \
    --model-name "SDAR-8B-Chat" \
    --server-port 23333 \
    --tp 8
``` What needs to be modified? I added this environment variable by referring to the documentation, but I have never used ray. Do I need to modify anything?

[Auraithm]

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-11-24 05:24:48,353 ERROR worker.py:430 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.update_params() (pid=47624, ip=10.247.56.69, actor_id=d1a95221a84a339f49f49b5d01000000, repr=<lmdeploy.pytorch.engine.executor.ray_executor.RayWorkerWrapper object at 0x7f1f0ebc3c20>)
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/executor/base_worker.py", line 124, in update_params
    self.model_agent.update_params(request)
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/model_agent.py", line 1028, in update_params
    weights = [(k, _construct(v)) for k, v in weights]
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/model_agent.py", line 1021, in _construct
    return func(*args).clone()
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/reductions.py", line 181, in rebuild_cuda_tensor
    storage = storage_cls._new_shared_cuda(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/storage.py", line 1457, in _new_shared_cuda
    return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[RunningLeon]

export LMDEPLOY_EXECUTOR_BACKEND=ray
lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \
    --model-name "SDAR-8B-Chat" \
    --server-port 23333 \
    --tp 8
``` What needs to be modified? I added this environment variable by referring to the documentation, but I have never used ray. Do I need to modify anything?

add backend_config.distributed_executor_backend='ray' in here

lmdeploy/lmdeploy/serve/openai/api_server.py

Line 1288 in 1984a5d

backend_config.enable_mp_engine = True

[Auraithm]

export LMDEPLOY_EXECUTOR_BACKEND=ray
lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \
    --model-name "SDAR-8B-Chat" \
    --server-port 23333 \
    --tp 8
``` What needs to be modified? I added this environment variable by referring to the documentation, but I have never used ray. Do I need to modify anything?

add backend_config.distributed_executor_backend='ray' in here

lmdeploy/lmdeploy/serve/openai/api_server.py

Line 1288 in 1984a5d

backend_config.enable_mp_engine = True

I added this, but still failed with the error above:

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2025-11-24 05:24:48,353 ERROR worker.py:430 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::RayWorkerWrapper.update_params() (pid=47624, ip=10.247.56.69, actor_id=d1a95221a84a339f49f49b5d01000000, repr=<lmdeploy.pytorch.engine.executor.ray_executor.RayWorkerWrapper object at 0x7f1f0ebc3c20>)
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
           ^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/executor/base_worker.py", line 124, in update_params
    self.model_agent.update_params(request)
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/model_agent.py", line 1028, in update_params
    weights = [(k, _construct(v)) for k, v in weights]
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/lmdeploy/pytorch/engine/model_agent.py", line 1021, in _construct
    return func(*args).clone()
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/reductions.py", line 181, in rebuild_cuda_tensor
    storage = storage_cls._new_shared_cuda(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/storage.py", line 1457, in _new_shared_cuda
    return torch.UntypedStorage._new_shared_cuda(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: invalid resource handle
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[Auraithm]

···

export LMDEPLOY_EXECUTOR_BACKEND=ray
lmdeploy serve api_server /inspire/hdd/global_user/liuxiaoran-240108120089/public/SDAR-8B-Chat \
    --model-name "SDAR-8B-Chat" \
    --server-port 23333 \
    --tp 8
``` What needs to be modified? I added this environment variable by referring to the documentation, but I have never used ray. Do I need to modify anything?

add backend_config.distributed_executor_backend='ray' in here

lmdeploy/lmdeploy/serve/openai/api_server.py

Line 1288 in 1984a5d

backend_config.enable_mp_engine = True

import torch
from typing import List, Dict
from transformers import AutoModelForCausalLM
import requests
from lmdeploy.utils import serialize_state_dict

# 1. 启动 API 服务
# lmdeploy serve /path/to/model --backend pytorch --server-port 23333

# 2. 加载训练后的模型
trained_model = AutoModelForCausalLM.from_pretrained(
    "xxx/SDAR-8B-Chat",
    torch_dtype=torch.float16,
    trust_remote_code=True,
    device_map="cuda"
)

# 3. 准备权重并分段
state_dict = trained_model.state_dict()

# 将 state_dict 分段（可根据需要调整分段策略）
segmented_state_dict: List[Dict[str, torch.Tensor]] = [state_dict]

# 4. 发送权重更新请求
BASE_URL = 'http://0.0.0.0:23333'
api_key = 'sk-xxx'

headers = {
    "Content-Type": "application/json",
    "Authorization": f"Bearer {api_key}",
}

num_segment = len(segmented_state_dict)
for seg_idx in range(num_segment):
    print(f"Serializing segment {seg_idx + 1}/{num_segment}...")
    serialized_data = serialize_state_dict(segmented_state_dict[seg_idx])
    data = dict(serialized_named_tensors=serialized_data, finished=seg_idx == num_segment-1)
    
    print(f"Sending segment {seg_idx + 1}/{num_segment}...")
    response = requests.post(f"{BASE_URL}/update_weights", headers=headers, json=data)
    
    if response.status_code != 200:
        print(f"Error response status: {response.status_code}")
        print(f"Error response text: {response.text}")
        assert False, f"Failed to update weights: status_code={response.status_code}"
    
    print(f"Segment {seg_idx + 1}/{num_segment} updated successfully")

print("All weights updated successfully!")

This is my code.