[Feature Request] Enable Serverless LLM to work with dynamically provisioned Serverless GPU providers

[andrei3131]

Prerequisites

• I have searched existing issues and reviewed documentation.

Problem Description

Serverless GPU does not support dynamically adding workers from Serverless GPU providers such as RunPod or Modal. Current architecture relies on static registration of workers which incurs infrastructure cost even with no network traffic.

Proposed Solution

• Use sllm_store model format to improve loading times in dynamically provisioned workers as Serverless GPU workloads
• Use a new model store to warm up the local cache on the newly provision Serverless GPU workload

Alternatives Considered

No response

Additional Context

No response

Importance

Nice to have