Configuring concurrency (num_threads or workers)

[sundaraa-deshaw]

Hi,

Is it possible to configure the underlying hyper server for more concurrency? Tabby fails to respond once the number of parallel requests hits around 30. Is there a way to scale this up or out?

Arun

[wsxiaoys]

Hey, Tabby's concurrency is primarily limited by --parallelism (since version 0.6.0) or LLAMA_CPP_PARALLELISM (in version 0.5.x). The CPU executor is unlikely to be a bottleneck for Tabby's inference purposes.

For further details, please check the follow-up discussion at #889.