Skip to content

Commit 31ec2cb

Browse files
committed
update README
1 parent e1e7e0a commit 31ec2cb

File tree

1 file changed

+6
-3
lines changed

1 file changed

+6
-3
lines changed

README.md

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -162,11 +162,14 @@ python -m lightllm.server.api_server --model_dir /path/llama-7B \
162162
--max_total_token_num 120000
163163
~~~
164164

165-
The parameter `max_total_token_num` is influenced by the GPU memory of the deployment environment. Use the following script to get the recommended values
165+
The parameter `max_total_token_num` is influenced by the GPU memory of the deployment environment. You can also specify --mem_faction to have it calculated automatically.
166166

167167
~~~shell
168-
python -m lightllm.utils.profile_max_tokens --model_dir /path/llama-7B \
169-
--tp 1
168+
python -m lightllm.server.api_server --model_dir /path/llama-7B \
169+
--host 0.0.0.0 \
170+
--port 8080 \
171+
--tp 1 \
172+
--mem_faction 0.9
170173
~~~
171174

172175
To initiate a query in the shell:

0 commit comments

Comments
 (0)