Inference tutorial - Part 3 of e2e series [WIP] #2343
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2343
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6a96697 with merge base 5239ce7 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
jainapurva
added
the
topic: documentation
docs/source/inference.rst
Outdated
| +----------------------------+----------------+------------------------------+ | ||
|
|
||
|
|
||
| Sparsity Integration |
There was a problem hiding this comment.
this should not be a separate section I think, it can be merged into Float8 Dynamic Quantization section, and just mention for more quantization/sparsity, please see https://huggingface.co/docs/transformers/main/en/quantization/torchao
| print("Response:", output_text[0][len(prompt):]) | ||
| [Optional] Float8 Dynamic Quantization + Semi-structured (2:4) sparsity |
There was a problem hiding this comment.
@jerryzh168 @jcaip Does this look good? Should I keep sparsity as a optional section or just mention it in note
There was a problem hiding this comment.
can we just add to huggingface torchao page?
| Memory Benchmarking | ||
| -------------------- | ||
|
|
||
| **Memory Usage Comparison**: |
| vllm serve pytorch/Phi-4-mini-instruct-float8dq --tokenizer microsoft/Phi-4-mini-instruct -O3 | ||
| Inference with vLLM |
There was a problem hiding this comment.
should we move this after Inference with Transformers
|
|
||
| vLLM automatically leverages torchao's optimized kernels when serving quantized models, providing significant throughput improvements. | ||
|
|
||
| Setting up vLLM with Quantized Models |
There was a problem hiding this comment.
nit: this doesn't have to be a new section I think
| Performance Breakdown | ||
| ===================== | ||
|
|
||
| When using vLLM with torchao: |
There was a problem hiding this comment.
this is not a comprehensive list, probably just remove, do we have a exhaustive list of all the techniques that we support?
|
Hi @jainapurva, by the way I'm adding a 
|
No description provided.