Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we have pushed an update to the development branch (and the Triton backend) this July 16, 2024.

This update includes:

• Features
  • Supported ReDrafter Speculative Decoding, see “ReDrafter” section in docs/source/speculative_decoding.md.
  • Supported padding removal for BERT, thanks to the contribution from @Altair-Alpha in support remove_input_padding for BertForSequenceClassification models #1834.
  • Added in-flight batching support for GLM 10B model.
• API
  • [BREAKING CHANGE] max_output_len is removed from trtllm-build command, if you want to limit sequence length on engine build stage, specify max_seq_len.
  • Added a version API to the C++ library, a cpp/include/tensorrt_llm/executor/version.h file is going to be generated.

Thanks,
The TensorRT-LLM Engineering Team