readme updates

Author: aarora79

README.md

@@ -83,6 +83,11 @@ Configuration files are available in the [configs](./src/fmbench/configs) folder
 
 ## New in this release
 
+## 2.1.2
+
+1. Deepseek prompt updates.
+1. Handle case for < 1 txn/minute.
+
 ## 2.1.1
 
 1. Optimized prompt templates and config files for DeepSeek-R1 and Amazon Nova for `ConvFinQA` and `LongBench` datasets.

docs/deepseek.md

@@ -15,3 +15,20 @@ python main.py --config-file configs/deepseek/deepseek-convfinqa.yml
 ```
 Change the `--config-file` parameter to [`configs/deepseek/deepseek-longbench.yml`](https://github.com/aws-samples/fmbench-orchestrator/blob/main/configs/deepseek/deepseek-longbench.yml) or [`configs/deepseek/deepseek-openorca.yml`](https://github.com/aws-samples/fmbench-orchestrator/blob/main/configs/deepseek/deepseek-openorca.yml) to use other datasets for benchmarking. These orchestrator files test various Deepseek-R1 distilled models on `g6e` instances, edit this file as per your requirements. 
 
+## Benchmark Deepseek-R1 quantized models on Amazon EC2
+
+👉 Make sure your account has enough service quota for vCPUs to run this benchmark. We would be using `g6e.12xlarge` instance for this test.
+
+
+1. Create a `g6e.12xlarge` instance and run the `DeepSeek-R1 1.58b quantized` model on this instance by following the steps 1 through 8 described [here](https://github.com/aarora79/deepseek-r1-ec2?tab=readme-ov-file#quantized-models).
+
+1. Follow steps 1 through 5 [here](https://aws-samples.github.io/foundation-model-benchmarking-tool/benchmarking_on_ec2.html#benchmarking-on-an-instance-type-with-nvidia-gpus-or-aws-chips) to setup `FMBench` on this instance.
+
+1. Next run the following command to benchmark LongBench 
+
+    ```{.bashrc}
+    TMP_DIR=/tmp
+    fmbench --config-file $TMP_DIR/fmbench-read/configs/deepseek/config-deepseek-r1-quant1.58-longbench-byoe.yml --local-mode yes --write-bucket placeholder --tmp-dir $TMP_DIR > fmbench.log 2>&1
+    ```
+
+1. Once the run completes you should see the benchmarking results in a folder called `results-DeepSeek-R1-quant-1.58bit-g6e.12xl` present in your current directory.

docs/manifest.md

@@ -37,6 +37,7 @@ Here is a listing of the various configuration files available out-of-the-box wi
 [└── byoe/config-model-byo-sagemaker-endpoint.yml](configs/byoe/config-model-byo-sagemaker-endpoint.yml)  
 **deepseek**  
 [├── deepseek/config-deepseek-r1-ollama.yml](configs/deepseek/config-deepseek-r1-ollama.yml)  
+[├── deepseek/config-deepseek-r1-quant1.58-longbench-byoe.yml](configs/deepseek/config-deepseek-r1-quant1.58-longbench-byoe.yml)  
 [├── deepseek/config-deepseek-r1-vllm-convfinqa.yml](configs/deepseek/config-deepseek-r1-vllm-convfinqa.yml)  
 [├── deepseek/config-deepseek-r1-vllm-longbench.yml](configs/deepseek/config-deepseek-r1-vllm-longbench.yml)  
 [└── deepseek/config-deepseek-r1-vllm-openorca.yml](configs/deepseek/config-deepseek-r1-vllm-openorca.yml)  

manifest.txt

@@ -30,6 +30,7 @@ configs/bedrock/config-nova-all-models.yml
 configs/bert/config-distilbert-base-uncased.yml
 configs/byoe/config-model-byo-sagemaker-endpoint.yml
 configs/deepseek/config-deepseek-r1-ollama.yml
+configs/deepseek/config-deepseek-r1-quant1.58-longbench-byoe.yml
 configs/deepseek/config-deepseek-r1-vllm-convfinqa.yml
 configs/deepseek/config-deepseek-r1-vllm-longbench.yml
 configs/deepseek/config-deepseek-r1-vllm-openorca.yml

pyproject.toml

@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "fmbench"
-version = "2.1.1"
+version = "2.1.2"
 description = "Benchmark performance of **any Foundation Model (FM)** deployed on **any AWS Generative AI service**, be it **Amazon SageMaker**, **Amazon Bedrock**, **Amazon EKS**, or **Amazon EC2**. The FMs could be deployed on these platforms either directly through `FMbench`, or, if they are already deployed then also they could be benchmarked through the **Bring your own endpoint** mode supported by `FMBench`."
 authors = [
     { name = "Amit Arora", email = "[email protected]" },