readme updates for new config file changes, docs updates

Author: aarora79

README.md

@@ -88,6 +88,11 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl
 
 ## New in this release
 
+## 2.0.26
+
+1. Bug fix for missing HuggingFace token file.
+1. Config file enhancements
+
 ## 2.0.25
 
 1. Fix bug with an alternate VariantName for SageMaker BYOE.

docs/build.md

@@ -10,6 +10,7 @@ If you would like to build a dev version of `FMBench` for your own development a
 
     ```{.bash}
     pip install poetry mkdocs-material mknotebooks
+    sudo apt-get install tree
     ```
 
 1. Change directory to the `FMBench` repo directory and run poetry build.

docs/cli.md

@@ -0,0 +1,31 @@
+# `FMBench` CLI
+
+Here are the command line options currently supported by the `fmbench` CLI.
+
+```{.bashrc}
+usage: fmbench [-h] --config-file CONFIG_FILE [--role-arn ROLE_ARN] [--local-mode {yes,no}] [--tmp-dir TMP_DIR] [--write-bucket WRITE_BUCKET] -A [key=value]
+
+Run FMBench with a specified config file.
+```
+
+options:  
+  `-h`, `--help`            show this help message and exit  
+
+  `--config-file` CONFIG_FILE
+                        The S3 URI of your Config File  
+
+  `--role-arn` ROLE_ARN   (_Optional_) The ARN of the role to be used for FMBench. If an Amazon SageMaker endpoint is being deployed through FMBench then this role would also be used by that endpoint  
+
+  `--local-mode` {yes,no}  Specify if running in local mode or not. Options: yes, no. Default is no.  
+
+  `--tmp-dir` TMP_DIR    (_Optional_)  An optional tmp directory if fmbench is running in local mode.  
+
+  `--write-bucket` WRITE_BUCKET  Write bucket that is used for sagemaker endpoints in local mode and storing metrics in s3 mode.  
+
+  `-A` key=value        Specify a key value pair which will be used to replace the {key} in the given config file. The key could be anything that you have templatized  
+                        in the config file as `{key}` and it will be replaced with `value`. This comes in handy when using a generic configuration file and replace keys such `model_id`,  
+                        `tp_degree` etc. Note that you are not limited to pre-defined set of keys, you can put any key in the config file as `{key}` and it will get replaced with its value.  
+                        If there are multiple key value pairs, then simply specificy the `-A` option multiple times in the command line.  
+
+```
+

docs/manifest.md

@@ -39,6 +39,10 @@ Here is a listing of the various configuration files available out-of-the-box wi
 [└── embeddings/bge-base-en-v1-5-g5-g4dn-c7-embeddings.yml](configs/embeddings/bge-base-en-v1-5-g5-g4dn-c7-embeddings.yml)  
 **gemma**  
 [└── gemma/config-gemma-2b-g5.yml](configs/gemma/config-gemma-2b-g5.yml)  
+**generic**  
+**└── generic/ec2**  
+[├── generic/ec2/djl.yml](configs/generic/ec2/djl.yml)  
+[└── generic/ec2/llama3.1_djl.yml](configs/generic/ec2/llama3.1_djl.yml)  
 **llama2**  
 **├── llama2/13b**  
 [│   ├── llama2/13b/config-bedrock-sagemaker-llama2.yml](configs/llama2/13b/config-bedrock-sagemaker-llama2.yml)  

manifest.txt

@@ -28,6 +28,8 @@ configs/embeddings/bge-base-en-v1-5-c5-embeddings.yml
 configs/embeddings/bge-base-en-v1-5-g5-embeddings.yml
 configs/embeddings/bge-base-en-v1-5-g5-g4dn-c7-embeddings.yml
 configs/gemma/config-gemma-2b-g5.yml
+configs/generic/ec2/djl.yml
+configs/generic/ec2/llama3.1_djl.yml
 configs/llama2/13b/config-bedrock-sagemaker-llama2.yml
 configs/llama2/13b/config-byo-rest-ep-llama2-13b.yml
 configs/llama2/13b/config-llama2-13b-inf2-g5-p4d.yml

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "fmbench"
-version = "2.0.25"
+version = "2.0.26"
 description ="Benchmark performance of **any Foundation Model (FM)** deployed on **any AWS Generative AI service**, be it **Amazon SageMaker**, **Amazon Bedrock**, **Amazon EKS**, or **Amazon EC2**. The FMs could be deployed on these platforms either directly through `FMbench`, or, if they are already deployed then also they could be benchmarked through the **Bring your own endpoint** mode supported by `FMBench`."
 authors = ["Amit Arora <[email protected]>", "Madhur Prashant <[email protected]>"]
 readme = "README.md"