You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+5-23Lines changed: 5 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -149,7 +149,6 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl
149
149
1. Don't delete endpoints for the bring your own endpoint case.
150
150
1. Fix bug with business summary chart.
151
151
152
-
153
152
### v1.0.32
154
153
155
154
1. Report enhancements: New business summary chart, config file embedded in the report, version numbering and others.
@@ -305,27 +304,10 @@ For some enterprise scenarios it might be desirable to run `FMBench` directly on
305
304
306
305
1. All metrics are stored in the `/tmp/fmbench-write` directory created automatically by the `fmbench` package. Once the run completes all files are copied locally in a `results-*` folder as usual.
307
306
308
-
### Steps to run
309
-
310
-
1. `pip install` the `FMBench` package from PyPi.
311
-
312
-
1. Create a config file using one of the config files available [here](https://github.com/aws-samples/foundation-model-benchmarking-tool/tree/main/src/fmbench/configs).
313
-
1. The configuration file is a YAML file containing configuration for all steps of the benchmarking process. It is recommended to create a copy of an existing config file and tweak it as necessary to create a new one for your experiment.
314
-
315
-
1. Create the read and write buckets as mentioned in the prerequisites section. Mention the respective directories for your read and write buckets within the config files.
316
-
317
-
1. Run the `FMBench` tool from the command line.
318
-
319
-
```{.bash}
320
-
# the config file path could be an S3 path and https path
321
-
# or even a path to a file on the local filesystem
322
-
fmbench --config-file /path/to/config/file
323
-
```
324
-
325
-
1. Depending upon the experiments in the config file, the `FMBench` run may take a few minutes to several hours. Once the run completes, you can find the report and metrics in the write S3 bucket set in the [config file](https://github.com/aws-samples/foundation-model-benchmarking-tool/blob/main/src/fmbench/configs/config-mistral-7b-tgi-g5.yml#L12). The report is generated as a markdown file called `report.md` and is available in the metrics directory in the write S3 bucket.
326
-
327
307
## Results
328
308
309
+
Depending upon the experiments in the config file, the `FMBench` run may take a few minutes to several hours. Once the run completes, you can find the report and metrics in the local `results-*` folder in the directory from where `FMBench` was run. The rpeort and metrics are also written to the write S3 bucket set in the [config file](https://github.com/aws-samples/foundation-model-benchmarking-tool/blob/main/src/fmbench/configs/config-mistral-7b-tgi-g5.yml#L12).
310
+
329
311
Here is a screenshot of the `report.md` file generated by `FMBench`.
@@ -368,15 +350,15 @@ You can create an internal `FMBench` website to view results from multiple runs
368
350
Choose any config file from the [`bedrock`](https://github.com/aws-samples/foundation-model-benchmarking-tool/tree/main/src/fmbench/configs/bedrock) folder and either run these directly or use them as templates for creating new config files specific to your use-case. Here is an example for benchmarking the `Llama3` models on Bedrock.
Choose any config file from the model specific folders, for example the [`Llama3`](https://github.com/aws-samples/foundation-model-benchmarking-tool/tree/main/src/fmbench/configs/llama3) folder for `Llama3` family of models. These configuration files also include instructions for `FMBench` to first deploy the model on SageMaker using your configured instance type and inference parameters of choice and then run the benchmarking. Here is an example for benchmarking `Llama3-8b` model on an `ml.inf2.24xlarge` and `ml.g5.12xlarge` instance.
@@ -408,7 +390,7 @@ The steps for deploying the model on your EKS cluster are described below.
408
390
1. Run the `Llama3-8b` benchmarking using the command below (replace the config file as needed for a different model). This will first deploy the model on your EKS cluster and then run benchmarking on the deployed model.
1. As the model is getting deployed you might want to run the following `kubectl` commands to monitor the deployment progress. Set the _model_namespace_ to `llama3` or `mistral` or a different model as appropriate.
0 commit comments