Tutorial: deploy Llama 2 7B with ncnn

[lrw04]

Download weights from Meta

https://ai.meta.com/resources/models-and-libraries/llama-downloads/

Convert the weights to llama2.c format

*instructions written for Linux.

cd <your-work-directory>
git clone https://github.com/lrw04/llama2.c-to-ncnn
cd llama2.c-to-ncnn
python -m venv venv
source venv/bin/activate
pip install numpy
python convert.py --outfile <output file> <model directory>

Convert the model in llama2.c format into an ncnn model

Note: this process is done using a converter custom-built for Llama 2 models to avoid using pnnx because of pnnx being memory-inefficient.

make
./convert <output file in previous step> <ncnn model name>

Use the provided inference code

./inference <ncnn model name> <prompt> <total-out-tokens>

[LRY89757]

Wonderful! Great Job!

[lrw04]

int8 quantization work is in progress. The new layer code is being upstreamed at PR #5007.

[lrw04]

Pushed int8 support anyway. You need to pull the branch from my PR to use quantization.

[lrw04]

The same instructions also work for the 13B model. The 70B model is not tested due to insufficient memory on my machines.

[ahmedmustahid]

Download weights from Meta

https://ai.meta.com/resources/models-and-libraries/llama-downloads/

Convert the weights to llama2.c format

*instructions written for Linux.

cd <your-work-directory>
git clone https://github.com/lrw04/llama2.c-to-ncnn
cd llama2.c-to-ncnn
python -m venv venv
source venv/bin/activate
pip install numpy
python convert.py --outfile <output file> <model directory>

Convert the model in llama2.c format into an ncnn model

Note: this process is done using a converter custom-built for Llama 2 models to avoid using pnnx because of pnnx being memory-inefficient.

make
./convert <output file in previous step> <ncnn model name>

Use the provided inference code

./inference <ncnn model name> <prompt> <total-out-tokens>

Would you please share the custom made converter? I intend to run llama2 with ncnn.

[lrw04]

You would be using the custom converter if you were following the instructions. (Look at what you are downloading) Also, this is very flaky and you should probably use something like llama.cpp instead. They support Llama 3 AND a lot of other models.

[ha-seungwon]

Is it possible to do the inference with 4bit?