Elias James Howell b3067b55dc fix typos and spelling errors		%!s(int64=2) %!d(string=hai) anos
..
hf-text-generation-inference	b3067b55dc fix typos and spelling errors	%!s(int64=2) %!d(string=hai) anos
README.md	822b37bbc0 Update README.md	%!s(int64=2) %!d(string=hai) anos
chat_completion.py	557e881fcc aliginng the pad token with HF latest	%!s(int64=2) %!d(string=hai) anos
chat_utils.py	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos
chats.json	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos
inference.py	7ec390bfc8 aliging special tokens in toeknizer with HF latest	%!s(int64=2) %!d(string=hai) anos
model_utils.py	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos
safety_utils.py	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos
samsum_prompt.txt	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos
vLLM_inference.py	4767f09ecd Initial commit	%!s(int64=2) %!d(string=hai) anos

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

inference script script provides support for Hugging Face accelerate and PEFT fine tuned models.
vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.
The hf-text-generation-inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.