Matthias Reso ce9501f22c remove relative imports		%!s(int64=2) %!d(string=hai) anos
..
code_llama	ce9501f22c remove relative imports	%!s(int64=2) %!d(string=hai) anos
hf_text_generation_inference	e804e2ba3b Address more spell checker issues	%!s(int64=2) %!d(string=hai) anos
README.md	18d4c3c580 Fix broken links	%!s(int64=2) %!d(string=hai) anos
__init__.py	207d2f80e9 Make code-llama and hf-tgi inference runnable as module	%!s(int64=2) %!d(string=hai) anos
__main__.py	ce9501f22c remove relative imports	%!s(int64=2) %!d(string=hai) anos
chat_completion.py	ce9501f22c remove relative imports	%!s(int64=2) %!d(string=hai) anos
chat_utils.py	cf678b9bf0 Adjust imports to package structure + cleaned up imports	%!s(int64=2) %!d(string=hai) anos
chats.json	4c9cc7d223 Move modules into separate src folder	%!s(int64=2) %!d(string=hai) anos
checkpoint_converter_fsdp_hf.py	ce9501f22c remove relative imports	%!s(int64=2) %!d(string=hai) anos
inference.py	ce9501f22c remove relative imports	%!s(int64=2) %!d(string=hai) anos
model_utils.py	4c9cc7d223 Move modules into separate src folder	%!s(int64=2) %!d(string=hai) anos
safety_utils.py	cf678b9bf0 Adjust imports to package structure + cleaned up imports	%!s(int64=2) %!d(string=hai) anos
samsum_prompt.txt	4c9cc7d223 Move modules into separate src folder	%!s(int64=2) %!d(string=hai) anos
vLLM_inference.py	cf678b9bf0 Adjust imports to package structure + cleaned up imports	%!s(int64=2) %!d(string=hai) anos

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.
vLLM_inference.py script takes advantage of vLLM's paged attention concept for low latency.
The hf_text_generation_inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.