Matthias Reso 7702d702cc Add missing file extension 2 лет назад
..
chat_completion 3ddf755539 Move examples into subfolders 2 лет назад
code_llama ccda6fb8ca Move inference scripts into example folder 2 лет назад
hf_text_generation_inference ccda6fb8ca Move inference scripts into example folder 2 лет назад
vllm 3ddf755539 Move examples into subfolders 2 лет назад
README.md 42d09ac202 Document installation of optional dependencies 2 лет назад
finetuning.py 7702d702cc Add missing file extension 2 лет назад
inference.py ccda6fb8ca Move inference scripts into example folder 2 лет назад
multi_node.slurm 682943551f Move quickstart nb, finetuning script and slurm config to examples 2 лет назад
quickstart.ipynb 682943551f Move quickstart nb, finetuning script and slurm config to examples 2 лет назад
samsum_prompt.txt ccda6fb8ca Move inference scripts into example folder 2 лет назад

README.md

Inference

This folder contains inference examples for Llama 2. So far, we have provided support for three methods of inference:

  1. inference script script provides support for Hugging Face accelerate, PEFT and FSDP fine tuned models.

  2. vllm/inference.py script takes advantage of vLLM's paged attention concept for low latency.

  3. The hf_text_generation_inference folder contains information on Hugging Face Text Generation Inference (TGI).

For more in depth information on inference including inference safety checks and examples, see the inference documentation here.

Note The vLLM example requires additional dependencies. Please refer to installation section of the main README.md for details