![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"id": "cef1e2cc",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"id": "80395fc3-cc6e-4a1c-ad70-9d6f91bf6c95",
"metadata": {},
"source": [
"\n",
"\t![](\"https://img.shields.io/badge/Llama_API-Sign_up-4BA9FE?logo=meta\"){kind=icon}
![](\"https://img.shields.io/badge/Llama_API-Documentation-E4E6Eb?logo=meta\"){kind=icon}
\n",
"\t![\"Llama](\"https://img.shields.io/badge/Llama-Model_cards-green?logo=meta\"){kind=icon}
![\"Llama](\"https://img.shields.io/badge/Llama-Documentation-e4e6eb?logo=meta\"){kind=icon}
![\"Hugging](\"https://img.shields.io/badge/Hugging_Face-meta--llama-yellow?logo=huggingface\"){kind=icon}
![\"\"](\"../src/docs/img/Llama4_MoE_Architecture.png\")
\n",
"Where the MoE layer has some routed experts and one shared expert. All tokens go through the shared expert.\n",
"![\"\"](\"../src/docs/img/Llama4_MoE_Layer.png\")
"
]
},
{
"cell_type": "markdown",
"id": "6664fc98-f00f-4e64-a03e-46336541d909",
"metadata": {},
"source": [
"## Models\n",
"\n",
"Llama 4 is a collection of 2 large language models (LLMs):\n",
"\n",
"#### Pretrained & Instruct Models:\n",
"1. `llama 4 Omni` (text+multi image+speech(no OSS) input; text+speech(no OSS) output) - MoE with 16 expets\n",
"1. `llama 17b Reasoning` (text+single image input; text output) - MoE with 16 routed expets and shared experts and capacity factor 1, 17B active parameters, 108B total parameters\n",
"\n",
"\n",
"[Model Card](https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD.md#instruction-tuned-models) [TO DO: UPdate later]\n"
]
},
{
"cell_type": "markdown",
"id": "53dfa636-2e3b-4bc7-952d-161381b9d989",
"metadata": {},
"source": [
"# Llama 4 Reasoning\n",
"\n",
"* *Open-source* \n",
"* *Multimodal:* input: Text, Image(Single); Output: Text\n",
"* *Language Support:* English Only\n"
]
},
{
"cell_type": "markdown",
"id": "14534f5e-135d-4c05-a54d-237f83224033",
"metadata": {},
"source": [
"## Getting Llama 4 Reasoning [TO DO: Update Later]\n",
"\n",
"Large language models are deployed and accessed in a variety of ways, including:\n",
"\n",
"1. **Self-hosting**: Using local hardware to run inference. Ex. running Llama on your Macbook Pro using [llama.cpp](https://github.com/ggerganov/llama.cpp) or running inference with lightweight models in both [Android](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/xnnpack_README.md) and [iOS](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/apple_ios/LLaMA/docs/delegates/xnnpack_README.md) using the [PyTorch ExecuTotch](https://github.com/pytorch/executorch) framework.\n",
"1. **Cloud hosting**: Using a cloud provider to deploy a model. Ex. AWS, Azure, GCP, and others.\n",
"1. **Hosted API**: Llama API as a service. Ex. AWS Bedrock, Replicate, Anyscale, Groq, Together and others."
]
},
{
"cell_type": "markdown",
"id": "53ad1085-58ef-4e68-bb9b-bbc562a0196c",
"metadata": {},
"source": [
"### Hosted APIs\n",
"\n",
"Hosted APIs are the easiest way to get started. We'll use them here. As an example, we'll call Llama 4 using [Llama API](https://llama.developer.meta.com/).\n"
]
},
{
"cell_type": "markdown",
"id": "6598e09b-2055-4aba-9cc9-3cac80b8e058",
"metadata": {},
"source": [
"## Notebook Setup\n",
"\n",
"To install prerequisites run:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e150e71b-b612-4e1a-852b-19f205919d0b",
"metadata": {},
"outputs": [],
"source": [
"!pip install --upgrade pip"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "02d90969-1663-4c5c-87de-b43064df1aba",
"metadata": {},
"outputs": [],
"source": [
"!pip install git+ssh://git@github.com/facebookincubator/llama-api-python.git"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "4a106719-1dca-4024-b3af-dc128e672ed6",
"metadata": {},
"outputs": [],
"source": [
"!pip install matplotlib\n",
"!pip list pillow"
]
},
{
"cell_type": "markdown",
"id": "ab05e73a-41bc-47b0-85b6-594002c1de28",
"metadata": {},
"source": [
"# Prompting Llama 4 Reasoning Models\n",
"\n",
"Prompt engineering is using natural language to produce a desired response from a large language model (LLM).\n",
"\n",
"In this section, we'll focus on Llama 4 17B reasoning model. You'll first learn special tokens that are supported by Llama 4 for prompt formating, then learn how to perform over 10 interesting or practical LLM tasks, including:\n",
"[TO DO: Update]\n",
"1. Llama 4 Prompt Formatting Special Tokens\n",
"1. Multimodal use-cases\n",
" * Image captioning/labeling\n",
" * Cooking/Shopping assistant\n",
" * Travel assistant\n",
"1. Tool calling"
]
},
{
"cell_type": "markdown",
"id": "ea3334a1-83e9-4893-ba1f-ca48d99f83da",
"metadata": {},
"source": [
"## Llama 4 Prompt Formatting Special Tokens\n",
"Here is a list of special tokens that are supported by Llama 4:\n",
"\n",
"| Token | \n", "Description | \n", "
|---|---|
| <|begin_of_text|> | \n", "Specifies the start of the prompt | \n", "
| <|end_of_text|> | \n", "Model will cease to generate more tokens. This token is generated only by the base models. | \n", "
| <|header_start|> and <|header_end|> | \n", "These tokens enclose the role for a particular message. The possible roles are: [system, user and assistant]. | \n", "
| <|eom|> | \n", "End of message. A message represents a possible stopping point for execution where the model can inform the executor that a tool call needs to be made. | \n", "
| <|eot|> | \n", "End of turn. Represents when the model has determined that it has finished interacting with the user message that initiated its response. | \n", "
| <|image_start|> and <|image_end|> | \n", "These tokens enclose the image data in the prompt. | \n", "
| <|patch|> | \n", "This token represents a piece of the tile/ | \n", "
| <|tile_y_separator|> and <|tile_x_separator|> | \n", "These tokens are used to separate the y and x tiles of an image | \n", "
| <|image|> | \n", "In the new architecture, this token now separates the regular sized image information from a downsized version of it that fits in a single tile. | \n", "