{ "cells": [ { "cell_type": "markdown", "id": "e1c90e9d-27a9-4d02-b5ab-b9eed4d77aba", "metadata": {}, "source": [ "# Google Calendar Assistant with with Llama 3.2 3B Tool Calling\n", "\n", "This notebook showcases how to go about building a digital assistant to schedule meetings with the Llama 3.2 3B model. The core concepts used to implement this are Prompt Engineering and Tool Calling. This demo shows how Llama can be used to interact with 3rd party apps like Google Contacts & Google Calendar and schedule a meeting requested by the user. Even though we are using prompt engineering to achieve this, the approach described doesn't degrade the model's ability to answer general queries. This approach can extended to perform other tasks in a similar manner without affecting the quality of other tasks\n", "\n", "\n", "## Approach\n", "\n", "Instead of using a complex system prompt with multiple conditions & expecting Llama to perform various tasks accurately out of the box, the approach here is to treat this as a 2 step process\n", "- Determine user intent - Task classification\n", "- Take action for the specific task using Tool Calling\n", "\n", "\n", "\n", "In the diagram shown below,\n", "- system prompt 1 determines the classfication of the query\n", "- In steps 2 & 3, we classify the task being requested.\n", "- system prompt 2 is chosen based on the classification result\n", "- Steps 4 & 5 implement the classified task.\n", "- For the sake of demo, we show 2 classes: General & Meeting\n", "\n", "![Tool Calling Flow Diagram](./assets/flow_diagram.png)\n", "\n", "Both these tasks have a specific prompt. We use the same model with different system prompts depending on the classification result.\n", "Additionally, this demo also showcases how Llama can be used to do double tool calling with 1 prompt. In the case of Meeting, Llama returns 2 function calls in Step 5\n", "```\n", "{{\"name\": \"John Constantine\"}}\n", "{{\"date\": \"Mar 31 \", \"time\": \"5:30 pm\", \"attendees\": \"John Constantine\"}}\n", "```\n", "\n", "Actions based on tool calling output\n", "- The google_contact function call returned by the model is used to call [People API](https://developers.google.com/people) to look up the email address of the person of interest\n", "- The email address from the previous step is used to call [Calendar API](https://developers.google.com/calendar) along with the other information in the google_calendar toolcall output returned by the model \n", "\n", "The end result is that a google meeting is scheduled with the person of interest at the date & time specified\n" ] }, { "cell_type": "markdown", "id": "9ca337cc-7908-4daf-be57-66eeb4f8f703", "metadata": {}, "source": [ "## Load Llama 3.2-3B-Instruct model\n", "\n", "This demo also intends to show that tool calling can be done effectively even with the 3B model" ] }, { "cell_type": "code", "execution_count": 7, "id": "bb085f6c-f069-48ef-8085-32f8a301b090", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "efc10dfe3b754f5eb17271d10da9e9d1", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Loading checkpoint shards: 0%| | 0/2 [00:00\"\n", " start, N = 0, len(llama_output)\n", " count = 0\n", " email = None\n", "\n", " # In this example, we are expecting Llama to produce 2 tool calling outputs\n", " while count < 2:\n", " begin, end = llama_output[start:].find(prefix) + len(prefix) + start, llama_output[start:].find(suffix) + start\n", " func_name, params = llama_output[begin:end-1].split(\">{\")\n", " end += len(suffix)\n", " start = end\n", " count += 1\n", " params = json.loads(params.strip())\n", " \n", " if not email and func_name in functions:\n", " email = functions[func_name](**params)\n", " \n", " elif email and func_name in functions:\n", " attendees = {}\n", " attendees[\"name\"] = params[\"attendees\"]\n", " attendees[\"email\"] = email\n", " params[\"attendees\"] = attendees\n", "\n", " result += \"\\n-----------------------------------------------------\\n\"\n", " result += functions[func_name](**params)\n", " return result\n" ] }, { "cell_type": "markdown", "id": "a94cecf8-239a-4d57-9f73-2281786ad6f2", "metadata": {}, "source": [ "### Functions for calling Llama for the 2 tasks: general & setup meeting" ] }, { "cell_type": "code", "execution_count": 3, "id": "a75b7fac-7553-40e5-aedd-67ebdc98cce7", "metadata": {}, "outputs": [], "source": [ "def general_query(user_prompt, result):\n", " \"\"\"\n", " Function to call Llama for general queries\n", " \"\"\"\n", " \n", " result += \"\\n-----------------------------------------------------\\n\"\n", " result += \"SECOND PASS\"\n", " result += \"\\n-----------------------------------------------------\\n\"\n", "\n", " max_new_tokens = 100\n", " SYSTEM_PROMPT1 = f\"\"\"\n", "\n", " You are a helpful assistant. Answer the query in {max_new_tokens} words\n", " \"\"\"\n", "\n", " prompt = SYSTEM_PROMPT1 + f\"\\n\\nQuestion: {user_prompt} \\n\" + \"\\n Answer:\\n\"\n", " input_prompt_length = len(prompt)\n", "\n", " input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n", "\n", " output = model.generate(**input_ids, max_new_tokens=max_new_tokens)\n", "\n", " llama_output = tokenizer.decode(output[0], skip_special_tokens=True)\n", "\n", "\n", " result += llama_output\n", "\n", " return result\n", "\n", "\n", "def setup_meeting(user_prompt, result):\n", " \"\"\"\n", " Function to call Llama for setting up meeting using Google APIs\n", " \"\"\"\n", " \n", " result += \"\\n-----------------------------------------------------\\n\"\n", " result += \"SECOND PASS\"\n", " result += \"\\n-----------------------------------------------------\\n\"\n", "\n", " max_new_tokens = 100\n", " SYSTEM_PROMPT2 = \"\"\"\n", "\n", " You are a helpful assistant.\n", " Here are the tools you have access to: google_contact and google_calendar which you can use for ONLY setting up meetings:\n", " Use the function 'google_contact' to: look up a name and get the email address\n", " {\n", " \"name\": \"google_contact\",\n", " \"description\": \"Get email address\",\n", " \"parameters\": {\n", " \"name\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"name\",\n", " \"required\": true\n", " }\n", " }\n", " }\n", " Use the function 'google_calendar' to: schedule a meeting\n", " {\n", " \"name\": \"google_calendar\",\n", " \"description\": \"Schedule a meeting\",\n", " \"parameters\": {\n", " \"date\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"date\",\n", " \"required\": true\n", " },\n", " \"time\"\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"time\",\n", " \"required\": true\n", " },\n", " \"attendees\"\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"name\",\n", " \"required\": true\n", " }\n", " }\n", " }\n", " Identify the name in the query, lookup the name using 'google_contact' to find the email address of the contact \n", " and then use 'google_calndar' to schedule a meeeting with this person.\n", " DO NOT reply with any other reasoning or exaplanation\n", " ONLY reply in the following format with no prefix or suffix:\n", " {{\"example_name\": \"example_value\"}}\n", " Reminder:\n", " - ONLY return the function call and don't reply anything else\n", " - DO NOT give any explanation\n", " - I REPEAT ONLY return function calls\n", " - Function calls MUST follow the specified format, start with \n", " - Required parameters MUST be specified\n", " - First return google_contact and then google_calendar\n", " \"\"\"\n", "\n", "\n", " prompt = SYSTEM_PROMPT2 + f\"\\n\\nQuestion: {user_prompt} \\n\"\n", "\n", " input_prompt_length = len(prompt)\n", "\n", " input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n", "\n", " output = model.generate(**input_ids, max_new_tokens=max_new_tokens)\n", "\n", " llama_output = tokenizer.decode(output[0], skip_special_tokens=True)\n", "\n", "\n", " result += llama_output\n", "\n", "\n", " result += \"\\n\\n\\n\"\n", " result += process_llama_output(llama_output[input_prompt_length:])\n", " return result" ] }, { "cell_type": "markdown", "id": "e1260bc4-9a04-47b6-be29-f82d5c949642", "metadata": {}, "source": [ "## Demo with a general query\n", "\n", "To help explain the final output, we also print the system prompts being used in both the calls to Llama" ] }, { "cell_type": "code", "execution_count": 4, "id": "48a57e28-82cf-4f8f-b9ac-2b357f7aea7e", "metadata": {}, "outputs": [], "source": [ "user_prompt = \"Tell me about Paris\"" ] }, { "cell_type": "code", "execution_count": 5, "id": "f3c35189-4152-45a8-81ab-386466d6284c", "metadata": {}, "outputs": [], "source": [ "SYSTEM_PROMPT = \"\"\"\n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \"\"\"\n", "\n", "prompt = SYSTEM_PROMPT + f\"\\n\\nQuestion: {user_prompt} \\n\" + \"\\n Answer:\\n\"\n", "\n", "input_prompt_length = len(prompt)" ] }, { "cell_type": "code", "execution_count": 8, "id": "58801f79-08eb-49f2-bb35-aff1e026e41c", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n", "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "-----------------------------------------------------\n", "FIRST PASS\n", "-----------------------------------------------------\n", "\n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \n", "\n", "Question: Tell me about Paris \n", "\n", " Answer:\n", " general \n", "\n", "\n", "\n", "-----------------------------------------------------\n", "SECOND PASS\n", "-----------------------------------------------------\n", "\n", "\n", " You are a helpful assistant. Answer the query in 100 words\n", " \n", "\n", "Question: Tell me about Paris \n", "\n", " Answer:\n", "Paris, the City of Light, is the capital of France, known for its stunning architecture, art, fashion, and romance. The city is home to iconic landmarks like the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum, which houses the Mona Lisa. The city's charming streets, cafes, and gardens offer a unique blend of history, culture, and beauty. Visitors can stroll along the Seine River, explore the Latin Quarter, and indulge in French cuisine, wine\n" ] } ], "source": [ "\n", "result = \"\\n-----------------------------------------------------\\n\"\n", "result += \"FIRST PASS\"\n", "result += \"\\n-----------------------------------------------------\\n\"\n", "\n", "input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n", "\n", "output = model.generate(**input_ids, max_new_tokens=2)\n", "\n", "llama_output = tokenizer.decode(output[0], skip_special_tokens=True)\n", "\n", "result += llama_output\n", "result += \"\\n\"\n", "\n", "\n", "classification = llama_output[input_prompt_length:].strip()\n", "\n", "\n", "if \"meeting\" in classification:\n", " result = setup_meeting(prompt, result)\n", "elif \"general\" in classification:\n", " result = general_query(user_prompt, result)\n", "print(result)" ] }, { "cell_type": "code", "execution_count": 9, "id": "93a74c4b-a918-43a2-8099-a4f0efb8902f", "metadata": {}, "outputs": [], "source": [ "user_prompt = \"Can you describe how an internal combustion engine works \"\n", "\n", "prompt = SYSTEM_PROMPT + f\"\\n\\nQuestion: {user_prompt} \\n\" + \"\\n Answer:\\n\"\n", "\n", "input_prompt_length = len(prompt)" ] }, { "cell_type": "code", "execution_count": 10, "id": "b87f590d-d697-4f8f-af8d-5099c59019f0", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n", "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "-----------------------------------------------------\n", "FIRST PASS\n", "-----------------------------------------------------\n", "\n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \n", "\n", "Question: Can you describe how an internal combustion engine works \n", "\n", " Answer:\n", "general\n", "\n", "\n", "\n", "-----------------------------------------------------\n", "SECOND PASS\n", "-----------------------------------------------------\n", "\n", "\n", " You are a helpful assistant. Answer the query in 100 words\n", " \n", "\n", "Question: Can you describe how an internal combustion engine works \n", "\n", " Answer:\n", "An internal combustion engine is a type of heat engine that generates power by burning fuel, typically gasoline or diesel, inside a combustion chamber within the engine. The engine consists of several major components, including cylinders, pistons, crankshafts, and valves. Here's a simplified overview of the process:\n", "\n", "1. Air and fuel are drawn into the cylinders through the intake valves.\n", "2. The mixture is compressed by the pistons, creating a high-pressure mixture.\n", "3. The spark plug ignites the\n" ] } ], "source": [ "\n", "result = \"\\n-----------------------------------------------------\\n\"\n", "result += \"FIRST PASS\"\n", "result += \"\\n-----------------------------------------------------\\n\"\n", "\n", "input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n", "\n", "output = model.generate(**input_ids, max_new_tokens=2)\n", "\n", "llama_output = tokenizer.decode(output[0], skip_special_tokens=True)\n", "\n", "result += llama_output\n", "result += \"\\n\"\n", "\n", "\n", "classification = llama_output[input_prompt_length:].strip()\n", "\n", "\n", "if \"meeting\" in classification:\n", " result = setup_meeting(prompt, result)\n", "elif \"general\" in classification:\n", " result = general_query(user_prompt, result)\n", "print(result)" ] }, { "cell_type": "markdown", "id": "c022c93e-4886-40f4-b3a3-3384896f5d7c", "metadata": {}, "source": [ "## Demo with a set up meeting query, which uses tool calling & calls Google APIs\n", "\n", "To help explain the final output, we also print the system prompts being used in both the calls to Llama" ] }, { "cell_type": "code", "execution_count": 53, "id": "437a1467-ff3c-4559-b778-fe725045b0ac", "metadata": {}, "outputs": [], "source": [ "user_prompt = \"Schedule a meeting with John Constantine on Mar 31 at 5:30 pm\"" ] }, { "cell_type": "code", "execution_count": 54, "id": "e346c0de-1fd4-43a5-b067-6347a4a87f83", "metadata": {}, "outputs": [], "source": [ "SYSTEM_PROMPT = \"\"\"\n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \"\"\"\n", "\n", "prompt = SYSTEM_PROMPT + f\"\\n\\nQuestion: {user_prompt} \\n\" + \"\\n Answer:\\n\"\n", "\n", "input_prompt_length = len(prompt)" ] }, { "cell_type": "code", "execution_count": 55, "id": "fe5c51e0-facb-4889-9bea-0496db6b15d7", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n", "Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "-----------------------------------------------------\n", "FIRST PASS\n", "-----------------------------------------------------\n", "\n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \n", "\n", "Question: Schedule a meeting with John Constantine on Mar 31 at 5:30 pm \n", "\n", " Answer:\n", " meeting\n", "\n", "-----------------------------------------------------\n", "SECOND PASS\n", "-----------------------------------------------------\n", "\n", "\n", " You are a helpful assistant.\n", " Here are the tools you have access to: google_contact and google_calendar which you can use for ONLY setting up meetings:\n", " Use the function 'google_contact' to: look up a name and get the email address\n", " {\n", " \"name\": \"google_contact\",\n", " \"description\": \"Get email address\",\n", " \"parameters\": {\n", " \"name\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"name\",\n", " \"required\": true\n", " }\n", " }\n", " }\n", " Use the function 'google_calendar' to: schedule a meeting\n", " {\n", " \"name\": \"google_calendar\",\n", " \"description\": \"Schedule a meeting\",\n", " \"parameters\": {\n", " \"date\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"date\",\n", " \"required\": true\n", " },\n", " \"time\"\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"time\",\n", " \"required\": true\n", " },\n", " \"attendees\"\": {\n", " \"param_type\": \"string\",\n", " \"description\": \"name\",\n", " \"required\": true\n", " }\n", " }\n", " }\n", " Identify the name in the query, lookup the name using 'google_contact' to find the email address of the contact \n", " and then use 'google_calndar' to schedule a meeeting with this person.\n", " DO NOT reply with any other reasoning or exaplanation\n", " ONLY reply in the following format with no prefix or suffix:\n", " {{\"example_name\": \"example_value\"}}\n", " Reminder:\n", " - ONLY return the function call and don't reply anything else\n", " - DO NOT give any explanation\n", " - I REPEAT ONLY return function calls\n", " - Function calls MUST follow the specified format, start with \n", " - Required parameters MUST be specified\n", " - First return google_contact and then google_calendar\n", " \n", "\n", "Question: \n", "\n", " Classify the given prompt\n", " A message can be classified as one of the following categories: meeting, general.\n", "\n", " Examples:\n", " - meeting: \"Can you schedule a meeting with Ankith on Dec 1 at 1 pm.\"\n", " - general: \"Tell me about San Francisco\"\n", "\n", " Reminder:\n", " - ONLY return the classification label and NOTHING else\n", " - DO NOT give any explanation or examples\n", " \n", "\n", "Question: Schedule a meeting with John Constantine on Mar 31 at 5:30 pm \n", "\n", " Answer:\n", " \n", " {{\"name\": \"John Constantine\"}}\n", " {{\"date\": \"Mar 31\", \"time\": \"5:30 pm\", \"attendees\": \"John Constantine\"}}\n", "\n", "\n", "\n", "-----------------------------------------------------\n", "\n", "Event created with John Constantine with email constantine@example.com on Mar 31 at 5:30 pm: \n", " https://www.google.com/calendar/event?eid=MG1rMnBsbzRvaGYwdXUzc2tydnZvNGZyNWMgYW5raXRodGVzdHRvb2xAbQ\n" ] } ], "source": [ "\n", "result = \"\\n-----------------------------------------------------\\n\"\n", "result += \"FIRST PASS\"\n", "result += \"\\n-----------------------------------------------------\\n\"\n", "\n", "input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n", "\n", "output = model.generate(**input_ids, max_new_tokens=2)\n", "\n", "llama_output = tokenizer.decode(output[0], skip_special_tokens=True)\n", "\n", "result += llama_output\n", "result += \"\\n\"\n", "\n", "\n", "classification = llama_output[input_prompt_length:].strip()\n", "\n", "if \"meeting\" in classification:\n", " result = setup_meeting(prompt, result)\n", "elif \"general\" in classification:\n", " result = general_query(user_prompt, result)\n", "print(result)" ] }, { "cell_type": "markdown", "id": "6f2b5788-762e-44d2-be26-b7e5bb823f68", "metadata": {}, "source": [ "## Result\n", "\n", "In your Google Calendar, you should see an invite generated as shown below\n", "\n", "\"Google" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 5 }