{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "lbfIu_3eEaAh" }, "source": [ "# Using Amazon Bedrock with Llama\n", "\n", "\"Open\n", "\n", "Use this notebook to quickly get started with Llama on Bedrock. You can access the Amazon Bedrock API using the AWS Python SDK.\n", "\n", "In this notebook, we will give you some simple code to confirm to get up and running with the AWS Python SDK, setting up credentials, looking up the list of available Meta Llama models, and using bedrock to inference.\n", "\n", "### Resources\n", "Set up the Amazon Bedrock API - https://docs.aws.amazon.com/bedrock/latest/userguide/api-setup.html\n", "\n", "### To connect programmatically to an AWS service, you use an endpoint. Amazon Bedrock provides the following service endpoints:\n", "\n", "* **bedrock** – Contains control plane APIs for managing, training, and deploying models.\n", "* **bedrock-runtime** – Contains runtime plane APIs for making inference requests for models hosted in Amazon Bedrock.\n", "* **bedrock-agent** – Contains control plane APIs for creating and managing agents and knowledge bases.\n", "* **bedrock-agent-runtime** – Contains control plane APIs for managing, training, and deploying models.\n", "\n", "### Prerequisite\n", "Before you can access Amazon Bedrock APIs, you will need an AWS Account, and you will need to request access to the foundation models that you plan to use. For more information on model access - https://docs.aws.amazon.com/bedrock/latest/userguide/model-access.html\n", "\n", "#### Setting up the AWS CLI (TBD)\n", "https://docs.aws.amazon.com/bedrock/latest/userguide/api-setup.html#api-using-cli-prereq\n", "\n", "#### Setting up an AWS SDK\n", "https://docs.aws.amazon.com/bedrock/latest/userguide/api-setup.html#api-sdk\n", "\n", "#### Using SageMaker Notebooks\n", "https://docs.aws.amazon.com/bedrock/latest/userguide/api-setup.html#api-using-sage\n", "\n", "For more information on Amazon Bedrock, please refer to the official documentation here: https://docs.aws.amazon.com/bedrock/" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "gVz1Y1HpxWdv" }, "outputs": [], "source": [ "# install packages\n", "# !python3 -m pip install -qU boto3\n", "from getpass import getpass\n", "from urllib.request import urlopen\n", "import boto3\n", "import json" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Security Note\n", "\n", "For this notebook, we will use `getpass()` to reference your AWS Account credentials. This is just to help you get-started with this notebook more quickly. Otherwise, the we recommend that you avoid using getpass for your AWS credentials in a Jupyter notebook. It's not secure to expose your AWS credentials in this way. Instead, consider using AWS IAM roles or environment variables to securely handle your credentials.\n" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JHu-V-4ayNjB", "outputId": "4a1e856b-3ab1-480c-97fd-81a9b9e3724b" }, "outputs": [], "source": [ "\n", "# Set default AWS region\n", "default_region = \"us-east-1\"\n", "\n", "# Get AWS credentials from user input (not recommended for production use)\n", "AWS_ACCESS_KEY = getpass(\"AWS Access key: \")\n", "AWS_SECRET_KEY = getpass(\"AWS Secret key: \")\n", "SESSION_TOKEN = getpass(\"AWS Session token: \")\n", "AWS_REGION = input(f\"AWS Region [default: {default_region}]: \") or default_region\n" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "def create_bedrock_client(service_name):\n", " \"\"\"\n", " Create a Bedrock client using the provided service name and global AWS credentials.\n", " \"\"\"\n", " return boto3.client(\n", " service_name=service_name,\n", " region_name=AWS_REGION,\n", " aws_access_key_id=AWS_ACCESS_KEY,\n", " aws_secret_access_key=AWS_SECRET_KEY,\n", " aws_session_token=SESSION_TOKEN\n", " )" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "def list_all_meta_bedrock_models(bedrock):\n", " \"\"\"\n", " List all Meta Bedrock models using the provided Bedrock client.\n", " \"\"\"\n", " try:\n", " list_models = bedrock.list_foundation_models(byProvider='meta')\n", " print(\"\\n\".join(list(map(lambda x: f\"{x['modelName']} : { x['modelId'] }\", list_models['modelSummaries']))))\n", " except Exception as e:\n", " print(f\"Failed to list models: {e}\")" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "def invoke_model(bedrock_runtime, model_id, prompt, max_gen_len=256):\n", " \"\"\"\n", " Invoke a model with a given prompt using the provided Bedrock Runtime client.\n", " \"\"\"\n", " body = json.dumps({\n", " \"prompt\": prompt,\n", " \"temperature\": 0.1,\n", " \"top_p\": 0.9,\n", " \"max_gen_len\":max_gen_len,\n", " })\n", " accept = 'application/json'\n", " content_type = 'application/json'\n", " try:\n", " response = bedrock_runtime.invoke_model(body=body, modelId=model_id, accept=accept, contentType=content_type)\n", " response_body = json.loads(response.get('body').read())\n", " generation = response_body.get('generation')\n", " print(generation)\n", " except Exception as e:\n", " print(f\"Failed to invoke model: {e}\")\n", "\n", " return generation" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [], "source": [ "import difflib\n", "def print_diff(text1, text2):\n", " \"\"\"\n", " Print the differences between two strings with labels for each line.\n", " \"\"\"\n", " diff = difflib.ndiff(text1.splitlines(), text2.splitlines())\n", " for line in diff:\n", " if line.startswith('-'):\n", " label = 'LLAMA-3-8B'\n", " elif line.startswith('+'):\n", " label = 'LLAMA-3-70B'\n", " else:\n", " label = ''\n", " if label != '':\n", " print() # add a newline before the first line of a difference\n", " print(f\"{label} {line}\", end='')" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Llama 2 Chat 13B : meta.llama2-13b-chat-v1:0:4k\n", "Llama 2 Chat 13B : meta.llama2-13b-chat-v1\n", "Llama 2 Chat 70B : meta.llama2-70b-chat-v1:0:4k\n", "Llama 2 Chat 70B : meta.llama2-70b-chat-v1\n", "Llama 2 13B : meta.llama2-13b-v1:0:4k\n", "Llama 2 13B : meta.llama2-13b-v1\n", "Llama 2 70B : meta.llama2-70b-v1:0:4k\n", "Llama 2 70B : meta.llama2-70b-v1\n" ] } ], "source": [ "bedrock = create_bedrock_client(\"bedrock\")\n", "bedrock_runtime = create_bedrock_client(\"bedrock-runtime\")\n", "\n", "# Let's test that your credentials are correct by using the bedrock client to list all meta models\n", "list_all_meta_bedrock_models(bedrock)" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ ".\n", "Llamas are domesticated mammals that are native to South America. They are known for their distinctive long necks, ears, and legs, as well as their soft, woolly coats. Llamas are members of the camel family, and they are closely related to alpacas and vicuñas.\n", "\n", "Here are some interesting facts about llamas:\n", "\n", "1. Llamas are known for their intelligence and curious nature. They\n" ] }, { "data": { "text/plain": [ "'.\\nLlamas are domesticated mammals that are native to South America. They are known for their distinctive long necks, ears, and legs, as well as their soft, woolly coats. Llamas are members of the camel family, and they are closely related to alpacas and vicuñas.\\n\\nHere are some interesting facts about llamas:\\n\\n1. Llamas are known for their intelligence and curious nature. They'" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# Now we can utilize Invoke to do a simple prompt\n", "invoke_model(bedrock_runtime, 'meta.llama3-8b-instruct-v1:0', 'Tell me about llamas', 100)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prompt_1 = \"Explain black holes to 8th graders\"\n", "prompt_2 = \"Tell me about llamas\"\n", "\n", "# Let's now run the same prompt with Llama 3 8B and 70B to compare responses\n", "print(\"\\n=======LLAMA-3-8B====PROMPT 1================>\", prompt_1)\n", "response_8b_prompt1 = invoke_model(bedrock_runtime, 'meta.llama3-8b-instruct-v1:0', prompt_1, 256)\n", "print(\"\\n=======LLAMA-3-70B====PROMPT 1================>\", prompt_1)\n", "response_70b_prompt1 = invoke_model(bedrock_runtime, 'meta.llama3-70b-instruct-v1:0', prompt_1, 256)\n", "\n", "\n", "# Print the differences in responses\n", "print(\"==========================\")\n", "print(\"\\nDIFF VIEW for PROMPT 1:\")\n", "print_diff(response_8b_prompt1, response_70b_prompt1)\n", "print(\"==========================\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"\\n=======LLAMA-3-8B====PROMPT 2================>\", prompt_2)\n", "response_8b_prompt2 = invoke_model(bedrock_runtime, 'meta.llama2-13b-chat-v1', prompt_2, 128)\n", "print(\"\\n=======LLAMA-3-70B====PROMPT 2================>\", prompt_2)\n", "response_70b_prompt2 = invoke_model(bedrock_runtime, 'meta.llama2-70b-chat-v1', prompt_2, 128)\n", "\n", "# Print the differences in responses\n", "print(\"==========================\")\n", "print(\"\\nDIFF VIEW for PROMPT 2:\")\n", "print_diff(response_8b_prompt2, response_70b_prompt2)\n", "print(\"==========================\")" ] } ], "metadata": { "colab": { "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.6" } }, "nbformat": 4, "nbformat_minor": 4 }