{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6912ab05-f66a-40a9-a4a5-4deb80d2e0d9",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/meta-llama/llama-cookbook/blob/main/3p-integrations/langchain/langgraph_rag_agent.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79e9c850-d829-4266-a2b6-4e69ad24e30e",
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -U langchain_groq langchain langgraph langchain_community sentence_transformers tavily-python tiktoken langchainhub chromadb"
   ]
  },
  {
   "attachments": {
    "dccfae03-f250-494e-82d6-f229eafb0ea6.png": {
     "image/png": "[encoded data, 183672 characters]"
    }
   },
   "cell_type": "markdown",
   "id": "783f3dba-888c-4b74-a29e-b5d7a2386712",
   "metadata": {},
   "source": [
    "# LangGraph RAG agent with Llama 3\n",
    "\n",
    "Previously, we showed how to build simple agents with LangGraph and Llama 3.\n",
    "\n",
    "Now, we'll pick a more advanced use-case: advanced RAG.\n",
    "\n",
    "## Ideas\n",
    "\n",
    "We'll combine ideas from three RAG papers into a RAG agent:\n",
    "\n",
    "- **Routing:**  Adaptive RAG ([paper](https://arxiv.org/abs/2403.14403)). Route questions to different retrieval approaches\n",
    "- **Fallback:** Corrective RAG ([paper](https://arxiv.org/pdf/2401.15884.pdf)). Fallback to web search if docs are not relevant to query\n",
    "- **Self-correction:** Self-RAG ([paper](https://arxiv.org/abs/2310.11511)). Fix answers w/ hallucinations or don’t address question\n",
    "\n",
    "![Screenshot 2024-05-03 at 10.50.02 AM.png](attachment:dccfae03-f250-494e-82d6-f229eafb0ea6.png)\n",
    "\n",
    "Note that this will incorporate [a few general ideas for agents](https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance/):\n",
    "\n",
    "- **Reflection**: The self-correction mechanism is a form of reflection, where the LangGraph agent reflects on its retrieval and generations\n",
    "- **Planning**: The control flow laid out in the graph is a form of planning \n",
    "- **Tool use**: Specific nodes in the control flow (e.g., web search) will use tools\n",
    "\n",
    "## Models\n",
    "\n",
    "### LLM\n",
    "\n",
    "We can use one of the providers that (1) offer Llama 3 and (2) [provide structure outputs](https://python.langchain.com/docs/how_to/structured_output/).\n",
    "\n",
    "Here, we use [Groq](https://groq.com/).\n",
    "\n",
    "### Tracing\n",
    "\n",
    "```\n",
    "### Tracing (optional)\n",
    "os.environ['LANGCHAIN_TRACING_V2'] = 'true'\n",
    "os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'\n",
    "os.environ['LANGCHAIN_API_KEY'] = 'LANGCHAIN_API_KEY'\n",
    "```\n",
    "\n",
    "### Search\n",
    "\n",
    "Uses [Tavily](https://tavily.com/#api)m for web search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a688f62-a11e-4f44-8e66-0bf2a533dedf",
   "metadata": {},
   "outputs": [],
   "source": [
    "### LLMs\n",
    "import os\n",
    "\n",
    "os.environ['LANGCHAIN_TRACING_V2'] = 'true'\n",
    "os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'\n",
    "os.environ['LANGCHAIN_API_KEY'] = 'LANGCHAIN_API_KEY'\n",
    "\n",
    "os.environ['TAVILY_API_KEY'] = 'YOUR_TAVILY_API_KEY'\n",
    "os.environ['GROQ_API_KEY'] = 'YOUR_GROQ_API_KEY'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27a89322-0e88-4886-bcb4-3ac9bc1db316",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Build Index\n",
    "\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_community.vectorstores import Chroma\n",
    "from langchain_community.embeddings import HuggingFaceEmbeddings\n",
    "\n",
    "# Docs to index\n",
    "urls = [\n",
    "    \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
    "    \"https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/\",\n",
    "    \"https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/\",\n",
    "]\n",
    "\n",
    "# Load\n",
    "docs = [WebBaseLoader(url).load() for url in urls]\n",
    "docs_list = [item for sublist in docs for item in sublist]\n",
    "\n",
    "# Split\n",
    "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",
    "    chunk_size=500, chunk_overlap=0\n",
    ")\n",
    "doc_splits = text_splitter.split_documents(docs_list)\n",
    "\n",
    "# Add to vectorstore\n",
    "vectorstore = Chroma.from_documents(\n",
    "    documents=doc_splits,\n",
    "    collection_name=\"rag-chroma\",\n",
    "    embedding=HuggingFaceEmbeddings(),\n",
    ")\n",
    "retriever = vectorstore.as_retriever()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1041afc4-6680-4be0-950a-1c5b80b6b895",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Router\n",
    "\n",
    "from typing import Literal\n",
    "\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_core.pydantic_v1 import BaseModel, Field\n",
    "from langchain_groq import ChatGroq\n",
    "\n",
    "# Data model\n",
    "class RouteQuery(BaseModel):\n",
    "    \"\"\"Route a user query to the most relevant datasource.\"\"\"\n",
    "\n",
    "    datasource: Literal[\"vectorstore\", \"web_search\"] = Field(\n",
    "        ...,\n",
    "        description=\"Given a user question choose to route it to web search or a vectorstore.\",\n",
    "    )\n",
    "\n",
    "# LLM with function call \n",
    "llm = ChatGroq(temperature=0, model=\"llama3-70b-8192\")\n",
    "structured_llm_router = llm.with_structured_output(RouteQuery)\n",
    "\n",
    "# Prompt \n",
    "system = \"\"\"You are an expert at routing a user question to a vectorstore or web search.\n",
    "The vectorstore contains documents related to agents, prompt engineering, and adversarial attacks.\n",
    "Use the vectorstore for questions on these topics. Otherwise, use web-search.\"\"\"\n",
    "route_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"{question}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "question_router = route_prompt | structured_llm_router\n",
    "print(question_router.invoke({\"question\": \"Who will the Bears draft first in the NFL draft?\"}))\n",
    "print(question_router.invoke({\"question\": \"What are the types of agent memory?\"}))### Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bd753fb-9330-4ffa-8721-f133dc8a86aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Retrieval Grader \n",
    "\n",
    "# Data model\n",
    "class GradeDocuments(BaseModel):\n",
    "    \"\"\"Binary score for relevance check on retrieved documents.\"\"\"\n",
    "\n",
    "    score: str = Field(description=\"Documents are relevant to the question, 'yes' or 'no'\")\n",
    "\n",
    "# LLM with function call \n",
    "llm = ChatGroq(temperature=0, model=\"llama3-70b-8192\")\n",
    "structured_llm_grader = llm.with_structured_output(GradeDocuments)\n",
    "\n",
    "# Prompt \n",
    "system = \"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n \n",
    "    If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \\n\n",
    "    It does not need to be a stringent test. The goal is to filter out erroneous retrievals. \\n\n",
    "    Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\"\n",
    "grade_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"Retrieved document: \\n\\n {document} \\n\\n User question: {question}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "retrieval_grader = grade_prompt | structured_llm_grader\n",
    "question = \"agent memory\"\n",
    "docs = retriever.get_relevant_documents(question)\n",
    "doc_txt = docs[1].page_content\n",
    "print(retrieval_grader.invoke({\"question\": question, \"document\": doc_txt}))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d1fc9af-3426-491a-9d1c-3ccb3b7aba1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Generate\n",
    "\n",
    "from langchain import hub\n",
    "from langchain_core.output_parsers import StrOutputParser\n",
    "\n",
    "# Prompt\n",
    "prompt = hub.pull(\"rlm/rag-prompt\")\n",
    "\n",
    "# LLM\n",
    "llm = ChatGroq(temperature=0, model=\"llama3-70b-8192\")\n",
    "\n",
    "# Post-processing\n",
    "def format_docs(docs):\n",
    "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
    "\n",
    "# Chain\n",
    "rag_chain = prompt | llm | StrOutputParser()\n",
    "\n",
    "# Run\n",
    "generation = rag_chain.invoke({\"context\": docs, \"question\": question})\n",
    "print(generation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e522e7-4347-4b80-9972-8c9ed582995e",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Hallucination Grader \n",
    "\n",
    "# Data model\n",
    "class GradeHallucinations(BaseModel):\n",
    "    \"\"\"Binary score for hallucination present in generation answer.\"\"\"\n",
    "\n",
    "    score: str = Field(description=\"Answer is grounded in the facts, 'yes' or 'no'\")\n",
    "\n",
    "# LLM with function call \n",
    "llm = ChatGroq(temperature=0, model=\"llama3-70b-8192\")\n",
    "structured_llm_grader = llm.with_structured_output(GradeHallucinations)\n",
    "\n",
    "# Prompt \n",
    "system = \"\"\"You are a grader assessing whether an LLM generation is grounded in / supported by a set of retrieved facts. \\n \n",
    "     Give a binary score 'yes' or 'no'. 'Yes' means that the answer is grounded in / supported by the set of facts.\"\"\"\n",
    "hallucination_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"Set of facts: \\n\\n {documents} \\n\\n LLM generation: {generation}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "hallucination_grader = hallucination_prompt | structured_llm_grader\n",
    "hallucination_grader.invoke({\"documents\": docs, \"generation\": generation})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daf65df4-72a2-4805-92a4-0025cb7db5ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Answer Grader \n",
    "\n",
    "# Data model\n",
    "class GradeAnswer(BaseModel):\n",
    "    \"\"\"Binary score to assess answer addresses question.\"\"\"\n",
    "\n",
    "    score: str = Field(description=\"Answer addresses the question, 'yes' or 'no'\")\n",
    "\n",
    "# LLM with function call \n",
    "llm = ChatGroq(temperature=0, model=\"llama3-70b-8192\")\n",
    "structured_llm_grader = llm.with_structured_output(GradeAnswer)\n",
    "\n",
    "# Prompt \n",
    "system = \"\"\"You are a grader assessing whether an answer addresses / resolves a question \\n \n",
    "     Give a binary score 'yes' or 'no'. Yes' means that the answer resolves the question.\"\"\"\n",
    "answer_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", system),\n",
    "        (\"human\", \"User question: \\n\\n {question} \\n\\n LLM generation: {generation}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "answer_grader = answer_prompt | structured_llm_grader\n",
    "answer_grader.invoke({\"question\": question,\"generation\": generation})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06a0af45-8b57-4ac6-a034-b368309f02cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Search\n",
    "\n",
    "from langchain_community.tools.tavily_search import TavilySearchResults\n",
    "web_search_tool = TavilySearchResults(k=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b1c0474-4cd1-4802-b49b-dcbb70842411",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing_extensions import TypedDict\n",
    "from typing import List\n",
    "\n",
    "### State\n",
    "\n",
    "class GraphState(TypedDict):\n",
    "    \"\"\"\n",
    "    Represents the state of our graph.\n",
    "\n",
    "    Attributes:\n",
    "        question: question\n",
    "        generation: LLM generation\n",
    "        web_search: whether to add search\n",
    "        documents: list of documents \n",
    "    \"\"\"\n",
    "    question : str\n",
    "    generation : str\n",
    "    web_search : str\n",
    "    documents : List[str]\n",
    "\n",
    "from langchain.schema import Document\n",
    "\n",
    "### Nodes\n",
    "\n",
    "def retrieve(state):\n",
    "    \"\"\"\n",
    "    Retrieve documents from vectorstore\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): New key added to state, documents, that contains retrieved documents\n",
    "    \"\"\"\n",
    "    print(\"---RETRIEVE---\")\n",
    "    question = state[\"question\"]\n",
    "\n",
    "    # Retrieval\n",
    "    documents = retriever.invoke(question)\n",
    "    return {\"documents\": documents, \"question\": question}\n",
    "\n",
    "def generate(state):\n",
    "    \"\"\"\n",
    "    Generate answer using RAG on retrieved documents\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): New key added to state, generation, that contains LLM generation\n",
    "    \"\"\"\n",
    "    print(\"---GENERATE---\")\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "    \n",
    "    # RAG generation\n",
    "    generation = rag_chain.invoke({\"context\": documents, \"question\": question})\n",
    "    return {\"documents\": documents, \"question\": question, \"generation\": generation}\n",
    "\n",
    "def grade_documents(state):\n",
    "    \"\"\"\n",
    "    Determines whether the retrieved documents are relevant to the question\n",
    "    If any document is not relevant, we will set a flag to run web search\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): Filtered out irrelevant documents and updated web_search state\n",
    "    \"\"\"\n",
    "\n",
    "    print(\"---CHECK DOCUMENT RELEVANCE TO QUESTION---\")\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "    \n",
    "    # Score each doc\n",
    "    filtered_docs = []\n",
    "    web_search = \"No\"\n",
    "    for d in documents:\n",
    "        score = retrieval_grader.invoke({\"question\": question, \"document\": d.page_content})\n",
    "        grade = score.score\n",
    "        # Document relevant\n",
    "        if grade.lower() == \"yes\":\n",
    "            print(\"---GRADE: DOCUMENT RELEVANT---\")\n",
    "            filtered_docs.append(d)\n",
    "        # Document not relevant\n",
    "        else:\n",
    "            print(\"---GRADE: DOCUMENT NOT RELEVANT---\")\n",
    "            # We do not include the document in filtered_docs\n",
    "            # We set a flag to indicate that we want to run web search\n",
    "            web_search = \"Yes\"\n",
    "            continue\n",
    "    return {\"documents\": filtered_docs, \"question\": question, \"web_search\": web_search}\n",
    "    \n",
    "def web_search(state):\n",
    "    \"\"\"\n",
    "    Web search based based on the question\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        state (dict): Appended web results to documents\n",
    "    \"\"\"\n",
    "\n",
    "    print(\"---WEB SEARCH---\")\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "\n",
    "    # Web search\n",
    "    docs = web_search_tool.invoke({\"query\": question})\n",
    "    web_results = \"\\n\".join([d[\"content\"] for d in docs])\n",
    "    web_results = Document(page_content=web_results)\n",
    "    if documents is not None:\n",
    "        documents.append(web_results)\n",
    "    else:\n",
    "        documents = [web_results]\n",
    "    return {\"documents\": documents, \"question\": question}\n",
    "\n",
    "### Conditional edge\n",
    "\n",
    "def route_question(state):\n",
    "    \"\"\"\n",
    "    Route question to web search or RAG.\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        str: Next node to call\n",
    "    \"\"\"\n",
    "\n",
    "    print(\"---ROUTE QUESTION---\")\n",
    "    question = state[\"question\"]\n",
    "    source = question_router.invoke({\"question\": question})  \n",
    "    if source.datasource == 'web_search':\n",
    "        print(\"---ROUTE QUESTION TO WEB SEARCH---\")\n",
    "        return \"websearch\"\n",
    "    elif source.datasource == 'vectorstore':\n",
    "        print(\"---ROUTE QUESTION TO RAG---\")\n",
    "        return \"vectorstore\"\n",
    "\n",
    "def decide_to_generate(state):\n",
    "    \"\"\"\n",
    "    Determines whether to generate an answer, or add web search\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        str: Binary decision for next node to call\n",
    "    \"\"\"\n",
    "\n",
    "    print(\"---ASSESS GRADED DOCUMENTS---\")\n",
    "    question = state[\"question\"]\n",
    "    web_search = state[\"web_search\"]\n",
    "    filtered_documents = state[\"documents\"]\n",
    "\n",
    "    if web_search == \"Yes\":\n",
    "        # All documents have been filtered check_relevance\n",
    "        # We will re-generate a new query\n",
    "        print(\"---DECISION: ALL DOCUMENTS ARE NOT RELEVANT TO QUESTION, INCLUDE WEB SEARCH---\")\n",
    "        return \"websearch\"\n",
    "    else:\n",
    "        # We have relevant documents, so generate answer\n",
    "        print(\"---DECISION: GENERATE---\")\n",
    "        return \"generate\"\n",
    "\n",
    "### Conditional edge\n",
    "\n",
    "def grade_generation_v_documents_and_question(state):\n",
    "    \"\"\"\n",
    "    Determines whether the generation is grounded in the document and answers question.\n",
    "\n",
    "    Args:\n",
    "        state (dict): The current graph state\n",
    "\n",
    "    Returns:\n",
    "        str: Decision for next node to call\n",
    "    \"\"\"\n",
    "\n",
    "    print(\"---CHECK HALLUCINATIONS---\")\n",
    "    question = state[\"question\"]\n",
    "    documents = state[\"documents\"]\n",
    "    generation = state[\"generation\"]\n",
    "\n",
    "    score = hallucination_grader.invoke({\"documents\": documents, \"generation\": generation})\n",
    "    grade = score.score\n",
    "\n",
    "    # Check hallucination\n",
    "    if grade == \"yes\":\n",
    "        print(\"---DECISION: GENERATION IS GROUNDED IN DOCUMENTS---\")\n",
    "        # Check question-answering\n",
    "        print(\"---GRADE GENERATION vs QUESTION---\")\n",
    "        score = answer_grader.invoke({\"question\": question,\"generation\": generation})\n",
    "        grade = score.score\n",
    "        if grade == \"yes\":\n",
    "            print(\"---DECISION: GENERATION ADDRESSES QUESTION---\")\n",
    "            return \"useful\"\n",
    "        else:\n",
    "            print(\"---DECISION: GENERATION DOES NOT ADDRESS QUESTION---\")\n",
    "            return \"not useful\"\n",
    "    else:\n",
    "        pprint(\"---DECISION: GENERATION IS NOT GROUNDED IN DOCUMENTS, RE-TRY---\")\n",
    "        return \"not supported\"\n",
    "\n",
    "from langgraph.graph import END, StateGraph\n",
    "workflow = StateGraph(GraphState)\n",
    "\n",
    "# Define the nodes\n",
    "workflow.add_node(\"websearch\", web_search) # web search\n",
    "workflow.add_node(\"retrieve\", retrieve) # retrieve\n",
    "workflow.add_node(\"grade_documents\", grade_documents) # grade documents\n",
    "workflow.add_node(\"generate\", generate) # generatae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26a0ad7a-87d3-4e32-9608-ab9e4408ad76",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build graph\n",
    "workflow.set_conditional_entry_point(\n",
    "    route_question,\n",
    "    {\n",
    "        \"websearch\": \"websearch\",\n",
    "        \"vectorstore\": \"retrieve\",\n",
    "    },\n",
    ")\n",
    "\n",
    "workflow.add_edge(\"retrieve\", \"grade_documents\")\n",
    "workflow.add_conditional_edges(\n",
    "    \"grade_documents\",\n",
    "    decide_to_generate,\n",
    "    {\n",
    "        \"websearch\": \"websearch\",\n",
    "        \"generate\": \"generate\",\n",
    "    },\n",
    ")\n",
    "workflow.add_edge(\"websearch\", \"generate\")\n",
    "workflow.add_conditional_edges(\n",
    "    \"generate\",\n",
    "    grade_generation_v_documents_and_question,\n",
    "    {\n",
    "        \"not supported\": \"generate\",\n",
    "        \"useful\": END,\n",
    "        \"not useful\": \"websearch\",\n",
    "    },\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a2af1cb-4bba-4baf-9345-d21ce0e65503",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compile\n",
    "app = workflow.compile()\n",
    "\n",
    "# Test\n",
    "from pprint import pprint\n",
    "inputs = {\"question\": \"What are the types of agent memory?\"}\n",
    "for output in app.stream(inputs):\n",
    "    for key, value in output.items():\n",
    "        pprint(f\"Finished running: {key}:\")\n",
    "pprint(value[\"generation\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4411adb6-a98d-41b4-ac00-5f643e008389",
   "metadata": {},
   "source": [
    "Trace: \n",
    "\n",
    "https://smith.langchain.com/public/2babc6ec-a243-40d0-844b-5e6b40f70fc9/r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0642aaaa-4657-45df-92f2-6a279d696497",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
