add integration for llama-prompt-ops

getting-started/automatic-prompt-optimization/getting_started.ipynb

@@ -0,0 +1,527 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Getting Started with [llama-prompt-ops](https://github.com/meta-llama/llama-prompt-ops)\n",
+    "\n",
+    "This notebook will guide you through the process of using [llama-prompt-ops](https://github.com/meta-llama/llama-prompt-ops) to optimize your prompts for Llama models. We'll cover:\n",
+    "\n",
+    "1. Introduction to llama-prompt-ops\n",
+    "2. Setting up your environment\n",
+    "3. Creating a sample project\n",
+    "4. Running prompt optimization\n",
+    "5. Analyzing the results\n",
+    "6. Advanced usage and customization"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Introduction to llama-prompt-ops\n",
+    "\n",
+    "### What is llama-prompt-ops?\n",
+    "\n",
+    "llama-prompt-ops is a Python package that **automatically optimizes prompts** for Llama models. It transforms prompts that work well with other LLMs into prompts that are optimized for Llama models, improving performance and reliability.\n",
+    "\n",
+    "### How It Works\n",
+    "\n",
+    "llama-prompt-ops takes three key inputs:\n",
+    "1. Your existing system prompt\n",
+    "2. A dataset of query-response pairs for evaluation and optimization\n",
+    "3. A configuration file specifying model parameters and optimization details\n",
+    "\n",
+    "It then applies optimization techniques to transform your prompt into one that works better with Llama models, and provides metrics to measure the improvement."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Setting up your environment\n",
+    "\n",
+    "Let's start by installing the Llama Prompt Ops package and setting up our environment. You can install it either from PyPI or directly from the source code."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Install from PyPI\n",
+    "!pip install llama-prompt-ops"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Setting up your API key\n",
+    "\n",
+    "Llama Prompt Ops requires an API key to access LLM services. You can use OpenRouter, which provides access to various models including Llama models.\n",
+    "\n",
+    "Create a `.env` file in your project directory with your API key:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a .env file with your API key\n",
+    "%%writefile .env\n",
+    "OPENROUTER_API_KEY=your_key_here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You'll need to replace `your_key_here` with your actual OpenRouter API key. You can get an OpenRouter API key by creating an account at [OpenRouter](https://openrouter.ai/).\n",
+    "\n",
+    "Let's load the environment variables from the .env file:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load environment variables from .env file\n",
+    "import os\n",
+    "from dotenv import load_dotenv\n",
+    "\n",
+    "load_dotenv()\n",
+    "\n",
+    "# Verify the API key is loaded (will show as masked)\n",
+    "api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
+    "if api_key:\n",
+    "    print(f\"API key loaded: {'*' * len(api_key)}\")\n",
+    "else:\n",
+    "    print(\"API key not found. Please check your .env file.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Creating a Sample Project\n",
+    "\n",
+    "Llama Prompt Ops provides a convenient way to create a sample project with all the necessary files. Let's create a sample project to get started."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Create a sample project\n",
+    "!llama-prompt-ops create my-notebook-project"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This command creates a directory called `my-notebook-project` with a sample configuration and dataset. Let's explore the files that were created:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!ls -la my-notebook-project"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The sample project includes:\n",
+    "- `.env`: A file for your API key\n",
+    "- `README.md`: Documentation for the project\n",
+    "- `config.yaml`: Configuration file for prompt optimization\n",
+    "- `data/dataset.json`: Sample dataset for evaluation and optimization\n",
+    "- `prompts/prompt.txt`: Sample system prompt to optimize\n",
+    "\n",
+    "Let's examine the configuration file:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cat my-notebook-project/config.yaml"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The configuration file specifies:\n",
+    "- The system prompt to optimize\n",
+    "- The dataset to use for evaluation and optimization\n",
+    "- The model to use for optimization and evaluation\n",
+    "- The metric to use for evaluation\n",
+    "- The optimization strategy to use\n",
+    "\n",
+    "Let's also look at the sample prompt and dataset:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!cat my-notebook-project/prompts/prompt.txt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!head -n 20 my-notebook-project/data/dataset.json"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Running Prompt Optimization\n",
+    "\n",
+    "Now that we have our sample project set up, let's run the prompt optimization process. We'll use the `migrate` command, which takes a configuration file as input and outputs an optimized prompt."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Run prompt optimization\n",
+    "!cd my-notebook-project && llama-prompt-ops migrate"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The optimization process will take a few minutes to complete. It involves:\n",
+    "1. Loading your system prompt and dataset\n",
+    "2. Analyzing the prompt structure and content\n",
+    "3. Applying optimization techniques specific to Llama models\n",
+    "4. Evaluating the optimized prompt against the original prompt\n",
+    "5. Saving the optimized prompt to the `results/` directory\n",
+    "\n",
+    "Let's check the results directory to see the optimized prompt. If the optimizer successfully found a better prompt, it will be saved in the `results/` directory. You may need to run the optimization process again with different parameters or a larger dataset if the prompt is the same as the original."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!ls -la my-notebook-project/results/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The optimized prompt is saved as a YAML file with a timestamp. Let's examine the contents of the optimized prompt:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import glob\n",
+    "import yaml\n",
+    "\n",
+    "# Find the most recent result file\n",
+    "result_files = glob.glob('my-notebook-project/results/*.yaml')\n",
+    "if result_files:\n",
+    "    latest_result = max(result_files, key=os.path.getctime)\n",
+    "    print(f\"Latest result file: {latest_result}\")\n",
+    "    \n",
+    "    # Load and display the optimized prompt\n",
+    "    with open(latest_result, 'r') as f:\n",
+    "        result = yaml.safe_load(f)\n",
+    "        print(\"\\nOptimized System Prompt:\")\n",
+    "        print(result.get('system', 'No system prompt found'))\n",
+    "else:\n",
+    "    print(\"No result files found. Make sure the optimization process completed successfully.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Analyzing the Results\n",
+    "\n",
+    "Let's compare the original prompt with the optimized prompt to understand the changes made during optimization."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Load the original prompt\n",
+    "with open('my-notebook-project/prompts/prompt.txt', 'r') as f:\n",
+    "    original_prompt = f.read()\n",
+    "\n",
+    "# Find the most recent result file again\n",
+    "result_files = glob.glob('my-notebook-project/results/*.yaml')\n",
+    "if result_files:\n",
+    "    latest_result = max(result_files, key=os.path.getctime)\n",
+    "    \n",
+    "    # Load the optimized prompt\n",
+    "    with open(latest_result, 'r') as f:\n",
+    "        result = yaml.safe_load(f)\n",
+    "        optimized_prompt = result.get('system', 'No system prompt found')\n",
+    "    \n",
+    "    # Print the comparison\n",
+    "    print(\"Original Prompt:\")\n",
+    "    print(\"-\" * 80)\n",
+    "    print(original_prompt)\n",
+    "    print(\"\\n\" + \"-\" * 80)\n",
+    "    print(\"\\nOptimized Prompt:\")\n",
+    "    print(\"-\" * 80)\n",
+    "    print(optimized_prompt)\n",
+    "    print(\"-\" * 80)\n",
+    "else:\n",
+    "    print(\"No result files found. Make sure the optimization process completed successfully.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Key Differences in the Optimized Prompt\n",
+    "\n",
+    "The optimized prompt typically includes several improvements:\n",
+    "\n",
+    "1. **Better Structure**: Llama models respond better to clear, structured instructions\n",
+    "2. **Llama-Specific Formatting**: Formatting that works better with Llama's training patterns\n",
+    "3. **Few-Shot Examples**: Examples that help the model understand the expected output format\n",
+    "4. **Clear Output Expectations**: More explicit instructions about what the output should look like\n",
+    "\n",
+    "These changes can significantly improve the model's performance on your specific task."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Advanced Usage and Customization\n",
+    "\n",
+    "### Using Your Own Data\n",
+    "\n",
+    "To use your own data with Llama Prompt Ops, you'll need to:\n",
+    "\n",
+    "1. Prepare your dataset in JSON format\n",
+    "2. Create a system prompt file\n",
+    "3. Create a configuration file\n",
+    "\n",
+    "Check out the comprehensive guide [here](https://github.com/meta-llama/llama-prompt-ops/tree/main/docs) to learn more.\n",
+    "\n",
+    "Now, let's see how to create a custom configuration file:\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Overwriting custom_config.yaml\n"
+     ]
+    }
+   ],
+   "source": [
+    "%%writefile custom_config.yaml\n",
+    "system_prompt:\n",
+    "  file: \"path/to/your/prompt.txt\"\n",
+    "  inputs: [\"question\"]\n",
+    "  outputs: [\"answer\"]\n",
+    "\n",
+    "# Dataset configuration\n",
+    "dataset:\n",
+    "  path: \"path/to/your/dataset.json\"\n",
+    "  input_field: \"question\"  # or [\"fields\", \"input\"] for nested fields\n",
+    "  golden_output_field: \"answer\"\n",
+    "\n",
+    "# Model configuration\n",
+    "model:\n",
+    "  name: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "  task_model: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "  proposer_model: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "\n",
+    "# Metric configuration\n",
+    "metric:\n",
+    "  class: \"llama_prompt_ops.core.metrics.StandardJSONMetric\"\n",
+    "  strict_json: false\n",
+    "  output_field: \"answer\"\n",
+    "\n",
+    "# Optimization settings\n",
+    "optimization:\n",
+    "  strategy: \"llama\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Using Different Metrics\n",
+    "\n",
+    "Llama Prompt Ops supports different metrics for evaluating prompt performance. The default is `StandardJSONMetric`, but you can use other metrics like `FacilityMetric` for specific use cases.\n",
+    "\n",
+    "Here's an example of using the `FacilityMetric` for the facility support analyzer use case:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%writefile facility_config.yaml\n",
+    "system_prompt:\n",
+    "  file: \"prompts/facility_prompt.txt\"\n",
+    "  inputs: [\"question\"]\n",
+    "  outputs: [\"answer\"]\n",
+    "\n",
+    "# Dataset configuration\n",
+    "dataset:\n",
+    "  path: \"data/facility_dataset.json\"\n",
+    "  input_field: [\"fields\", \"input\"]\n",
+    "  golden_output_field: \"answer\"\n",
+    "\n",
+    "# Model configuration\n",
+    "model:\n",
+    "  name: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "  task_model: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "  proposer_model: \"openrouter/meta-llama/llama-3.3-70b-instruct\"\n",
+    "\n",
+    "# Metric configuration\n",
+    "metric:\n",
+    "  class: \"llama_prompt_ops.core.metrics.FacilityMetric\"\n",
+    "  strict_json: false\n",
+    "  output_field: \"answer\"\n",
+    "\n",
+    "# Optimization settings\n",
+    "optimization:\n",
+    "  strategy: \"llama\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Using Different Models\n",
+    "\n",
+    "Llama Prompt Ops supports different models through various inference providers. You can use OpenRouter, vLLM, or NVIDIA NIMs depending on your infrastructure needs.\n",
+    "\n",
+    "Here's an example of using a different model through OpenRouter:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%writefile different_model_config.yaml\n",
+    "system_prompt:\n",
+    "  file: \"prompts/prompt.txt\"\n",
+    "  inputs: [\"question\"]\n",
+    "  outputs: [\"answer\"]\n",
+    "\n",
+    "# Dataset configuration\n",
+    "dataset:\n",
+    "  path: \"data/dataset.json\"\n",
+    "  input_field: \"question\"\n",
+    "  golden_output_field: \"answer\"\n",
+    "\n",
+    "# Model configuration\n",
+    "model:\n",
+    "  name: \"openrouter/meta-llama/llama-3.1-8b-instruct\"\n",
+    "  task_model: \"openrouter/meta-llama/llama-3.1-8b-instruct\"\n",
+    "  proposer_model: \"openrouter/meta-llama/llama-3.1-8b-instruct\"\n",
+    "\n",
+    "# Metric configuration\n",
+    "metric:\n",
+    "  class: \"llama_prompt_ops.core.metrics.StandardJSONMetric\"\n",
+    "  strict_json: false\n",
+    "  output_field: \"answer\"\n",
+    "\n",
+    "# Optimization settings\n",
+    "optimization:\n",
+    "  strategy: \"llama\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conclusion\n",
+    "\n",
+    "In this notebook, we've covered:\n",
+    "\n",
+    "1. Introduction to Llama Prompt Ops and its benefits\n",
+    "2. Creating a sample project\n",
+    "3. Setting up your environment and API key\n",
+    "4. Running prompt optimization\n",
+    "5. Analyzing the results\n",
+    "6. Advanced usage and customization options\n",
+    "\n",
+    "Llama Prompt Ops provides a powerful way to optimize your prompts for Llama models, improving performance and reliability. By following the steps in this notebook, you can start optimizing your own prompts and building more effective LLM applications.\n",
+    "\n",
+    "For more information, check out the [llama-prompt-ops documentation](https://github.com/meta-llama/llama-prompt-ops/tree/main/docs) and explore the example use cases in the repository."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}