{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Automatic Issues Triaging with Llama\n", "\n", "We utilize an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository. \n", "\n", "This notebook walks you through the tool's working. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!git clone https://github.com/meta-llama/llama-recipes\n", "\n", "%cd recipes/use_cases/github_triage\n", "\n", "!pip install -r requirements.txt" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "import yaml\n", "import os\n", "\n", "from llm import run_llm_inference\n", "from utils import fetch_repo_issues, validate_df_values\n", "from plots import draw_all_plots\n", "from pdf_report import create_report_pdf\n", "from triage import generate_executive_reports, generate_issue_annotations" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set access keys and tokens\n", "\n", "Set your GitHub token for API calls. Some privileged information may not be available if you don't have push-access to the target repository.\n", "\n", "Set your groq token for inference. Get one at https://console.groq.com/keys" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "github_token = input(\"Enter your Github API token\")\n", "groq_token = input(\"Enter your Groq token\")\n", "\n", "with open(\"config.yaml\", \"r\") as f:\n", " CFG = yaml.safe_load(f)\n", "CFG['github_token'] = github_token\n", "CFG['model']['groq']['key'] = groq_token\n", "with open(\"config.yaml\", \"w\") as f:\n", " yaml.dump(CFG, f)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Set target repo and period to analyze" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Running AutoTriage for pytorch/pytorch from 2024-08-28 to 2024-08-28\n" ] } ], "source": [ "repo_name = input(\"What repository do you want to analyze? (eg: meta-llama/llama-recipes)\")\n", "start_date = input(\"Start analysis (eg: 2024-08-23): \")\n", "end_date = input(\"End analysis (eg: 2024-08-30): \")\n", "\n", "out_folder = f'output/{repo_name}/{start_date}_{end_date}'\n", "os.makedirs(out_folder, exist_ok=True)\n", "\n", "print(f\"Running AutoTriage for {repo_name} from {start_date} to {end_date}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Fetch issues from the repository\n", "\n", "Use the github API to retrieve issues (including the full discussion on them) and store it in a dataframe." ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Fetching issues on pytorch/pytorch from 2024-08-28 to 2024-08-28\n", "Fetched 26 issues on pytorch/pytorch from 2024-08-28 to 2024-08-28\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[Showing 5 out of 26 rows]\n", "\n", " repo_name number html_url \\\n", "0 pytorch/pytorch 134637 https://github.com/pytorch/pytorch/issues/134637 \n", "1 pytorch/pytorch 134640 https://github.com/pytorch/pytorch/issues/134640 \n", "2 pytorch/pytorch 134641 https://github.com/pytorch/pytorch/issues/134641 \n", "3 pytorch/pytorch 134642 https://github.com/pytorch/pytorch/issues/134642 \n", "4 pytorch/pytorch 134644 https://github.com/pytorch/pytorch/issues/134644 \n", "\n", " closed num_comments created_at \\\n", "0 False 0 2024-08-28T01:14:14Z \n", "1 False 0 2024-08-28T01:58:33Z \n", "2 False 1 2024-08-28T02:00:59Z \n", "3 False 4 2024-08-28T02:15:07Z \n", "4 False 0 2024-08-28T02:25:25Z \n", "\n", " discussion \n", "0 LSTM inference c++ threads block on DropoutSta... \n", "1 Meet “No module named'tools. setup_helpers‘“ w... \n", "2 nn.Module.to(memory_format= channels_last form... \n", "3 Memory leak starting with torch==2.5.0dev20240... \n", "4 Deduce Tangents Stride For Channels Last Tenso... \n" ] } ], "source": [ "issues_df = fetch_repo_issues(repo_name, start_date, end_date)\n", "\n", "print(f\"[Showing 5 out of {issues_df.shape[0]} rows]\\n\")\n", "print(issues_df.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Use Llama to generate the annotations for this data\n", "\n", "We use the `parse_issues` prompt defined in `config.yaml` to generate annotations and other metadata for each issue thread.\n", "\n", "`assign_category` tags each issue with the most relevant category defined in the prompt." ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prompt for generating annotations:\n", "{'system': 'You are an expert maintainer of an open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can.', 'json_schema': '{ \"type\": \"object\", \"properties\": { \"summary\": { \"description\": \"Summary of the issue and discussion along with any details about platform or tooling.\", \"type\": \"string\" }, \"possible_causes\": { \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"remediations\": { \"description\": \"How we can improve the code or documentation to prevent this issue.\", \"type\": \"array\", \"maxItems\": 2, \"items\": { \"type\": \"string\" } }, \"component\": { \"description\": \"The specific module or component affected by the issue\", \"type\": \"string\" }, \"sentiment\": { \"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"] }, \"issue_type\": { \"description\": \"Any issue not related to LLMs, Llama or code in this repository should be marked as \\\\\"invalid\\\\\"\", \"type\": \"string\", \"enum\": [\"bug_report\", \"feature_request\", \"documentation\", \"installation\", \"discussion\", \"invalid\"] }, \"severity\": { \"type\": \"string\", \"enum\": [\"critical\", \"major\", \"minor\", \"trivial\"] }, \"op_expertise\": { \"description\": \"Assess the reporters level of expertise.\", \"type\": \"string\", \"enum\": [\"beginner\", \"intermediate\", \"advanced\"] } }, \"required\": [\"summary\", \"possible_causes\", \"remediations\", \"component\", \"sentiment\", \"issue_type\", \"severity\", \"op_expertise\"] }'}\n", "\n", "Prompt for categorizing issues:\n", "{'system': 'You are the lead maintainer of an open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme.', 'json_schema': '{ \"type\": \"object\", \"properties\": { \"report\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"properties\": { \"theme\": { \"description\": \"key theme identified from the issues\", \"type\": \"string\", \"enum\": [\"Cloud Compute\", \"Installation and Environment\", \"Model Loading\", \"Model Fine-tuning and Training\", \"Model Conversion\", \"Model Inference\", \"Distributed Training and Multi-GPU\", \"Performance and Optimization\", \"Quantization and Mixed Precision\", \"Documentation\", \"CUDA Compatibility\", \"Model Evaluation and Benchmarking\", \"Miscellaneous\", \"Invalid\"] }, \"description\": { \"type\": \"string\" }, \"related_issues\": { \"description\": \"Issue numbers related to this theme\", \"type\": \"array\", \"items\": { \"type\": \"number\" } } }, \"required\": [\"theme\", \"description\", \"related_issues\"] } } }, \"required\": [\"report\"] }'}\n", "\n", "Issues can be categorized into: ['Cloud Compute', 'Installation and Environment', 'Model Loading', 'Model Fine-tuning and Training', 'Model Conversion', 'Model Inference', 'Distributed Training and Multi-GPU', 'Performance and Optimization', 'Quantization and Mixed Precision', 'Documentation', 'CUDA Compatibility', 'Model Evaluation and Benchmarking', 'Miscellaneous', 'Invalid']\n" ] } ], "source": [ "print(f\"Prompt for generating annotations:\\n{CFG['prompts']['parse_issue']}\\n\")\n", "print(f\"Prompt for categorizing issues:\\n{CFG['prompts']['assign_category']}\\n\")\n", "print(f\"Issues can be categorized into: {eval(CFG['prompts']['assign_category']['json_schema'])['properties']['report']['items']['properties']['theme']['enum']}\")" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/annotated_issues.csv\n" ] } ], "source": [ "# Generate annotations and metadata\n", "annotated_issues, theme_counts = generate_issue_annotations(issues_df)\n", "# Validate and save generated data\n", "annotated_issues = validate_df_values(annotated_issues, out_folder, 'annotated_issues')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The generated annotations include a summary and category (theme, component affected) for each issue, that can help triagers quickly identify the problem and the right POC to take a look." ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "metadata generated by LLM: {'sentiment', 'themes', 'summary', 'remediations', 'component', 'possible_causes', 'severity', 'op_expertise', 'issue_type'}\n", "\n", "\n", "[Showing 5 out of 26 rows]\n", "\n", " repo_name number html_url \\\n", "0 pytorch/pytorch 134637 https://github.com/pytorch/pytorch/issues/134637 \n", "1 pytorch/pytorch 134640 https://github.com/pytorch/pytorch/issues/134640 \n", "2 pytorch/pytorch 134641 https://github.com/pytorch/pytorch/issues/134641 \n", "3 pytorch/pytorch 134642 https://github.com/pytorch/pytorch/issues/134642 \n", "4 pytorch/pytorch 134644 https://github.com/pytorch/pytorch/issues/134644 \n", "\n", " closed num_comments created_at \\\n", "0 False 0 2024-08-28T01:14:14Z \n", "1 False 0 2024-08-28T01:58:33Z \n", "2 False 1 2024-08-28T02:00:59Z \n", "3 False 4 2024-08-28T02:15:07Z \n", "4 False 0 2024-08-28T02:25:25Z \n", "\n", " discussion \\\n", "0 LSTM inference c++ threads block on DropoutSta... \n", "1 Meet “No module named'tools. setup_helpers‘“ w... \n", "2 nn.Module.to(memory_format= channels_last form... \n", "3 Memory leak starting with torch==2.5.0dev20240... \n", "4 Deduce Tangents Stride For Channels Last Tenso... \n", "\n", " summary \\\n", "0 Threads block on DropoutState in LSTM inferenc... \n", "1 The user encountered an error when installing ... \n", "2 nn.Module.to(memory_format=channels_last forma... \n", "3 A memory leak has been reported with torch ver... \n", "4 The discussion is about improving the performa... \n", "\n", " possible_causes \\\n", "0 [Mutex lock on DropoutState in get_dropout_sta... \n", "1 [Incompatible version of caffe2 with the curre... \n", "2 [The convert function in nn.Module does not ha... \n", "3 [Activations not being freed when gradient che... \n", "4 [AOTAutograd forcing tangents to be contiguous... \n", "\n", " remediations component sentiment \\\n", "0 [Remove or bypass the mutex lock on DropoutSta... RNN negative \n", "1 [Try installing an older version of caffe2 tha... caffe2 negative \n", "2 [Modify the convert function in nn.Module to h... nn.Module negative \n", "3 [Disable gradient checkpointing to prevent mem... torchtune negative \n", "4 [Guess the stride using the forward-propagatio... torch.compile neutral \n", "\n", " issue_type severity op_expertise \\\n", "0 bug_report major advanced \n", "1 bug_report major intermediate \n", "2 bug_report major advanced \n", "3 bug_report critical advanced \n", "4 feature_request major advanced \n", "\n", " themes \n", "0 [Model Loading, Model Inference] \n", "1 [Miscellaneous] \n", "2 [Miscellaneous] \n", "3 [Model Loading, Model Fine-tuning and Training] \n", "4 [Model Fine-tuning and Training, Model Inferen... \n" ] } ], "source": [ "print(f\"metadata generated by LLM: {set(annotated_issues.columns).difference(set(issues_df.columns))}\\n\\n\")\n", "print(f\"[Showing 5 out of {annotated_issues.shape[0]} rows]\\n\")\n", "print(annotated_issues.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Use Llama to generate high-level insights\n", "\n", "The above data is good for OSS maintainers and developers to quickly address any issues. The next section will synthesize this data into high-level insights that can enable decisioning about management of this repository." ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Prompt for generating high-level overview:\n", "\n", "{'system': 'You are not only an experienced Open Source maintainer, but also an expert at paraphrasing raw data into clear succinct reports. Draft a concise report about the issues in this open source repository. Include an executive summary that provides an overview of the challenges faced, any open questions or decisions to be made, or actions that we can take. Group issues together if they ladder up to the same overall challenge, summarize the challenges and include any actionable resolutions we can take (more information in the \\\\\"remediations\\\\\" sections). Use your experience and judgement to ignore issues that are clearly unrelated to the open source project. Ensure the output is in JSON.', 'json_schema': '{ \"type\": \"object\", \"properties\": { \"executive_summary\": { \"description\": \"An executive summary of the analysis\", \"type\": \"string\" }, \"open_questions\": { \"description\": \"Any open questions or decisions that the product team needs to make in light of these issues\", \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"issue_analysis\": { \"type\": \"array\", \"items\": { \"type\": \"object\", \"properties\": { \"key_challenge\": { \"description\": \"A description of the challenge reported in these issues\", \"type\": \"string\" }, \"affected_issues\": { \"description\": \"A list of issues that are related to this challenge\", \"type\": \"array\", \"items\": { \"type\": \"number\" } }, \"possible_causes\": { \"description\": \"A list of possible causes or reasons for this challenge to occur\", \"type\": \"array\", \"items\": { \"type\": \"string\" } }, \"remediations\": { \"description\": \"Steps we can take to address this challenge\", \"type\": \"array\", \"items\": { \"type\": \"string\" } } }, \"required\": [\"key_challenge\", \"affected_issues\", \"possible_causes\", \"remediations\"] } } }, \"required\": [\"issue_analysis\", \"open_questions\", \"actions\", \"executive_summary\"] }'}\n" ] } ], "source": [ "print(f\"Prompt for generating high-level overview:\\n\\n{CFG['prompts']['get_overview']}\")" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Generating high-level summaries from annotations...\n", "Identifying key-challenges faced by users...\n", "Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/challenges.csv\n", "Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/overview.csv\n" ] } ], "source": [ "# Generate high-level analysis\n", "challenges, overview = generate_executive_reports(annotated_issues, theme_counts, repo_name, start_date, end_date)\n", "# Validate and save generated data\n", "challenges = validate_df_values(challenges, out_folder, 'challenges')\n", "overview = validate_df_values(overview, out_folder, 'overview')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We identify key areas that users are challenged by and tag which issues represent those challenges." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Showing 5 out of 5 rows]\n", "\n", " repo_name start_date end_date key_challenge \\\n", "0 pytorch/pytorch 2024-08-28 2024-08-28 Performance Regressions \n", "1 pytorch/pytorch 2024-08-28 2024-08-28 Compatibility Issues \n", "2 pytorch/pytorch 2024-08-28 2024-08-28 Security Vulnerabilities \n", "3 pytorch/pytorch 2024-08-28 2024-08-28 Tensor Parallelism and Autograd \n", "4 pytorch/pytorch 2024-08-28 2024-08-28 CUDA Support \n", "\n", " affected_issues \\\n", "0 [134679, 134686] \n", "1 [134640, 134682, 134684] \n", "2 [134664] \n", "3 [134668, 134676] \n", "4 [134682, 134684] \n", "\n", " possible_causes \\\n", "0 [Changes in the torch or torchvision libraries... \n", "1 [Incompatible version of caffe2 with the curre... \n", "2 [Using the affected version of protobuf (3.20.... \n", "3 [Lack of understanding of tensor parallelism i... \n", "4 [cuDNN version incompatibility between PyTorch... \n", "\n", " remediations \n", "0 [Investigate the suspected guilty commit and r... \n", "1 [Try installing an older version of caffe2 tha... \n", "2 [Update protobuf to a version mentioned in the... \n", "3 [Improve documentation on tensor parallelism a... \n", "4 [Ensure PyTorch can find the bundled cuDNN by ... \n" ] } ], "source": [ "print(f\"[Showing 5 out of {challenges.shape[0]} rows]\\n\")\n", "print(challenges.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As the name suggests, the `overview` dataframe contains columns that provide information about the overall activity in the repository during this period, including:\n", "* an executive summary of all the issues seen during this period\n", "* how many issues were created, discussed and closed\n", "* what are some open questions that the maintainers should address\n", "* how many issues were seen for each theme etc." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[Showing 5 out of 1 rows]\n", "\n", " repo_name start_date end_date issues_created open_discussion \\\n", "0 pytorch/pytorch 2024-08-28 2024-08-28 26 8 \n", "\n", " closed_discussion open_no_discussion closed_no_discussion \\\n", "0 0 18 0 \n", "\n", " open_questions \\\n", "0 [What is the root cause of the performance reg... \n", "\n", " executive_summary ... \\\n", "0 The PyTorch repository is facing various chall... ... \n", "\n", " themes_count_model_loading themes_count_model_fine_tuning_and_training \\\n", "0 3 3 \n", "\n", " themes_count_model_inference \\\n", "0 3 \n", "\n", " themes_count_distributed_training_and_multi_gpu \\\n", "0 3 \n", "\n", " themes_count_performance_and_optimization \\\n", "0 5 \n", "\n", " themes_count_quantization_and_mixed_precision themes_count_documentation \\\n", "0 1 1 \n", "\n", " themes_count_cuda_compatibility \\\n", "0 2 \n", "\n", " themes_count_model_evaluation_and_benchmarking themes_count_miscellaneous \n", "0 1 14 \n", "\n", "[1 rows x 28 columns]\n" ] } ], "source": [ "print(f\"[Showing 5 out of {overview.shape[0]} rows]\\n\")\n", "print(overview.head())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualizing the data\n", "\n", "Based on this data we can easily create some plots to graphically understand the activity in the repo.\n", "\n", "Some additional data can be accessed via the github API, but this requires you to have push-access to this repo." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Plotting traffic trends...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Github fetch failed for . Make sure you have push-access to pytorch/pytorch!\n", "Github fetch failed for . Make sure you have push-access to pytorch/pytorch!\n", "Github fetch failed for . Make sure you have push-access to pytorch/pytorch!\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Plotting issue trends...\n" ] } ], "source": [ "# Create graphs and charts\n", "plot_folder = out_folder + \"/plots\"\n", "os.makedirs(plot_folder, exist_ok=True)\n", "draw_all_plots(repo_name, plot_folder, overview)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Putting it together\n", "\n", "Now that we have all the data and insights, we can create a PDF report" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ " exec_summary = overview['executive_summary'].iloc[0]\n", " open_qs = overview['open_questions'].iloc[0]\n", " key_challenges_data = challenges[['key_challenge', 'possible_causes', 'remediations', 'affected_issues']].to_dict('records')\n", " create_report_pdf(repo_name, start_date, end_date, key_challenges_data, exec_summary, open_qs, out_folder)" ] } ], "metadata": { "kernelspec": { "display_name": "haystack", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 2 }