LLamaRecipes/getting-started/inference/api_inference

API Inference

This module provides a command-line interface for interacting with Llama models through the Llama API.

Overview

The api_inference.py script allows you to:

• Connect to Llama's API using your API key
• Launch a Gradio web interface for sending prompts to Llama models
• Get completions from models like Llama-4-Maverick-17B

Prerequisites

• Python 3.8 or higher
• A valid Llama API key
• Required Python packages:
  • gradio
  • llama_api_client

Installation

Ensure you have the required packages installed:

pip install gradio llama_api_client

Usage

You can run the script from the command line using:

python api_inference.py [OPTIONS]

Command-line Options

• --api-key: Your API key (optional)
  • If not provided, the script will look for the appropriate environment variable based on the provider
• --provider: API provider to use (optional, default: "Llama")
  • Available options: "Llama", "OpenAI"

Setting Up Your API Key

You can provide your API key in one of two ways:

1. Command-line argument:

   python api_inference.py --api-key YOUR_API_KEY --provider Llama
   

2. Environment variable: The environment variable name depends on the provider you choose: ```bash

   For Llama (default provider)

   export LLAMA_API_KEY=YOUR_API_KEY

# For OpenAI export OPENAI_API_KEY=YOUR_API_KEY

   For Windows:
   ```bash
   # Command Prompt (example for Llama)
   set LLAMA_API_KEY=YOUR_API_KEY

   # PowerShell (example for Llama)
   $env:LLAMA_API_KEY="YOUR_API_KEY"

Example

1. Run the script: ```bash

   Using Llama (default provider)

   python api_inference.py --api-key YOUR_API_KEY

# Using a different provider python api_inference.py --api-key YOUR_API_KEY --provider OpenAI

2. The script will launch a Gradio web interface (typically at http://127.0.0.1:7860)

3. In the interface:
   - Enter your prompt in the text box
   - The default model is "Llama-4-Maverick-17B-128E-Instruct-FP8" but you can change it
   - Click "Submit" to get a response from the model

## Troubleshooting

### API Key Issues

If you see an error like:

No API key provided and *_API_KEY environment variable not found

Make sure you've either:
- Passed the API key using the `--api-key` argument
- Set the appropriate environment variable for your chosen provider (LLAMA_API_KEY)

## Advanced Usage

You can modify the script to use different models or customize the Gradio interface as needed.

## Implementation Notes

- The script uses type hints for better code readability and IDE support:
  ```python
  api_key: Optional[str] = args.api_key

This line uses the Optional type from the typing module to indicate that api_key can be either a string or None. The Optional type is imported from the typing module at the beginning of the script.

License

[Include license information here]