# provider-litellm (LiteLLM Provider)

You can run this example with:

```bash
npx promptfoo@latest init --example provider-litellm
cd provider-litellm
```

This example demonstrates how to use the LiteLLM provider with promptfoo to evaluate multiple models through a unified interface.

## What is LiteLLM?

LiteLLM provides a unified interface to 400+ LLMs. Instead of managing different APIs and authentication methods for each provider, you can use a single interface to access models from OpenAI, Anthropic, Google, and many more.

## Quick Start

1. **Set the API keys for the full example**:

   The checked-in evaluation calls all three chat routes and uses OpenAI embeddings for similarity assertions, so running it unchanged requires all three keys:

   ```bash
   export OPENAI_API_KEY=your-openai-key
   export ANTHROPIC_API_KEY=your-anthropic-key
   export GOOGLE_AI_API_KEY=your-google-key
   ```

   The proxy can start with any one of these keys. To evaluate a subset, remove unused chat providers from `promptfooconfig.yaml` and their routes from `litellm_config.yaml`, or use a promptfoo config that selects only routes with configured credentials.

   Keep `OPENAI_API_KEY` for the default embedding route even if you omit GPT chat. To run without OpenAI, configure an embedding provider you can access in both configs, or remove the `similar` assertion and its `defaultTest.options.provider.embedding` setting.

2. **Install the LiteLLM proxy with Python 3.10–3.14**:

   ```bash
   python -m venv .venv
   source .venv/bin/activate # Windows: .venv\Scripts\activate
   python -m pip install --upgrade 'litellm[proxy]>=1.101.0,<2'
   ```

3. **Start the LiteLLM proxy**:

   ```bash
   # Use the provided script
   ./start-proxy.sh

   # Or manually:
   litellm --config litellm_config.yaml --port 4000
   ```

4. **Run the evaluation**:
   ```bash
   npx promptfoo@latest eval
   ```

## Features

- **Unified Interface**: Access OpenAI, Anthropic, Google, and 400+ other models through one API
- **Chat Models**: GPT-4.1, Claude Sonnet 5, Gemini 2.5
- **Embedding Models**: Support for similarity assertions via embedding models
- **Simple Configuration**: One provider syntax for all models
- **Cost Tracking**: LiteLLM proxy can track usage across providers
- **Load Balancing**: Distribute requests across multiple instances

## How It Works

The LiteLLM provider in promptfoo connects to a LiteLLM proxy server (default port 4000). The proxy handles:

- Authentication and routing to various providers
- Standardizing request/response formats
- Error handling and retries
- Optional features like caching and rate limiting

## Configuration Files

- `promptfooconfig.yaml` - Main evaluation configuration
- `litellm_config.yaml` - LiteLLM proxy server configuration
- `start-proxy.sh` - Helper script to start the proxy

The proxy keeps client-facing `model_name` aliases separate from backend routes. For Google AI Studio, the [LiteLLM Gemini backend](https://docs.litellm.ai/docs/providers/gemini) uses `gemini/gemini-2.5-pro`; promptfoo continues to select `litellm:gemini-2.5-pro`. API keys in the proxy YAML use LiteLLM's `os.environ/VARIABLE_NAME` syntax.

## Example Configuration

The example evaluates translation and creative writing tasks across three different providers:

Each test supplies its own prompt so translation assertions apply to translations
and the three-line assertion applies to the haiku. The evaluation runs six cases.

```yaml
providers:
  - litellm:gpt-4.1
  - litellm:claude-sonnet-5
  - litellm:gemini-2.5-pro

defaultTest:
  options:
    provider:
      embedding: litellm:embedding:text-embedding-3-large
```

## Troubleshooting

### Common Errors

- **"Connection refused 0.0.0.0:4000"**: The LiteLLM proxy server is not running. Start it first with `./start-proxy.sh`
- **"API key not found"**: Set the appropriate environment variables before starting the proxy
- **"Model not found"**: Ensure the model is included when starting the proxy server

### Verify Setup

1. **Check proxy is running**:

   ```bash
   curl http://localhost:4000/health/liveliness
   ```

2. **Verify a required key is set**:
   ```bash
   test -n "$OPENAI_API_KEY" && echo 'OpenAI key is set'
   ```

## Advanced Usage

### Custom Server URL

If your LiteLLM proxy runs on a different host or port:

```yaml
providers:
  - id: litellm:gpt-4.1
    config:
      apiBaseUrl: https://your-litellm-server.com
```

### Using Config File

For more complex setups, use the config file:

```bash
litellm --config litellm_config.yaml
```

## Learn More

- [LiteLLM Documentation](https://docs.litellm.ai/docs/)
- [Promptfoo LiteLLM Provider Docs](/docs/providers/litellm)
- [LiteLLM Proxy Setup](https://docs.litellm.ai/docs/proxy/quick_start)