---
sidebar_label: Web Browser
description: 'Execute LLM evaluations directly in browsers using WebGPU acceleration and local models for privacy-preserving testing'
---

# Browser Provider

The Browser Provider uses [Playwright](https://playwright.dev/) to navigate pages, interact with elements, and extract data from web applications. It supports Chromium (Chrome and Edge), Firefox, and WebKit (the Safari engine).

## When to Use the Browser Provider

The Browser Provider should only be used when simpler alternatives are not possible:

1. **Try these first:**
   - [HTTP Provider](/docs/providers/http) - For API calls and simple HTML responses
   - [WebSocket Provider](/docs/providers/websocket) - For real-time connections
   - [Custom Python Provider](/docs/providers/python) - For custom logic with existing libraries
   - [Custom JavaScript Provider](/docs/providers/custom-api) - For Node.js-based solutions

2. **Use Browser Provider only when:**
   - The application requires JavaScript execution to render content
   - You need to interact with complex UI elements (dropdowns, modals, etc.)
   - Authentication requires browser-based workflows (OAuth, SSO)
   - You need to test actual user interactions (clicks, typing, scrolling)

### Important Considerations

When using browser automation:

1. **Rate Limiting**: Always implement delays between requests to avoid overwhelming servers
2. **Anti-Bot Detection**: Many websites employ anti-bot measures that can detect and block automated browsers
3. **Resource Usage**: Browser automation uses more CPU and memory than direct API calls and adds page loading time
4. **Legal Compliance**: Always check the website's Terms of Service and robots.txt before automating

## Prerequisites

Browser automation is an optional feature. Install Promptfoo, Playwright, and the stealth plugin together in your project, then install the matching Chromium binary:

```bash
npm install promptfoo "playwright@^1.63.0" "playwright-extra@^4.3.6" "puppeteer-extra-plugin-stealth@^2.11.2"
npx playwright install chromium
```

For a global CLI, install the same packages with `npm install -g` and run `playwright install chromium`.

Playwright 1.63 and later no longer support Ubuntu 20.04. Check [Playwright's system requirements](https://playwright.dev/docs/intro#system-requirements) before installing the browser on Linux.

The browser provider supports Chromium-based browsers, including Chrome and Edge.

## Configuration

To use the Browser Provider, set the provider `id` to `browser` and define a series of `steps` to execute:

```yaml
providers:
  - id: browser
    config:
      steps:
        - action: navigate
          args:
            url: 'https://example.com'
        - action: type
          args:
            selector: '#search-input'
            text: '{{prompt}}'
        - action: click
          args:
            selector: '#search-button'
        - action: extract
          args:
            selector: '#results'
          name: searchResults
      transformResponse: 'extracted.searchResults'
```

### Connecting to Existing Browser Sessions

You can connect to an existing Chrome browser session (e.g., with OAuth authentication already completed):

```yaml
providers:
  - id: browser
    config:
      connectOptions:
        debuggingPort: 9222 # Chrome debugging port

      steps:
        # Your test steps here
```

**Setup Instructions**:

1. Start Chrome with debugging: `chrome --remote-debugging-port=9222 --user-data-dir=/tmp/test`
2. Complete authentication manually
3. Run your tests

**Connection Options**:

- `debuggingPort`: Port number for Chrome DevTools Protocol (default: 9222)
- `mode`: Connection mode - `'cdp'` (default) or `'websocket'`
- `wsEndpoint`: Direct WebSocket endpoint (when using `mode: 'websocket'`)

### Multi-Turn Session Persistence

For multi-turn strategies like Hydra, Crescendo, or GOAT, you can persist the browser session across turns. This keeps the same page open and maintains conversation state in chat-based applications.

```yaml
evaluateOptions:
  maxConcurrency: 1 # Persistent sessions share one stateful browser workflow

providers:
  - id: browser
    config:
      persistSession: true # Keep page open across turns
      connectOptions:
        debuggingPort: 9222

      steps:
        # Navigate only on first turn
        - action: navigate
          runOnce: true
          args:
            url: 'https://example.com/chat'

        # Wait for page load only on first turn
        - action: wait
          runOnce: true
          args:
            ms: 3000

        # These run on every turn
        - action: type
          args:
            selector: '#chat-input'
            text: '{{prompt}}<enter>'

        - action: wait
          args:
            ms: 5000

        - action: extract
          args:
            script: |
              // Extract the latest AI response
              const messages = document.querySelectorAll('.ai-message');
              return messages[messages.length - 1]?.textContent || '';
          name: response

      transformResponse: 'extracted.response'
```

**Key options:**

- `persistSession: true` - Keep the browser page open between `callApi()` invocations. Because this is a stateful workflow, Promptfoo runs it with concurrency `1`.
- `runOnce: true` on steps - Execute only on the first turn (skip on subsequent turns)

This is essential for testing multi-turn jailbreak strategies against chat interfaces where you need to maintain conversation context.

## Supported Actions

The Browser Provider supports the following actions:

### Core Actions

#### 1. `navigate` - Load a webpage

Navigate to a specified URL.

```yaml
- action: navigate
  args:
    url: 'https://example.com/search?q={{query}}'
```

#### 2. `click` - Click an element

Click on any clickable element (button, link, etc.).

```yaml
- action: click
  args:
    selector: 'button[type="submit"]'
    optional: true # Won't fail if element doesn't exist
```

#### 3. `type` - Enter text

Type text into input fields, textareas, or any editable element.

```yaml
- action: type
  args:
    selector: 'input[name="username"]'
    text: '{{username}}'
```

Special keys:

- `<enter>` - Press Enter key
- `<tab>` - Press Tab key
- `<escape>` - Press Escape key

#### 4. `extract` - Get text content

Extract text from any element or run custom JavaScript to extract data. The extracted content is available in `transformResponse`.

**Using a selector:**

```yaml
- action: extract
  args:
    selector: '.result-title'
  name: title # Access as extracted.title
```

**Using a custom script:**

```yaml
- action: extract
  args:
    script: |
      const fullText = document.body.innerText;
      return fullText.split('Response:')[1]?.trim() || '';
  name: aiResponse # Access as extracted.aiResponse
```

The `script` option runs JavaScript in the browser context and returns the result. This is useful when you need complex extraction logic that CSS selectors can't handle.

#### 5. `wait` - Pause execution

Wait for a specified duration (in milliseconds).

```yaml
- action: wait
  args:
    ms: 3000 # Wait 3 seconds
```

#### 6. `waitForNewChildren` - Wait for newly added direct children

Use this action to detect dynamic content that adds elements under a parent. It waits for the number of direct child elements to increase, so it does not detect streamed text or nested content updates inside an existing child element.

```yaml
- action: waitForNewChildren
  args:
    parentSelector: '#results-container'
    delay: 500 # Wait before capturing the initial child count
    timeout: 10000 # Max wait time 10 seconds
```

#### 7. `screenshot` - Capture the page

Take a screenshot of the current page state.

```yaml
- action: screenshot
  args:
    path: 'screenshot.png'
    fullPage: true # Capture entire page, not just viewport
```

### Action Parameters

| Action             | Required Args                  | Optional Args      | Description                                  |
| ------------------ | ------------------------------ | ------------------ | -------------------------------------------- |
| navigate           | `url`                          | -                  | URL to navigate to                           |
| click              | `selector`                     | `optional`         | CSS selector of element to click             |
| type               | `selector`, `text`             | -                  | CSS selector and text to type                |
| extract            | `selector` OR `script`, `name` | -                  | CSS selector or JS script, and variable name |
| wait               | `ms`                           | -                  | Milliseconds to wait                         |
| waitForNewChildren | `parentSelector`               | `delay`, `timeout` | Parent whose direct children are counted     |
| screenshot         | `path`                         | `fullPage`         | File path to save screenshot                 |

## Response Parsing

Use the `transformResponse` config option to extract specific data from the results. Function parsers receive two positional arguments; string expressions can access the same variables:

- `extracted`: An object containing named results from `extract` actions
- `finalHtml`: The final HTML content of the page after all actions are completed

## Variables and Templating

You can use Nunjucks templating in your configuration, including the `{{prompt}}` variable and any other variables passed in the test context.

```yaml
providers:
  - id: browser
    config:
      steps:
        - action: navigate
          args:
            url: 'https://example.com/search?q={{prompt}}'
        - action: extract
          args:
            selector: '#first-result'
          name: topResult
      transformResponse: 'extracted.topResult'

tests:
  - vars:
      prompt: 'What is the capital of France?'
```

## Using as a Library

If you are using promptfoo as a [node library](/docs/usage/node-package/), you can provide the equivalent provider config:

```js
{
  // ...
  providers: [{
    id: 'browser',
    config: {
      steps: [
        { action: 'navigate', args: { url: 'https://example.com' } },
        { action: 'type', args: { selector: '#search', text: '{{prompt}}' } },
        { action: 'click', args: { selector: '#submit' } },
        { action: 'extract', args: { selector: '#results' }, name: 'searchResults' }
      ],
      transformResponse: (extracted, finalHtml) => extracted.searchResults,
    }
  }],
}
```

## Reference

Supported config options:

| Option            | Type                                                                             | Description                                                                                                                  |
| ----------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| headless          | `boolean`                                                                        | Whether to run the browser in headless mode. Defaults to `true`.                                                             |
| cookies           | `string` \| `{ name: string; value: string; domain?: string; path?: string; }[]` | A string or array of cookies to set on the browser                                                                           |
| transformResponse | `string` \| `Function`                                                           | A function receiving `(extracted, finalHtml)`, or a string expression using those variables, to parse the response.          |
| steps             | `BrowserAction[]`                                                                | An array of actions to perform in the browser                                                                                |
| timeoutMs         | `number`                                                                         | The maximum time in milliseconds to wait for the browser operations to complete                                              |
| persistSession    | `boolean`                                                                        | Keep the browser page open across multiple `callApi()` invocations. Required for multi-turn strategies. Defaults to `false`. |
| connectOptions    | `object`                                                                         | Options for connecting to an existing browser (`debuggingPort`, `mode`, `wsEndpoint`)                                        |

Note: All string values in the config support Nunjucks templating. This means you can use the `{{prompt}}` variable or any other variables passed in the test context.

### Browser Support

While Playwright supports multiple browsers (Chromium, Firefox, and WebKit), promptfoo's browser provider only implements Chromium support. This includes:

- **Chrome** - Google's browser
- **Edge** - Microsoft's Chromium-based browser
- **Chromium** - Open-source browser project

The implementation uses `playwright-extra` with the Chromium engine for enhanced stealth capabilities to avoid detection.

### Supported Browser Actions

The `steps` array in the configuration can include the following actions:

| Action             | Description                                        | Required Args                                    | Optional Args                           |
| ------------------ | -------------------------------------------------- | ------------------------------------------------ | --------------------------------------- |
| navigate           | Navigate to a specified URL                        | `url`: string                                    | `runOnce`: boolean                      |
| click              | Click on an element                                | `selector`: string                               | `optional`: boolean, `runOnce`: boolean |
| extract            | Extract text content from element or run JS script | (`selector` OR `script`): string, `name`: string |                                         |
| screenshot         | Take a screenshot of the page                      | `path`: string                                   | `fullPage`: boolean                     |
| type               | Type text into an input field                      | `selector`: string, `text`: string               | `runOnce`: boolean                      |
| wait               | Wait for a specified amount of time                | `ms`: number                                     | `runOnce`: boolean                      |
| waitForNewChildren | Wait for new direct child elements under a parent  | `parentSelector`: string                         | `delay`: number, `timeout`: number      |

Each action in the `steps` array should be an object with the following structure:

```typescript
{
  action: string;
  args: {
    [key: string]: any;
  };
  name?: string;
  runOnce?: boolean;
}
```

Each step in the `steps` array should have the following structure:

- `action`: Specifies the type of action to perform (e.g., 'navigate', 'click', 'type').
- `args`: Contains the required and optional arguments for the action.
- `name` (optional): Used to name extracted content in the 'extract' action.
- `runOnce` (optional): If `true`, the step only executes on the first turn. Used with `persistSession` for multi-turn strategies.

Steps are executed sequentially, enabling complex web interactions.

All string values in `args` support Nunjucks templating, allowing use of variables like `{{prompt}}`.

## Advanced Features

### Playwright Recorder Tools

The easiest way to create browser automation scripts is to record your interactions:

#### Chrome Extension (Recommended)

The [Playwright Recorder Chrome Extension](https://chrome.google.com/webstore/detail/playwright-recorder/pbbgjmghmjcpeelnheiphabndacpdfbc) is helpful for quickly generating selectors:

1. Install the extension from the Chrome Web Store
2. Navigate to your target website
3. Click the extension icon and start recording
4. Perform your actions (click, type, etc.)
5. Stop recording and copy the generated selectors/code
6. Adapt the code for promptfoo's browser provider format

This extension is useful because it:

- Shows selectors in real-time as you hover over elements
- Generates multiple selector options (CSS, text, XPath)
- Allows you to copy individual selectors without recording full actions

#### Playwright Inspector (All Browsers)

For cross-browser recording, use Playwright's built-in recorder:

```bash
npx playwright codegen https://example.com
```

This opens an interactive browser window where you can perform actions and see generated code in real-time. You can choose between Chromium, Firefox, or WebKit.

### Selector Strategies

Playwright supports various selector strategies:

| Strategy | Example                          | Description                   |
| -------- | -------------------------------- | ----------------------------- |
| CSS      | `#submit-button`                 | Standard CSS selectors        |
| Text     | `text=Submit`                    | Find elements by text content |
| Role     | `role=button[name="Submit"]`     | ARIA role-based selectors     |
| Test ID  | `data-testid=submit`             | Data attribute selectors      |
| XPath    | `xpath=//button[@type="submit"]` | XPath expressions             |

For the most reliable selectors:

- Prefer stable attributes like IDs and data-testid
- Use role-based selectors for accessibility
- Avoid position-based selectors that can break with layout changes

### Debugging

#### 1. Disable Headless Mode

See exactly what's happening in the browser:

```yaml
providers:
  - id: browser
    config:
      headless: false # Opens visible browser window
```

#### 2. Enable Debug Logging

Get detailed information about each action:

```bash
npx promptfoo@latest eval --verbose
```

#### 3. Take Screenshots

Capture the page state during execution:

```yaml
steps:
  - action: navigate
    args:
      url: 'https://example.com'
  - action: screenshot
    args:
      path: 'debug-{{_attempt}}.png'
```

### Performance Optimization

1. **Use headless mode in production**: It's faster and uses fewer resources
2. **Minimize wait times**: Only wait as long as necessary
3. **Batch operations**: Group related actions together
4. **Reuse browser contexts**: For multiple tests against the same site

### Best Practices for Rate Limiting

Implementing proper rate limiting is crucial to avoid detection and server overload:

```yaml
providers:
  - id: browser
    config:
      steps:
        # Always start with a respectful delay
        - action: wait
          args:
            ms: 2000

        - action: navigate
          args:
            url: 'https://example.com'

        # Wait between actions
        - action: wait
          args:
            ms: 1000

        - action: click
          args:
            selector: '#button'

        # Final delay before next request
        - action: wait
          args:
            ms: 3000
```

**Tips for avoiding detection:**

- Randomize delays between actions (1-3 seconds)
- Use the stealth plugin (included with playwright-extra)
- Avoid patterns that look automated
- Consider using different user agents
- Respect robots.txt and rate limits

### Dealing with Anti-Bot Measures

Many websites implement anti-bot detection systems (like Cloudflare, reCAPTCHA, etc.). Here's how to handle common scenarios:

#### Common Anti-Bot Challenges

| Challenge              | Detection Method                 | Mitigation Strategy                             |
| ---------------------- | -------------------------------- | ----------------------------------------------- |
| Browser fingerprinting | JavaScript checks for automation | Stealth plugin helps mask automation            |
| Behavioral analysis    | Mouse movements, typing patterns | Add realistic delays and interactions           |
| IP rate limiting       | Too many requests from one IP    | Implement proper delays, use proxies cautiously |
| CAPTCHA challenges     | Human verification tests         | Consider if the site allows automation          |
| User-Agent detection   | Checking for headless browsers   | Use realistic user agent strings                |

#### Example with Anti-Bot Considerations

```yaml
providers:
  - id: browser
    config:
      headless: false # Some sites detect headless mode
      steps:
        # Human-like delay before starting
        - action: wait
          args:
            ms: 3000

        - action: navigate
          args:
            url: '{{url}}'

        # Wait for any anti-bot checks to complete
        - action: wait
          args:
            ms: 5000

        # Type slowly like a human would
        - action: type
          args:
            selector: '#search'
            text: '{{query}}'
            delay: 100 # Delay between keystrokes
```

**Note**: If a website has strong anti-bot measures, it's often a sign that automation is not welcome. Always respect the website owner's wishes and consider reaching out for API access instead.

## Example: Testing a Login Flow

Here's a template for testing a login workflow against your own application:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: Test login functionality

prompts:
  - 'Login with username {{username}} and password {{password}}'

providers:
  - id: browser
    config:
      headless: true
      steps:
        - action: navigate
          args:
            url: 'https://example.com/login'

        - action: type
          args:
            selector: '#username'
            text: '{{username}}'

        - action: type
          args:
            selector: '#password'
            text: '{{password}}'

        - action: click
          args:
            selector: 'button[type="submit"]'

        - action: wait
          args:
            ms: 2000

        - action: extract
          args:
            selector: '.welcome-message'
          name: welcomeText

      transformResponse: '({ output: extracted.welcomeText })'

tests:
  - vars:
      username: 'testuser'
      password: 'testpass123'
    assert:
      - type: contains
        value: 'Welcome'
```

## Troubleshooting

### Common Issues and Solutions

| Issue                          | Cause                                      | Solution                                                                                         |
| ------------------------------ | ------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| "Element not found"            | Selector incorrect or element not loaded   | • Verify selector in DevTools<br />• Add wait before action<br />• Check if element is in iframe |
| "Timeout waiting for selector" | Page loads slowly or element never appears | • Increase timeout<br />• Add explicit wait actions<br />• Check for failed network requests     |
| "Access denied" or 403 errors  | Anti-bot detection triggered               | • Use headless: false<br />• Add more delays<br />• Check if automation is allowed               |
| "Click intercepted"            | Element covered by overlay                 | • Wait for overlays to disappear<br />• Scroll element into view<br />• Use force click option   |
| Inconsistent results           | Timing or detection issues                 | • Add consistent delays<br />• Use stealth plugin<br />• Test during off-peak hours              |

### Debugging Anti-Bot Detection

If you suspect anti-bot measures are blocking your automation:

```yaml
providers:
  - id: browser
    config:
      headless: false # Always start with headed mode for debugging
      steps:
        - action: navigate
          args:
            url: '{{url}}'

        - action: screenshot
          args:
            path: 'debug-landing.png' # Check if you hit a challenge page

        - action: wait
          args:
            ms: 10000 # Longer wait to see what happens

        - action: screenshot
          args:
            path: 'debug-after-wait.png'
```

## Useful Resources

- [Playwright Documentation](https://playwright.dev/docs/intro) - Official Playwright docs
- [Playwright Browsers Guide](https://playwright.dev/docs/browsers) - Detailed information about supported browsers
- [Playwright Locators Guide](https://playwright.dev/docs/locators) - Learn about CSS, text, and other selector strategies
- [Playwright Best Practices](https://playwright.dev/docs/best-practices) - Tips for reliable automation
- [Playwright Debugging Guide](https://playwright.dev/docs/debug) - Playwright Inspector and other tools for authoring and debugging tests
- [Chrome DevTools Guide](https://developer.chrome.com/docs/devtools/) - For inspecting elements and finding selectors

---

For more examples, check out the [headless-browser example](https://github.com/promptfoo/promptfoo/tree/main/examples/integration-browser/headless) in our GitHub repository.