--- sidebar_label: Transformers.js description: Run local LLM inference with Transformers.js for embeddings and text generation without external APIs --- # Transformers.js The Transformers.js provider runs ONNX models locally in Node.js using [Transformers.js v4](https://huggingface.co/docs/transformers.js). It supports CPU inference and a WebGPU backend; no external inference API is required. ## Installation Transformers.js and its ONNX runtimes are not included in the default install. Install the runtime alongside Promptfoo in your project: ```bash npm install promptfoo @huggingface/transformers@^4.0.0 ``` For a global installation, use `npm install -g promptfoo @huggingface/transformers@^4.0.0`. For a one-off eval, run `npx --package=promptfoo --package=@huggingface/transformers@^4.0.0 promptfoo eval -c /absolute/path/to/promptfooconfig.yaml` from an empty directory outside an existing npm project, with neither package installed locally. If either package is already installed in your project, use the project installation command above. Model files are downloaded separately on first use. ## Quick Start ### Embeddings ```yaml providers: - transformers:feature-extraction:Xenova/all-MiniLM-L6-v2 ``` Popular models: `Xenova/all-MiniLM-L6-v2` (384d), `onnx-community/all-MiniLM-L6-v2-ONNX` (384d), `Xenova/bge-small-en-v1.5` (384d), `nomic-ai/nomic-embed-text-v1.5` (768d) ### Text Generation ```yaml providers: - transformers:text-generation:Xenova/gpt2 ``` Popular models: `Xenova/gpt2`, `onnx-community/Qwen3-0.6B-ONNX`, `onnx-community/Llama-3.2-1B-Instruct-ONNX` :::note Text generation runs on CPU and is best for testing. For production, consider [Ollama](/docs/providers/ollama) or cloud APIs. ::: ## Configuration ### Common Options These options apply to both embedding and text generation providers: | Option | Description | Default | | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------- | | `device` | `'auto'`, `'cpu'`, `'gpu'`, `'wasm'`, `'webgpu'`, `'cuda'`, `'dml'`, `'coreml'`, `'webnn'`, `'webnn-npu'`, `'webnn-gpu'`, `'webnn-cpu'` | `'auto'` | | `dtype` | Quantization: `'fp32'`, `'fp16'`, `'q8'`, `'int8'`, `'uint8'`, `'q4'`, `'bnb4'`, `'q4f16'` | `'auto'` | | `cacheDir` | Override model cache directory | System default | | `localFilesOnly` | Skip downloads, use cached models only | `false` | | `revision` | Model version/branch | `'main'` | | `sessionOptions` | ONNX runtime session options, passed through as `session_options` | - | ### Embedding Options ```yaml providers: - id: transformers:feature-extraction:Xenova/bge-small-en-v1.5 config: prefix: 'Represent this sentence for searching relevant passages: ' # BGE retrieval queries pooling: cls # BGE v1.5 uses the CLS token embedding normalize: true # L2 normalize embeddings dtype: q8 ``` **Model prefixes:** Follow the model card for your embedding model. [BGE v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) uses the instruction above for retrieval queries; documents need no prefix. [E5 v2](https://huggingface.co/intfloat/e5-small-v2) uses `prefix: 'query: '` for queries and `prefix: 'passage: '` for documents. MiniLM models need no prefix. [Nomic Embed v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5#task-instruction-prefixes) requires `prefix: 'search_query: '` for retrieval queries and `prefix: 'search_document: '` for indexed documents. Use the model card's task prefix for other workloads. Keep query and document preprocessing compatible with your vector index; rebuild affected stored embeddings when changing document preprocessing, model, or dimensions. :::tip `transformers:embeddings:` is an alias for `transformers:feature-extraction:`. ::: ### Text Generation Options ```yaml providers: - id: transformers:text-generation:onnx-community/Qwen3-0.6B-ONNX config: maxNewTokens: 256 temperature: 0.7 topK: 50 topP: 0.9 doSample: true repetitionPenalty: 1.1 noRepeatNgramSize: 3 numBeams: 1 returnFullText: false dtype: q4 ``` ## Using for Similarity Assertions Use local embeddings as a grading provider for `similar` assertions: ```yaml defaultTest: options: provider: embedding: id: transformers:feature-extraction:Xenova/all-MiniLM-L6-v2 providers: - openai:gpt-4o-mini tests: - vars: question: 'What is photosynthesis?' assert: - type: similar value: 'Photosynthesis converts light to chemical energy in plants' threshold: 0.8 ``` Or override per-assertion: ```yaml assert: - type: similar value: 'Expected output' threshold: 0.75 provider: transformers:feature-extraction:Xenova/all-MiniLM-L6-v2 ``` ## Performance - **Caching:** Pipelines are cached after first load. Initial model download may take time, but subsequent runs are fast. - **Quantization:** Use `dtype: q4` or `dtype: q8` for faster inference and lower memory. Use `dtype: q4f16` for WebGPU-optimized quantization. - **WebGPU:** v4 includes a WebGPU runtime written in C++ with improved performance. Use `device: webgpu` on supported systems. - **Concurrency:** For limited RAM, use `promptfoo eval -j 1` to run serially. ## Troubleshooting | Problem | Solution | | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | Dependency not installed | Follow the [installation instructions](#installation) to install the runtime alongside Promptfoo. | | Model not found | Verify model exists at [HuggingFace](https://huggingface.co/models?library=transformers.js) with ONNX weights. Try `Xenova` or `onnx-community` models. | | Out of memory | Use `dtype: q4`, run with `-j 1`, or try smaller models | | Slow first run | Models download on first use. Pre-download with `await pipeline('feature-extraction', 'model-name')` | ## Supported Models Browse compatible models at [huggingface.co/models?library=transformers.js](https://huggingface.co/models?library=transformers.js). Key organizations: **onnx-community** (optimized ONNX exports, recommended for v4), **Xenova** (legacy ONNX models, still compatible)