{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# PyRIT Scan\n", "\n", "`pyrit_scan` is the primary command-line tool for running automated security assessments and red teaming attacks against AI systems. It leverages [scenarios](../code/scenarios/0_scenarios.ipynb) to define attack techniques and supports flexible [configuration](../code/setup/1_configuration.ipynb) for targeting different AI endpoints.\n", "\n", "For configuration setup, see [Configuration](../getting_started/configuration.md).\n", "\n", "For scenario-specific examples, see [AIRT](airt.ipynb), [Foundry](foundry.ipynb), and [Garak](garak.ipynb).\n", "\n", "Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.\n", "\n", "## Starting a Backend Server\n", "\n", "`pyrit_scan` is a thin client that talks to a PyRIT backend server (by default at `http://localhost:8000`).\n", "Before running any command that reaches the backend (listing scenarios, running a scan, etc.) you need a\n", "server. Start a local one with `--start-server`; it launches a detached `pyrit_backend` process that stays\n", "up and is reused by every command below. We stop it again at the end of the notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Server is running at http://localhost:8000\n" ] } ], "source": [ "!pyrit_scan start-server" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "## Quick Start\n", "\n", "For help:" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "usage: pyrit_scan [-h] ...\n", "\n", "PyRIT Scanner - Run AI security scenarios from the command line.\n", "\n", "Requires a running PyRIT backend server. Use 'start-server' to launch one,\n", "or connect to an existing server with --server-url.\n", "\n", "Global options (usable with any command, before or after the verb):\n", " --server-url --config-file --log-level --request-timeout --start-server --startup-timeout\n", "Run 'pyrit_scan --help' for full option descriptions and a command's arguments.\n", "\n", "Examples:\n", " # Start the backend server\n", " pyrit_scan start-server\n", "\n", " # List scenarios, targets, or converters\n", " pyrit_scan list-scenarios\n", " pyrit_scan list-targets\n", "\n", " # Run single-turn cyber attacks against a target\n", " pyrit_scan run airt.cyber --target openai_chat --techniques single_turn\n", "\n", " # Run rapid response with specific datasets and concurrency\n", " pyrit_scan run airt.rapid_response --target openai_chat\n", " --techniques role_play_movie_script --dataset-names airt_hate\n", " --max-dataset-size 5 --max-concurrency 4\n", "\n", " # Attach registered converters to a technique (repeatable, applied in order)\n", " pyrit_scan run airt.rapid_response --target openai_chat\n", " --techniques role_play_movie_script:converter.translation_spanish:converter.leetspeak\n", "\n", " # List recent runs, then inspect one (overview by default; --view attacks for per-attack rows)\n", " pyrit_scan scenario-history 20\n", " pyrit_scan scenario-results 605d715b-7c07-4bde-a8f9-22fea0b50c4f --view attacks\n", "\n", " # Register a custom initializer from a Python script\n", " pyrit_scan add-initializer ./my_custom_init.py\n", "\n", " # Connect to a remote server\n", " pyrit_scan list-scenarios --server-url http://remote:8000\n", "\n", " # Stop the server\n", " pyrit_scan stop-server\n", "\n", "options:\n", " -h, --help show this help message and exit\n", "\n", "commands:\n", " \n", " run Run a scenario against a target\n", " list-scenarios List all available scenarios\n", " list-initializers\n", " List all available initializers\n", " list-targets List all available targets\n", " list-converters List all registered converter instances\n", " list-datasets List all available datasets\n", " add-initializer Register initializer(s) from Python script file(s)\n", " scenario-results\n", " Inspect the results of a completed scenario run\n", " scenario-history\n", " List recent scenario runs\n", " start-server Start a local backend server\n", " stop-server Stop the backend server\n" ] } ], "source": [ "!pyrit_scan --help" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "### Discovery\n", "\n", "List all available scenarios:" ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Available Scenarios:\n", "================================================================================\n", "\u001b[1m\u001b[36m\n", " adaptive.text_adaptive\u001b[0m\n", " Class: TextAdaptive\n", " Description:\n", " Adaptive text-attack scenario. Selects techniques per-objective via an\n", " epsilon-greedy selector over the set of selected techniques.\n", " ``prompt_sending`` runs as the baseline comparison and is excluded from\n", " the adaptive technique pool.\n", " Aggregate Techniques:\n", " - all, default, core, extra, light, multi_turn, single_turn\n", " Available Techniques (17):\n", " role_play_movie_script, role_play_video_game, role_play_trivia_game,\n", " role_play_persuasion, role_play_persuasion_written, many_shot, tap,\n", " crescendo_simulated, crescendo_movie_director,\n", " crescendo_history_lecture, crescendo_journalist_interview, red_teaming,\n", " context_compliance, flip, pair, skeleton_key, violent_durian\n", " Default Technique: default\n", " Default Datasets (7):\n", " airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,\n", " airt_misinformation, airt_leakage\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", " - max_attempts_per_objective (int) [default: '3']: Max techniques tried per objective. Defaults to 3.\n", "\u001b[1m\u001b[36m\n", " airt.cyber\u001b[0m\n", " Class: Cyber\n", " Description:\n", " Cyber scenario implementation for PyRIT. This scenario tests how willing\n", " models are to exploit cybersecurity harms by generating malware. The\n", " Cyber class contains different variations of the malware generation\n", " techniques.\n", " Aggregate Techniques:\n", " - all, default, core, light, multi_turn, single_turn\n", " Available Techniques (14):\n", " context_compliance, crescendo_history_lecture,\n", " crescendo_journalist_interview, crescendo_movie_director,\n", " crescendo_simulated, flip, many_shot, red_teaming,\n", " role_play_movie_script, role_play_persuasion,\n", " role_play_persuasion_written, role_play_trivia_game,\n", " role_play_video_game, tap\n", " Default Technique: default\n", " Default Datasets (1):\n", " airt_malware\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " airt.jailbreak\u001b[0m\n", " Class: Jailbreak\n", " Description:\n", " Jailbreak scenario implementation for PyRIT. Tests how vulnerable a\n", " model is to jailbreak templates. A run is the cross-product of three\n", " selectors: - **dataset** — the harmful objectives (HarmBench). -\n", " **techniques** — two delivery methods for each jailbreak:\n", " ``prompt_sending`` (the template rendered inline into the user message)\n", " and ``jailbreak_system_prompt`` (the template set as the system prompt\n", " with the objective sent as the user turn). - **jailbreaks** — which\n", " jailbreak templates to run (a random ``num_jailbreaks`` sample or an\n", " explicit ``jailbreak_names`` set). ``prompt_sending`` applies each\n", " template as a ``TextJailbreakConverter`` on the outgoing request, so the\n", " objective is rendered inline into the template's ``{{prompt}}`` slot.\n", " ``jailbreak_system_prompt`` instead sets the template as a native system\n", " prompt and sends the objective as its own user turn, so it is only built\n", " for targets that natively support editable history and system prompts\n", " (it is skipped for incapable targets, or raises if it is the only\n", " selected technique). Responses are scored to determine whether the\n", " jailbreak succeeded (non-refusal).\n", " Aggregate Techniques:\n", " - all, default, single_turn\n", " Available Techniques (2):\n", " prompt_sending, jailbreak_system_prompt\n", " Default Technique: default\n", " Default Datasets (1):\n", " harmbench\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: 4]: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: 0]: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", " - num_jailbreaks (int): Draw this many random jailbreak templates for the run. Mutually exclusive with jailbreak_names.\n", " - num_jailbreak_attempts (int) [default: 1]: Number of times to try each (technique x jailbreak template x objective).\n", " - jailbreak_names (list[str]): Explicit jailbreak template file names to run (e.g. aim.yaml dan_11.yaml). When omitted, a random sample is drawn. Mutually exclusive with num_jailbreaks.\n", "\u001b[1m\u001b[36m\n", " airt.leakage\u001b[0m\n", " Class: Leakage\n", " Description:\n", " Leakage scenario implementation for PyRIT. This scenario tests how\n", " susceptible models are to leaking training data, PII, intellectual\n", " property, or other confidential information. Uses the registry/factory\n", " pattern to construct attack techniques.\n", " Aggregate Techniques:\n", " - all, default, airt, core, leakage, light, multi_turn, single_turn\n", " Available Techniques (16):\n", " context_compliance, crescendo_history_lecture,\n", " crescendo_journalist_interview, crescendo_movie_director,\n", " crescendo_simulated, flip, many_shot, red_teaming,\n", " role_play_movie_script, role_play_persuasion,\n", " role_play_persuasion_written, role_play_trivia_game,\n", " role_play_video_game, tap, first_letter, image\n", " Default Technique: default\n", " Default Datasets (1):\n", " airt_leakage\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " airt.psychosocial\u001b[0m\n", " Class: Psychosocial\n", " Description:\n", " Psychosocial Harms scenario covering the imminent-crisis and\n", " licensed-therapist sub-harms. Evaluates whether a model harms the\n", " well-being or mental health of users -- for example by mistreating\n", " someone in a mental-health crisis, facilitating self-harm, or improperly\n", " presenting itself as a real licensed therapist. **Two axes.** The\n", " primary axis is ``sub_harm`` (``imminent_crisis`` and/or\n", " ``licensed_therapist``; both by default). Each sub-harm owns its\n", " dataset, its escalation prompt, and its own conversation-level scorer,\n", " so every attack and baseline is scored by the rubric that matches its\n", " harm. The secondary axis is the ``PsychosocialTechnique`` converter\n", " sweep, selected with ``--techniques``. **The base technique is a\n", " simulated crescendo.** For each sub-harm the scenario builds an\n", " escalating simulated conversation (via\n", " ``AttackTechniqueFactory.with_simulated_conversation`` using that\n", " sub-harm's escalation prompt) and delivers the final message to the\n", " target. Each selected converter is layered on top of that base; the live\n", " multi-turn ``Crescendo`` technique (``all`` only) swaps the simulated\n", " base for a real ``CrescendoAttack``. One baseline per sub-harm is\n", " emitted (toggle with ``include_baseline``). Dataset selection is bound\n", " to the sub-harms: the ``dataset_config`` parameter still tunes\n", " ``max_dataset_size`` and sampling, but the dataset names are always the\n", " selected sub-harms' datasets (``--dataset-names`` is ignored).\n", " Aggregate Techniques:\n", " - all, default, tone, language, persuasion, deterministic\n", " Available Techniques (24):\n", " none, tone_soften, tone_upset, tone_angry, tone_sad, tone_urgent,\n", " language_spanish, language_french, language_german, language_japanese,\n", " persuasion_logical_appeal, persuasion_authority_endorsement,\n", " persuasion_evidence_based, persuasion_expert_endorsement,\n", " persuasion_misrepresentation, tense_past, variation, noise,\n", " insert_punctuation, random_capitalization, diacritic, char_swap,\n", " colloquial_wordswap, crescendo\n", " Default Technique: default\n", " Default Datasets (2):\n", " airt_imminent_crisis, airt_licensed_therapist\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", " - sub_harm (str) [default: 'all']: Psychosocial sub-harm to run: 'imminent_crisis', 'licensed_therapist', or 'all'. Defaults to 'all'.\n", " - max_turns (int) [default: '5']: Number of turns in the simulated-crescendo escalation for each attack.\n", "\u001b[1m\u001b[36m\n", " airt.rapid_response\u001b[0m\n", " Class: RapidResponse\n", " Description:\n", " Rapid Response scenario for content-harms testing. Tests model behavior\n", " across multiple harm categories using selectable attack techniques.\n", " Aggregate Techniques:\n", " - all, default, core, light, multi_turn, single_turn\n", " Available Techniques (14):\n", " context_compliance, crescendo_history_lecture,\n", " crescendo_journalist_interview, crescendo_movie_director,\n", " crescendo_simulated, flip, many_shot, red_teaming,\n", " role_play_movie_script, role_play_persuasion,\n", " role_play_persuasion_written, role_play_trivia_game,\n", " role_play_video_game, tap\n", " Default Technique: default\n", " Default Datasets (7):\n", " airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,\n", " airt_misinformation, airt_leakage\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " airt.scam\u001b[0m\n", " Class: Scam\n", " Description:\n", " Scam scenario evaluates an endpoint's ability to generate scam-related\n", " materials (e.g., phishing emails, fraudulent messages) with primarily\n", " persuasion-oriented techniques.\n", " Aggregate Techniques:\n", " - all, default, single_turn, multi_turn\n", " Available Techniques (3):\n", " context_compliance, role_play_persuasion_written, persuasive_rta\n", " Default Technique: default\n", " Default Datasets (1):\n", " airt_scams\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", " - max_turns (int) [default: '5']: Maximum conversation turns for the persuasive_rta technique.\n", "\u001b[1m\u001b[36m\n", " benchmark.adversarial\u001b[0m\n", " Class: AdversarialBenchmark\n", " Description:\n", " Benchmark scenario that compares the attack success rate (ASR) across\n", " adversarial models. Adversarial targets are user-supplied via the\n", " ``adversarial_targets`` parameter (declared in\n", " ``supported_parameters``). Each target must already be registered in\n", " ``TargetRegistry`` — typically by ``TargetInitializer`` from\n", " ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via\n", " ``TargetRegistry.get_registry_singleton().instances.register``. At run\n", " time, ``_build_atomic_attacks_async`` performs the ``(technique ×\n", " adversarial_target × dataset)`` cross-product: for each selected\n", " adversarial-capable factory in the ``AttackTechniqueRegistry`` and each\n", " requested target, it calls ``factory.create(adversarial_chat=...)`` with\n", " the resolved target — no global registry mutation. The resulting\n", " ``AtomicAttack`` is named ``f\"{technique}__{target}_{dataset}\"`` with\n", " ``display_group`` set to the target's registry name so per-model ASR\n", " rolls up naturally in result displays.\n", " Aggregate Techniques:\n", " - all, default, core, light, multi_turn, single_turn\n", " Available Techniques (12):\n", " context_compliance, crescendo_history_lecture,\n", " crescendo_journalist_interview, crescendo_movie_director,\n", " crescendo_simulated, red_teaming, role_play_movie_script,\n", " role_play_persuasion, role_play_persuasion_written,\n", " role_play_trivia_game, role_play_video_game, tap\n", " Default Technique: default\n", " Default Datasets (1):\n", " harmbench\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", " - adversarial_targets (list[str]): Registry names of adversarial chat targets to benchmark. Each name must already be registered in TargetRegistry (via TargetInitializer or TargetRegistry instance registration). Use 'pyrit_scan list-targets' to see registered targets. Settable via --adversarial-targets [ ...] on the CLI, or scenario.args.adversarial_targets in .pyrit_conf.\n", "\u001b[1m\u001b[36m\n", " foundry.red_team_agent\u001b[0m\n", " Class: RedTeamAgent\n", " Description:\n", " RedTeamAgent is a preconfigured scenario that automatically generates\n", " multiple AtomicAttack instances based on the specified attack\n", " techniques. It supports both single-turn attacks (with various\n", " converters) and multi-turn attacks (Crescendo, RedTeaming), making it\n", " easy to quickly test a target against multiple attack vectors. The\n", " scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into\n", " their constituent attack techniques, or you can specify individual\n", " techniques directly. This scenario is designed for use with the Foundry\n", " AI Red Teaming Agent library, providing a consistent PyRIT contract for\n", " their integration.\n", " Aggregate Techniques:\n", " - all, easy, moderate, difficult\n", " Available Techniques (25):\n", " ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,\n", " character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,\n", " suffix_append, string_join, unicode_confusable, unicode_substitution,\n", " url, jailbreak, tense, multi_turn, crescendo, pair, tap\n", " Default Technique: easy\n", " Default Datasets (1):\n", " harmbench\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " garak.doctor\u001b[0m\n", " Class: Doctor\n", " Description:\n", " Doctor scenario implementation for PyRIT. Ports Garak's \"doctor\" probe,\n", " which implements the HiddenLayer Policy Puppetry universal bypass: a\n", " post-instruction-hierarchy prompt injection that frames a harmful\n", " request as policy/configuration (a fictional medical-drama script or a\n", " healthcare-chatbot interaction config) to override safety guardrails.\n", " Each objective from the ``garak_doctor`` dataset is wrapped in a Policy\n", " Puppetry template via ``PolicyPuppetryConverter`` and sent to the\n", " target. Responses are scored for non-refusal (mirroring Garak's\n", " ``mitigation.MitigationBypass`` detector). Reference:\n", " [@hiddenlayer2025policypuppetry]\n", " (https://hiddenlayer.com/innovation-hub/novel-universal-bypass-for-all-major-llms/)\n", " Aggregate Techniques:\n", " - all, default, single_turn\n", " Available Techniques (2):\n", " policy_puppetry, policy_puppetry_leet\n", " Default Technique: default\n", " Default Datasets (1):\n", " garak_doctor\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " garak.encoding\u001b[0m\n", " Class: Encoding\n", " Description:\n", " Encoding Scenario implementation for PyRIT. This scenario tests how\n", " resilient models are to various encoding attacks by encoding potentially\n", " harmful text (by default slurs and XSS payloads) and testing if the\n", " model will decode and repeat the encoded payload. It mimics the Garak\n", " encoding probe. The scenario works by: 1. Taking seed prompts (the\n", " harmful text to be encoded) 2. Encoding them using various encoding\n", " schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to\n", " decode the encoded text 4. Scoring whether the model successfully\n", " decoded and repeated the harmful content By default, this uses the same\n", " dataset as Garak: slur terms and web XSS payloads.\n", " Aggregate Techniques:\n", " - all, default\n", " Available Techniques (17):\n", " base64, base2048, base16, base32, ascii85, hex, quoted_printable,\n", " uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,\n", " leet_speak, ascii_smuggler\n", " Default Technique: default\n", " Default Datasets (2):\n", " garak_slur_terms_en, garak_web_html_js\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\u001b[1m\u001b[36m\n", " garak.web_injection\u001b[0m\n", " Class: WebInjection\n", " Description:\n", " Web Injection Scenario implementation for PyRIT. Ports garak's\n", " ``web_injection`` probe. It tests whether a model can be coaxed into\n", " emitting web-injection payloads — markdown-image data exfiltration\n", " (CSRF-style) and cross-site scripting (XSS) — when given\n", " prompt-injection style instructions. Each technique builds one\n", " ``PromptSendingAttack`` whose seeds pair a ``SeedObjective`` (the attack\n", " goal) with a ``SeedPrompt`` (the literal injection prompt to send).\n", " Exfil techniques are scored with ``MarkdownInjectionScorer``; XSS\n", " techniques are scored with ``XSSOutputScorer``. The default objective\n", " scorer (used for the baseline and metadata) is an OR composite of both.\n", " Aggregate Techniques:\n", " - all, default, exfil, xss\n", " Available Techniques (8):\n", " markdown_image_exfil, colab_ai_data_leakage, string_assembly_data_exfil,\n", " playground_markdown_exfil, markdown_uri_image_exfil_extended,\n", " markdown_uri_non_image_exfil_extended, task_xss, markdown_xss\n", " Default Technique: default\n", " Default Datasets (4):\n", " garak_example_domains_xss, garak_markdown_js, garak_web_html_js,\n", " garak_xss_normal_instructions\n", " Supported Parameters:\n", " - objective_target (any): Target system under attack: a registered target name or a PromptTarget instance.\n", " - scenario_techniques (any): Techniques to execute; defaults to the scenario's default aggregate when omitted.\n", " - technique_converters (any): Mapping of concrete technique name to extra request converters to append.\n", " - dataset_config (any): Dataset source configuration; defaults to the scenario's default when omitted.\n", " - memory_labels (any): Additional labels applied to every attack run in the scenario.\n", " - max_concurrency (int) [default: '4']: Maximum number of concurrent units of work for the scenario.\n", " - max_retries (int) [default: '0']: Maximum number of automatic retries if the scenario raises an exception.\n", " - include_baseline (bool): Whether to prepend a baseline atomic attack; None defers to BASELINE_ATTACK_POLICY.\n", "\n", "================================================================================\n", "\n", "Total scenarios: 12\n" ] } ], "source": [ "!pyrit_scan list-scenarios" ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ "**Tip**: You can also surface user-defined scenarios. List your initializer script in the\n", "`initialization_scripts` section of the config file the backend loads (see [here](../getting_started/pyrit_conf.md)).\n", "The backend runs those scripts at startup and auto-discovers any `Scenario` subclasses they\n", "define, so start the server with that config, then list:\n", "\n", "```shell\n", "pyrit_scan --config-file ./my_pyrit_conf.yaml start-server\n", "pyrit_scan list-scenarios\n", "```\n", "\n", "## Initializers\n", "\n", "PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the `--initializers` flag.\n", "\n", "The `--list-initializers` command shows all available initializers. Initializers are referenced by their filename (e.g., `target`, `scorer`) regardless of which subdirectory they're in.\n", "\n", "List the available initializers using the --list-initializers flag." ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Available Initializers:\n", "================================================================================\n", "\u001b[1m\u001b[36m\n", " load_default_datasets\u001b[0m\n", " Class: LoadDefaultDatasets\n", " Required Environment Variables: None\n", " Supported Parameters:\n", " - dataset_names: Explicit dataset names to load. Overrides the scenario-default selection.\n", " - tags: Load datasets whose metadata matches these tags. Overrides scenario-default selection.\n", " Description:\n", " Load datasets into memory so scenarios can run.\n", "\u001b[1m\u001b[36m\n", " preload_scenario_metadata\u001b[0m\n", " Class: PreloadScenarioMetadata\n", " Required Environment Variables: None\n", " Description:\n", " Instantiate every registered scenario once to warm the metadata cache.\n", "\u001b[1m\u001b[36m\n", " refresh_datasets\u001b[0m\n", " Class: RefreshDatasets\n", " Required Environment Variables: None\n", " Supported Parameters:\n", " - days [default: 30]: Refresh only datasets whose newest seed is older than this many days. 0 refreshes every selected dataset regardless of age.\n", " - dataset_names: Explicit dataset names to refresh; refreshes all in-memory datasets if omitted.\n", " Description:\n", " Refresh datasets already loaded in memory from their registered\n", " providers.\n", "\u001b[1m\u001b[36m\n", " scorer\u001b[0m\n", " Class: ScorerInitializer\n", " Required Environment Variables: None\n", " Supported Parameters:\n", " - tags [default: ['default']]: Tags for filtering (e.g., ['default'])\n", " Description:\n", " Instantiates a collection of scorers using targets from the\n", " TargetRegistry and adds them to the ScorerRegistry.\n", "\u001b[1m\u001b[36m\n", " target\u001b[0m\n", " Class: TargetInitializer\n", " Required Environment Variables: None\n", " Supported Parameters:\n", " - tags [default: ['default']]: Target tags to register (e.g., ['default'], ['default', 'scorer'], or ['all'])\n", " - auto_group [default: True]: Auto-create round-robin groups from targets with matching behavioral eval params\n", " Description:\n", " Target Initializer for registering pre-configured targets.\n", "\u001b[1m\u001b[36m\n", " technique\u001b[0m\n", " Class: TechniqueInitializer\n", " Required Environment Variables: None\n", " Supported Parameters:\n", " - tags [default: ['core']]: Technique groups to register (e.g., ['core'], ['core', 'extra'], or ['all'])\n", " Description:\n", " Register scenario attack technique factories into the\n", " AttackTechniqueRegistry.\n", "\n", "================================================================================\n", "\n", "Total initializers: 6\n" ] } ], "source": [ "!pyrit_scan list-initializers" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "### Running Scenarios\n", "\n", "You need a single scenario to run, you need two things:\n", "\n", "1. A Scenario. Many are defined in `pyrit.scenario.scenarios`. But you can also define your own in initialization_scripts.\n", "2. Initializers (which can be supplied via the `--initializers` flag on `run`, or the `initializers` / `initialization_scripts` sections of a config file (see [here](../getting_started/pyrit_conf.md))). Scenarios often don't need many arguments, but they can be configured in different ways. And at the very least, most need an `objective_target` (the thing you're running a scan against) which you can configure by using the `--target` flag if your initializer registers targets (e.g. `target` initializer)\n", "3. Scenario Techniques (optional). These are supplied by the `--techniques` flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running `--list-scenarios`\n", "\n", "Basic usage will look something like:\n", "\n", "```shell\n", "pyrit_scan run --target --initializers --techniques \n", "```\n", "\n", "You can also override scenario parameters directly from the CLI:\n", "\n", "```shell\n", "pyrit_scan run --max-concurrency 10 --max-retries 3 --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\"}'\n", "```\n", "\n", "Or concretely:\n", "\n", "```shell\n", "!pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques base64\n", "```\n", "\n", "Example with a basic configuration that runs the Foundry scenario against the objective target defined in the `target` initializer." ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Running scenario: foundry.red_team_agent\n", "\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] techniques: 0/1 (0%) | success rate: 0% | IN_PROGRESS\n", " [██████████████████████████████] techniques: 2/2 (100%) | success rate: 0% | IN_PROGRESS\n", " [██████████████████████████████] techniques: 2/2 (100%) | success rate: 0% | COMPLETED\n", "\u001b[36m====================================================================================================\u001b[0m\n", "\u001b[1m\u001b[36m 📊 SCENARIO RESULTS: RedTeamAgent \u001b[0m\n", "\u001b[36m====================================================================================================\u001b[0m\n", "\n", "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n", "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m 📋 Scenario Details\u001b[0m\n", "\u001b[36m • Name: RedTeamAgent\u001b[0m\n", "\u001b[36m • Result ID: 10840c79-567c-4ecd-a3c8-fb7e9349bd57\u001b[0m\n", "\u001b[36m • Scenario Version: 1\u001b[0m\n", "\u001b[36m • PyRIT Version: 1.1.0.dev0\u001b[0m\n", "\u001b[36m • Description:\u001b[0m\n", "\u001b[36m RedTeamAgent is a preconfigured scenario that automatically generates multiple AtomicAttack instances based on\u001b[0m\n", "\u001b[36m the specified attack techniques. It supports both single-turn attacks (with various converters) and multi-turn\u001b[0m\n", "\u001b[36m attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors. The\u001b[0m\n", "\u001b[36m scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack techniques, or\u001b[0m\n", "\u001b[36m you can specify individual techniques directly. This scenario is designed for use with the Foundry AI Red\u001b[0m\n", "\u001b[36m Teaming Agent library, providing a consistent PyRIT contract for their integration.\u001b[0m\n", "\n", "\u001b[1m 🎯 Target Information\u001b[0m\n", "\u001b[36m • Target Type: OpenAIChatTarget\u001b[0m\n", "\u001b[36m • Target Model: gpt-4o\u001b[0m\n", "\u001b[36m • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n", "\n", "\u001b[1m 📊 Scorer Information\u001b[0m\n", "\u001b[37m ▸ Scorer Identifier\u001b[0m\n", "\u001b[36m • Scorer Type: FloatScaleThresholdScorer\u001b[0m\n", "\u001b[36m • scorer_type: true_false\u001b[0m\n", "\u001b[36m • score_aggregator: OR_\u001b[0m\n", "\u001b[36m └─ Composite of 1 scorer(s):\u001b[0m\n", "\u001b[36m • Scorer Type: AzureContentFilterScorer\u001b[0m\n", "\u001b[36m • scorer_type: float_scale\u001b[0m\n", "\n", "\u001b[37m ▸ Performance Metrics\u001b[0m\n", "\u001b[31m • Accuracy: 59.24%\u001b[0m\n", "\u001b[36m • Accuracy Std Error: ±0.0247\u001b[0m\n", "\u001b[31m • F1 Score: 0.5306\u001b[0m\n", "\u001b[31m • Precision: 0.5987\u001b[0m\n", "\u001b[31m • Recall: 0.4764\u001b[0m\n", "\u001b[32m • Average Score Time: 0.04s\u001b[0m\n", "\n", "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n", "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\u001b[1m 📈 Summary\u001b[0m\n", "\u001b[32m • Total Techniques: 2\u001b[0m\n", "\u001b[32m • Total Attack Results: 8\u001b[0m\n", "\u001b[32m • Overall Success Rate: 0%\u001b[0m\n", "\u001b[32m • Unique Objectives: 4\u001b[0m\n", "\n", "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n", "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n", "\n", "\u001b[1m 🔸 Group: base64\u001b[0m\n", "\u001b[33m • Number of Results: 4\u001b[0m\n", "\u001b[32m • Success Rate: 0%\u001b[0m\n", "\n", "\u001b[1m 🔸 Group: baseline\u001b[0m\n", "\u001b[33m • Number of Results: 4\u001b[0m\n", "\u001b[32m • Success Rate: 0%\u001b[0m\n", "\n", "\u001b[36m====================================================================================================\u001b[0m\n", "\n" ] } ], "source": [ "!pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques base64" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "Or with all options and multiple techniques:\n", "\n", "```shell\n", "pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --techniques easy crescendo\n", "```\n", "\n", "You can also override scenario execution parameters:\n", "\n", "```shell\n", "# Override concurrency and retry settings\n", "pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --max-concurrency 10 --max-retries 3\n", "\n", "# Add custom memory labels for tracking (must be valid JSON)\n", "pyrit_scan run foundry.red_team_agent --target openai_chat --initializers target --memory-labels '{\"experiment\": \"test1\", \"version\": \"v2\", \"researcher\": \"alice\"}'\n", "```\n", "\n", "Available CLI parameter overrides:\n", "- `--max-concurrency `: Maximum number of concurrent attack executions\n", "- `--max-retries `: Maximum number of automatic retries if the scenario raises an exception\n", "- `--memory-labels `: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)\n", "\n", "Dataset-backed scenarios can also select a supported dataset. Requested datasets are fetched\n", "on demand, so a full dataset preload is not required:\n", "\n", "```shell\n", "pyrit_scan run garak.figstep --target openai_chat --dataset-names figstep_pro --max-dataset-size 1\n", "```\n", "\n", "Custom initialization scripts are loaded by the backend at startup: list them in the\n", "`initialization_scripts` section of the config the server loads (paths are relative to your\n", "working directory, but full paths avoid confusion). Once the server is running with that\n", "config, they apply to every `run`:\n", "\n", "\n", "```shell\n", "pyrit_scan --config-file ./my_pyrit_conf.yaml start-server\n", "pyrit_scan run garak.encoding\n", "```" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "#### Attaching Converters to a Technique\n", "\n", "Techniques (techniques) can have a registered converter instance appended to them with the\n", "`:converter.` syntax. The converter is added to the request side of every attack\n", "the technique produces, on top of any converters the technique already bakes in. This also works on\n", "aggregate techniques (the converter is applied to every technique the aggregate expands to).\n", "\n", "First discover the registered converter instances with `list-converters`. Converters are\n", "registered by initializers that the backend runs at startup, so the initializer that registers\n", "them must be part of the server's configuration — a built-in in the `initializers` section, or a\n", "custom script in the `initialization_scripts` section of the config the server loads (see\n", "[here](../getting_started/pyrit_conf.md)). Start the server with that config, then list:\n", "\n", "```shell\n", "pyrit_scan --config-file ./my_pyrit_conf.yaml start-server\n", "pyrit_scan list-converters\n", "```\n", "\n", "Then reference a converter by name in `--techniques`:\n", "\n", "```shell\n", "# Add the registered \"translation_spanish\" converter to role_play_movie_script only\n", "pyrit_scan run airt.rapid_response --target openai_chat --initializers target my_converters --techniques role_play_movie_script:converter.translation_spanish\n", "\n", "```\n", "\n", "#### Chain multiple converters (applied in order) and combine with plain techniques\n", "```shell\n", "pyrit_scan run airt.rapid_response --target openai_chat --initializers load_default_datasets target my_converters --techniques role_play_movie_script:converter.translation_spanish:converter.base64 many_shot\n", "```" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "#### Using Custom Scenarios\n", "\n", "You can define your own scenarios in initialization scripts. The CLI will automatically discover any `Scenario` subclasses and make them available:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": { "lines_to_next_cell": 2 }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[pyrit:alembic] No new upgrade operations detected.\n" ] }, { "data": { "text/plain": [ "<__main__.MyCustomScenario at 0x1d2391a9d30>" ] }, "execution_count": null, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# my_custom_scenarios.py\n", "\n", "from pyrit.common import apply_defaults\n", "from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget\n", "from pyrit.scenario import DatasetAttackConfiguration, Scenario, ScenarioTechnique\n", "from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer\n", "from pyrit.setup import initialize_pyrit_async\n", "\n", "\n", "class MyCustomTechnique(ScenarioTechnique):\n", " \"\"\"Techniques for my custom scenario.\"\"\"\n", "\n", " ALL = (\"all\", {\"all\"})\n", " Technique1 = (\"technique1\", set[str]())\n", " Technique2 = (\"technique2\", set[str]())\n", "\n", "\n", "class MyCustomScenario(Scenario):\n", " \"\"\"My custom scenario that does XYZ.\"\"\"\n", "\n", " @apply_defaults\n", " def __init__(self, *, scenario_result_id=None, **kwargs):\n", " # Scenario-specific configuration only - no runtime parameters\n", " super().__init__(\n", " name=\"My Custom Scenario\",\n", " version=1,\n", " objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),\n", " technique_class=MyCustomTechnique,\n", " default_dataset_config=DatasetAttackConfiguration(dataset_names=[\"harmbench\"]),\n", " scenario_result_id=scenario_result_id,\n", " )\n", " # ... your scenario-specific initialization code\n", "\n", " async def _build_atomic_attacks_async(self, *, context):\n", " # The single abstract extension point every scenario implements.\n", " # Read runtime inputs from `context`; return the list of AtomicAttack to run.\n", " # Matrix-shaped scenarios can delegate to build_matrix_atomic_attacks(context=...).\n", " # Example: create attacks for each technique composite\n", " return []\n", "\n", "\n", "await initialize_pyrit_async(memory_db_type=\"InMemory\") # type: ignore\n", "MyCustomScenario()" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "Then discover and run it:\n", "\n", "```shell\n", "# Start the backend with a config whose initialization_scripts lists my_custom_scenarios.py\n", "pyrit_scan --config-file ./my_pyrit_conf.yaml start-server\n", "\n", "# List to confirm it's available\n", "pyrit_scan list-scenarios\n", "\n", "# Run it with parameter overrides\n", "pyrit_scan run my_custom_scenario --max-concurrency 10\n", "\n", "```The scenario name is automatically converted from the class name (e.g., `MyCustomScenario` becomes `my_custom_scenario`).\n" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "## Inspecting Results\n", "\n", "Each run is saved under a `scenario_result_id` (printed when the run finishes), so you can revisit it later without re-running. Use `scenario-history` to find recent run ids:\n", "\n", "```shell\n", "pyrit_scan scenario-history # last 10 runs\n", "pyrit_scan scenario-history 25 # last 25\n", "```\n", "\n", "Then inspect a run with `scenario-results` at the granularity you want via `--view`:\n", "\n", "```shell\n", "# Aggregate stats + per-group success rates (the default; same as the run summary)\n", "pyrit_scan scenario-results \n", "\n", "# One row per attack: id, technique, objective, outcome, turns, score\n", "pyrit_scan scenario-results --view attacks\n", "\n", "# Full message transcripts with the objective score per turn\n", "pyrit_scan scenario-results --view conversations\n", "\n", "# Both: the attacks table followed by the transcripts\n", "pyrit_scan scenario-results --view full\n", "```\n", "\n", "`--view` picks *how much* detail per attack; `--attack-result-ids` picks *which* attacks (ids come from the `attacks` view); the two combine. Use `--limit` to cap how many attacks are shown; the `conversations` and `full` views default to 5 when you scope neither:\n", "\n", "```shell\n", "pyrit_scan scenario-results --view conversations --attack-result-ids \n", "pyrit_scan scenario-results --view conversations --limit 3\n", "```" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "## Stopping the Backend Server\n", "\n", "When you're done, stop the local backend that we started at the top of the notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Server on port 8000 stopped.\n" ] } ], "source": [ "!pyrit_scan stop-server" ] } ], "metadata": { "jupytext": { "main_language": "python" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.4" } }, "nbformat": 4, "nbformat_minor": 5 }