{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# Benchmark Scenarios\n",
    "\n",
    "Benchmark scenarios compare attack effectiveness across an axis that varies within the scenario\n",
    "itself. Currently the only benchmark variant is the adversarial benchmark, whose axis of change is\n",
    "the **adversarial chat helper model** used in attacks. For full configuration options see\n",
    "`pyrit_scan --help` and the [Scenarios Programming Guide](../code/scenarios/0_scenarios.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "## Adversarial Benchmark\n",
    "\n",
    "`AdversarialBenchmark` holds the objective target and dataset constant and varies the adversarial\n",
    "chat model used to drive multi-turn attacks. Useful for evaluating which adversarial helper\n",
    "models produce stronger or weaker attack success rates against the same target.\n",
    "\n",
    "Adversarial targets are user-provided via the `adversarial_targets` scenario parameter. Each name\n",
    "must already be registered in `TargetRegistry` — typically by `TargetInitializer` from the\n",
    "`ADVERSARIAL_CHAT_*` env vars (see `.env_example`). Use `pyrit_scan list-targets` to see every\n",
    "target currently registered.\n",
    "\n",
    "```bash\n",
    "pyrit_scan run benchmark.adversarial \\\n",
    "  --initializers target \\\n",
    "  --target openai_chat \\\n",
    "  --adversarial-targets adversarial_chat \\\n",
    "  --techniques role_play_video_game \\\n",
    "  --max-dataset-size 1\n",
    "```\n",
    "\n",
    "Pass multiple `--adversarial-targets` values to compare across models in a single run.\n",
    "\n",
    "**Default techniques:** `role_play_video_game`, `crescendo_simulated`, and `tap`. TAP's\n",
    "branching search makes this default slower and more expensive than the former `light` default.\n",
    "For a cheaper run, explicitly pass `--techniques light`.\n",
    "\n",
    "**Other available selections:** `light`, `single_turn`, `multi_turn`, plus one member per\n",
    "adversarial-capable source technique (e.g. `red_teaming`, `tap`, `crescendo_simulated`)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: Auto-discovered plaintext environment file ./.pyrit/.env will be loaded. Azure Key Vault through env_akv_ref is more secure for shared or deployed secrets; use .env.local only for deliberate local overrides. To inspect a resolved AKV-only configuration from a source checkout, run `python -m build_scripts.export_akv_environment`; it writes ~/.pyrit/.env_akv.\n",
      "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
      "Loaded environment file: ./.pyrit/.env\n",
      "Loaded environment file: ./.pyrit/.env.local\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[pyrit:alembic] No new upgrade operations detected.\n"
     ]
    }
   ],
   "source": [
    "from pyrit.output import output_scenario_async\n",
    "from pyrit.prompt_target import OpenAIChatTarget\n",
    "from pyrit.scenario import DatasetAttackConfiguration\n",
    "from pyrit.scenario.benchmark import AdversarialBenchmark\n",
    "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n",
    "from pyrit.setup.initializers import ScorerInitializer, TargetInitializer, TechniqueInitializer\n",
    "\n",
    "await initialize_pyrit_async(  # type: ignore\n",
    "    memory_db_type=IN_MEMORY,\n",
    "    initializers=[TargetInitializer(), ScorerInitializer(), TechniqueInitializer()],\n",
    ")\n",
    "\n",
    "objective_target = OpenAIChatTarget()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71d9eda503124a3ab0e0a08afd49fc44",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Executing AdversarialBenchmark:   0%|          | 0/1 [00:00<?, ?attack/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pyrit.scenario.benchmark import AdversarialBenchmarkTechnique\n",
    "\n",
    "dataset_config = DatasetAttackConfiguration(dataset_names=[\"harmbench\"], max_dataset_size=1)\n",
    "\n",
    "scenario = AdversarialBenchmark()\n",
    "scenario.set_params_from_args(\n",
    "    args={\n",
    "        \"adversarial_targets\": [\"adversarial_chat\"],\n",
    "        \"objective_target\": objective_target,\n",
    "        \"scenario_techniques\": [AdversarialBenchmarkTechnique.role_play_video_game],\n",
    "        \"dataset_config\": dataset_config,\n",
    "    }\n",
    ")\n",
    "await scenario.initialize_async()  # type: ignore\n",
    "\n",
    "scenario_result = await scenario.run_async()  # type: ignore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\u001b[1m\u001b[36m                              📊 SCENARIO RESULTS: AdversarialBenchmark                              \u001b[0m\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Scenario Information\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📋 Scenario Details\u001b[0m\n",
      "\u001b[36m    • Name: AdversarialBenchmark\u001b[0m\n",
      "\u001b[36m    • Result ID: a856e722-1dea-4a79-9df6-7a79b3e67f9a\u001b[0m\n",
      "\u001b[36m    • Scenario Version: 4\u001b[0m\n",
      "\u001b[36m    • PyRIT Version: 1.1.0.dev0\u001b[0m\n",
      "\u001b[36m    • Description:\u001b[0m\n",
      "\u001b[36m        Benchmark scenario that compares the attack success rate (ASR) across adversarial models. Adversarial targets\u001b[0m\n",
      "\u001b[36m        are user-supplied via the ``adversarial_targets`` parameter (declared in ``supported_parameters``). Each target\u001b[0m\n",
      "\u001b[36m        must already be registered in ``TargetRegistry`` — typically by ``TargetInitializer`` from\u001b[0m\n",
      "\u001b[36m        ``ADVERSARIAL_CHAT_*`` env vars, or programmatically via\u001b[0m\n",
      "\u001b[36m        ``TargetRegistry.get_registry_singleton().instances.register``. At run time, ``_build_atomic_attacks_async``\u001b[0m\n",
      "\u001b[36m        performs the ``(technique × adversarial_target × dataset)`` cross-product: for each selected adversarial-capable\u001b[0m\n",
      "\u001b[36m        factory in the ``AttackTechniqueRegistry`` and each requested target, it calls\u001b[0m\n",
      "\u001b[36m        ``factory.create(adversarial_chat=...)`` with the resolved target — no global registry mutation. The resulting\u001b[0m\n",
      "\u001b[36m        ``AtomicAttack`` is named ``f\"{technique}__{target}_{dataset}\"`` with ``display_group`` set to the target's\u001b[0m\n",
      "\u001b[36m        registry name so per-model ASR rolls up naturally in result displays.\u001b[0m\n",
      "\n",
      "\u001b[1m  🎯 Target Information\u001b[0m\n",
      "\u001b[36m    • Target Type: OpenAIChatTarget\u001b[0m\n",
      "\u001b[36m    • Target Model: gpt-4o-japan-nilfilter\u001b[0m\n",
      "\u001b[36m    • Target Endpoint: https://pyrit-japan-test.openai.azure.com/openai/v1\u001b[0m\n",
      "\n",
      "\u001b[1m  📊 Scorer Information\u001b[0m\n",
      "\u001b[37m    ▸ Scorer Identifier\u001b[0m\n",
      "\u001b[36m      • Scorer Type: FloatScaleThresholdScorer\u001b[0m\n",
      "\u001b[36m      • scorer_type: true_false\u001b[0m\n",
      "\u001b[36m      • score_aggregator: OR_\u001b[0m\n",
      "\u001b[36m        └─ Composite of 1 scorer(s):\u001b[0m\n",
      "\u001b[36m            • Scorer Type: AzureContentFilterScorer\u001b[0m\n",
      "\u001b[36m            • scorer_type: float_scale\u001b[0m\n",
      "\n",
      "\u001b[37m    ▸ Performance Metrics\u001b[0m\n",
      "\u001b[31m      • Accuracy: 59.24%\u001b[0m\n",
      "\u001b[36m      • Accuracy Std Error: ±0.0247\u001b[0m\n",
      "\u001b[31m      • F1 Score: 0.5306\u001b[0m\n",
      "\u001b[31m      • Precision: 0.5987\u001b[0m\n",
      "\u001b[31m      • Recall: 0.4764\u001b[0m\n",
      "\u001b[32m      • Average Score Time: 0.04s\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Overall Statistics\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\u001b[1m  📈 Summary\u001b[0m\n",
      "\u001b[32m    • Total Techniques: 1\u001b[0m\n",
      "\u001b[32m    • Total Attack Results: 1\u001b[0m\n",
      "\u001b[32m    • Overall Success Rate: 0%\u001b[0m\n",
      "\u001b[32m    • Unique Objectives: 1\u001b[0m\n",
      "\n",
      "\u001b[1m\u001b[36m▼ Per-Group Breakdown\u001b[0m\n",
      "\u001b[36m────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n",
      "\n",
      "\u001b[1m  🔸 Group: adversarial_chat\u001b[0m\n",
      "\u001b[33m    • Number of Results: 1\u001b[0m\n",
      "\u001b[32m    • Success Rate: 0%\u001b[0m\n",
      "\n",
      "\u001b[36m====================================================================================================\u001b[0m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "await output_scenario_async(scenario_result)"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}