{ "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Target Capabilities\n", "\n", "Every `PromptTarget` carries a `TargetConfiguration` that declares what it natively supports, what to do\n", "when a capability is missing, and how to adapt the conversation when adaptation is permitted. This notebook\n", "walks through how to inspect, validate, and override capabilities on a real target — the same machinery\n", "attacks, scorers, and converters use under the hood.\n", "\n", "A `TargetConfiguration` composes three concerns:\n", "\n", "* **`TargetCapabilities`** — declarative, immutable description of what the target natively supports.\n", "* **`CapabilityHandlingPolicy`** — for each adaptable capability, whether to `ADAPT` (run a normalizer)\n", " or `RAISE` (fail immediately) when the target lacks it.\n", "* **`ConversationNormalizationPipeline`** — the ordered set of normalizers derived from the gap between\n", " the declared capabilities and the policy.\n", "\n", "See [Target Capabilities](./0_prompt_targets.md#target-capabilities) in the overview for the full list\n", "of capability flags." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## 1. Inspect a real target's configuration\n", "\n", "We use `OpenAIChatTarget` throughout this notebook. Constructing the target does not make any network\n", "calls — we are only inspecting its declared configuration." ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n", "Loaded environment file: ./.pyrit/.env\n", "Loaded environment file: ./.pyrit/.env.local\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "No new upgrade operations detected.\n", "supports_multi_turn: True\n", "supports_editable_history: True\n", "supports_system_prompt: True\n", "supports_json_output: True\n", "supports_json_schema: False\n", "input_modalities: [['image_path'], ['image_path', 'text'], ['text']]\n", "output_modalities: [['text']]\n" ] } ], "source": [ "from pyrit.prompt_target import OpenAIChatTarget\n", "from pyrit.setup import IN_MEMORY, initialize_pyrit_async\n", "\n", "await initialize_pyrit_async(memory_db_type=IN_MEMORY) # type: ignore\n", "\n", "target = OpenAIChatTarget(model_name=\"gpt-4o\", endpoint=\"https://example.invalid/\", api_key=\"sk-not-a-real-key\")\n", "caps = target.configuration.capabilities\n", "\n", "print(\"supports_multi_turn: \", caps.supports_multi_turn)\n", "print(\"supports_editable_history: \", caps.supports_editable_history)\n", "print(\"supports_system_prompt: \", caps.supports_system_prompt)\n", "print(\"supports_json_output: \", caps.supports_json_output)\n", "print(\"supports_json_schema: \", caps.supports_json_schema)\n", "print(\"input_modalities: \", sorted(sorted(m) for m in caps.input_modalities))\n", "print(\"output_modalities: \", sorted(sorted(m) for m in caps.output_modalities))" ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## 2. Default configurations and known model profiles\n", "\n", "Each target class declares a `_DEFAULT_CONFIGURATION` class attribute. For well-known underlying models,\n", "`get_default_configuration(underlying_model=...)` returns a richer profile from\n", "`get_known_capabilities` — for example, `gpt-5` gains `supports_json_schema=True`\n", "and other models pick up the right modality combinations automatically. Unknown models fall back to\n", "the class default." ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "capability class default gpt-4o gpt-5 unknown \n", "--------------------------------------------------------------------------------\n", "supports_multi_turn True True True True \n", "supports_editable_history True True True True \n", "supports_system_prompt True True True True \n", "supports_json_output True True True True \n", "supports_json_schema False False True False \n" ] } ], "source": [ "class_default = OpenAIChatTarget._DEFAULT_CONFIGURATION.capabilities\n", "gpt_4o = OpenAIChatTarget.get_default_configuration(underlying_model=\"gpt-4o\").capabilities\n", "gpt_5 = OpenAIChatTarget.get_default_configuration(underlying_model=\"gpt-5\").capabilities\n", "unknown = OpenAIChatTarget.get_default_configuration(underlying_model=\"not-a-real-model\").capabilities\n", "\n", "print(f\"{'capability':<32}{'class default':<18}{'gpt-4o':<10}{'gpt-5':<10}{'unknown':<10}\")\n", "print(\"-\" * 80)\n", "for flag in (\n", " \"supports_multi_turn\",\n", " \"supports_editable_history\",\n", " \"supports_system_prompt\",\n", " \"supports_json_output\",\n", " \"supports_json_schema\",\n", "):\n", " row = (\n", " f\"{flag:<32}\"\n", " f\"{str(getattr(class_default, flag)):<18}\"\n", " f\"{str(getattr(gpt_4o, flag)):<10}\"\n", " f\"{str(getattr(gpt_5, flag)):<10}\"\n", " f\"{str(getattr(unknown, flag)):<10}\"\n", " )\n", " print(row)" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## 3. Declare and validate consumer requirements\n", "\n", "Components that need particular capabilities declare them as a `TargetRequirements` and validate at\n", "construction time. PyRIT ships a `CHAT_TARGET_REQUIREMENTS` constant for the common case of needing\n", "multi-turn + editable history — the replacement for the former `PromptChatTarget` type check.\n", "\n", "`TargetRequirements.validate` collects every missing capability and raises a single `ValueError` so\n", "callers see all violations at once.\n", "\n", "`TargetRequirements` can also enforce **modality** constraints via `required_input_modalities` and\n", "`required_output_modalities`. Each entry is a set of `PromptDataType` values the consumer needs\n", "the target to accept (or produce). At least one of the target's modality combos must be a superset\n", "of each required combo." ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "OpenAIChatTarget satisfies CHAT_TARGET_REQUIREMENTS\n" ] } ], "source": [ "from pyrit.prompt_target import CHAT_TARGET_REQUIREMENTS\n", "\n", "CHAT_TARGET_REQUIREMENTS.validate(target=target)\n", "print(\"OpenAIChatTarget satisfies CHAT_TARGET_REQUIREMENTS\")" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "To check a single capability, call `target.configuration.ensure_can_handle(capability=...)` directly." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Multi-turn check passed\n" ] } ], "source": [ "from pyrit.prompt_target.common.target_capabilities import CapabilityName\n", "\n", "target.configuration.ensure_can_handle(capability=CapabilityName.MULTI_TURN)\n", "print(\"Multi-turn check passed\")" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## 4. Override the configuration per instance\n", "\n", "For targets whose capabilities depend on deployment (HTTP endpoints, Playwright UIs, custom backends —\n", "or simply an OpenAI-compatible model whose actual capabilities differ from `gpt-4o`), pass a\n", "`TargetConfiguration` via `custom_configuration`. The instance uses your override instead of the class\n", "default." ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "class default supports_multi_turn: True\n", "instance supports_multi_turn: False\n", "\n", "Validation failed as expected:\n", "Target does not satisfy 2 required capability(ies):\n", " - Target does not support 'supports_editable_history' and no handling policy exists for it.\n", " - Target does not support 'supports_multi_turn' and the handling policy is RAISE.\n" ] } ], "source": [ "from pyrit.prompt_target.common.target_capabilities import TargetCapabilities\n", "from pyrit.prompt_target.common.target_configuration import TargetConfiguration\n", "\n", "restricted_config = TargetConfiguration(\n", " capabilities=TargetCapabilities(\n", " supports_multi_turn=False,\n", " supports_system_prompt=False,\n", " supports_multi_message_pieces=True,\n", " ),\n", ")\n", "restricted_target = OpenAIChatTarget(\n", " model_name=\"custom-model\",\n", " endpoint=\"https://example.invalid/\",\n", " api_key=\"sk-not-a-real-key\",\n", " custom_configuration=restricted_config,\n", ")\n", "\n", "print(\"class default supports_multi_turn: \", class_default.supports_multi_turn)\n", "print(\"instance supports_multi_turn: \", restricted_target.configuration.capabilities.supports_multi_turn)\n", "\n", "try:\n", " CHAT_TARGET_REQUIREMENTS.validate(target=restricted_target)\n", "except ValueError as exc:\n", " print(\"\\nValidation failed as expected:\")\n", " print(exc)" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "## 5. ADAPT vs RAISE\n", "\n", "When a capability is missing, the `CapabilityHandlingPolicy` decides what happens. Only *adaptable*\n", "capabilities (currently `MULTI_TURN` and `SYSTEM_PROMPT`) can be papered over by PyRIT — for these,\n", "you can switch the behavior from `RAISE` (default) to `ADAPT`. With `ADAPT`, the conversation goes\n", "through a normalizer that flattens history or merges system prompts before reaching the target.\n", "\n", "Below we wrap a single-turn endpoint two ways and watch the pipeline change. Note that the `RAISE`\n", "pipeline is **empty**: when a missing capability is configured to raise, there is nothing to\n", "normalize. The error surfaces later, when a consumer calls `ensure_can_handle` or\n", "`TargetRequirements.validate`." ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "RAISE pipeline normalizers: []\n", "ADAPT pipeline normalizers: ['GenericSystemSquashNormalizer', 'HistorySquashNormalizer']\n" ] } ], "source": [ "from pyrit.prompt_target.common.target_capabilities import (\n", " CapabilityHandlingPolicy,\n", " UnsupportedCapabilityBehavior,\n", ")\n", "\n", "single_turn_caps = TargetCapabilities(supports_multi_turn=False, supports_system_prompt=False)\n", "\n", "raise_config = TargetConfiguration(\n", " capabilities=single_turn_caps,\n", " policy=CapabilityHandlingPolicy(\n", " behaviors={\n", " CapabilityName.MULTI_TURN: UnsupportedCapabilityBehavior.RAISE,\n", " CapabilityName.SYSTEM_PROMPT: UnsupportedCapabilityBehavior.RAISE,\n", " }\n", " ),\n", ")\n", "adapt_config = TargetConfiguration(\n", " capabilities=single_turn_caps,\n", " policy=CapabilityHandlingPolicy(\n", " behaviors={\n", " CapabilityName.MULTI_TURN: UnsupportedCapabilityBehavior.ADAPT,\n", " CapabilityName.SYSTEM_PROMPT: UnsupportedCapabilityBehavior.ADAPT,\n", " }\n", " ),\n", ")\n", "\n", "raise_target = OpenAIChatTarget(\n", " model_name=\"custom-model\",\n", " endpoint=\"https://example.invalid/\",\n", " api_key=\"sk-not-a-real-key\",\n", " custom_configuration=raise_config,\n", ")\n", "adapt_target = OpenAIChatTarget(\n", " model_name=\"custom-model\",\n", " endpoint=\"https://example.invalid/\",\n", " api_key=\"sk-not-a-real-key\",\n", " custom_configuration=adapt_config,\n", ")\n", "\n", "print(\"RAISE pipeline normalizers: \", [type(n).__name__ for n in raise_target.configuration.pipeline._normalizers])\n", "print(\"ADAPT pipeline normalizers: \", [type(n).__name__ for n in adapt_target.configuration.pipeline._normalizers])" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "With `ADAPT`, running a multi-turn conversation through `normalize_async` collapses it into a single\n", "user message — the exact payload the target will see when a prompt is sent." ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "original turns: 3\n", "normalized turns: 1\n", "flattened text:\n", "[Conversation History]\n", "User: What is the capital of France?\n", "Assistant: Paris.\n", "\n", "[Current Message]\n", "And of Germany?\n" ] } ], "source": [ "from pyrit.models import Message\n", "\n", "conversation = [\n", " Message.from_prompt(prompt=\"What is the capital of France?\", role=\"user\"),\n", " Message.from_prompt(prompt=\"Paris.\", role=\"assistant\"),\n", " Message.from_prompt(prompt=\"And of Germany?\", role=\"user\"),\n", "]\n", "\n", "normalized = await adapt_target.configuration.normalize_async(messages=conversation) # type: ignore\n", "print(f\"original turns: {len(conversation)}\")\n", "print(f\"normalized turns: {len(normalized)}\")\n", "print(\"flattened text:\")\n", "print(normalized[-1].message_pieces[0].original_value)" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "By contrast, the `RAISE` configuration validates eagerly: any consumer requiring `MULTI_TURN` will\n", "get a `ValueError` before a single prompt is sent." ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Target does not support 'supports_multi_turn' and the handling policy is RAISE.\n" ] } ], "source": [ "try:\n", " raise_target.configuration.ensure_can_handle(capability=CapabilityName.MULTI_TURN)\n", "except ValueError as exc:\n", " print(exc)" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## 6. Non-adaptable capabilities\n", "\n", "Some capabilities cannot be safely emulated — for example, `supports_editable_history` is a property\n", "of the underlying API contract and there is no normalizer that can fake it. These capabilities are\n", "not represented in the `CapabilityHandlingPolicy` at all; requesting them on a target that lacks\n", "them always raises, regardless of policy." ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Target does not support 'supports_editable_history' and no handling policy exists for it.\n" ] } ], "source": [ "no_editable_history = TargetConfiguration(\n", " capabilities=TargetCapabilities(supports_multi_turn=True, supports_editable_history=False),\n", ")\n", "\n", "try:\n", " no_editable_history.ensure_can_handle(capability=CapabilityName.EDITABLE_HISTORY)\n", "except ValueError as exc:\n", " print(exc)" ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "## 7. Discovering live target capabilities\n", "\n", "Declared capabilities describe what a target *should* support. For deployments where the actual\n", "behavior is uncertain — custom OpenAI-compatible endpoints, gateways that strip features, models\n", "whose support drifts over time — you can probe what the target *actually* accepts at runtime with\n", "`discover_target_capabilities_async`. It runs both the boolean capability probes and the input\n", "modality probes and returns a best-effort `TargetCapabilities`.\n", "\n", "Internally it walks each capability that has a registered probe (currently\n", "`SYSTEM_PROMPT`, `MULTI_MESSAGE_PIECES`, `MULTI_TURN`, `JSON_OUTPUT`, `JSON_SCHEMA`), sends a\n", "minimal request, and includes the capability in the result only if the call succeeds.\n", "During probing the target's configuration is temporarily replaced with a permissive one so\n", "`ensure_can_handle` does not short-circuit a probe for a capability the target declares as\n", "unsupported. The original configuration is restored before the function returns. The same\n", "treatment is applied to each input modality combination declared in\n", "`capabilities.input_modalities`, sending a small payload built from optional `test_assets`.\n", "\n", "Each probe call is bounded by `per_probe_timeout_s` (default 30s) and is retried once on\n", "transient errors before being declared failed. The returned `TargetCapabilities` is a merged\n", "view: probed where possible, declared where probing is unavailable or out of scope.\n", "\"Supported\" here means *the request was accepted* — a target that silently ignores a system\n", "prompt or `response_format` directive will still be reported as supporting that capability.\n", "\n", "This function is **not safe to call concurrently** with other operations on the same target\n", "instance: it temporarily mutates `target._configuration` and writes probe rows to\n", "`target._memory`. Probe-written memory rows are tagged with\n", "`prompt_metadata[\"capability_probe\"] == \"1\"` so consumers can filter them.\n", "\n", "Typical usage against a real endpoint:\n", "\n", "```python\n", "from pyrit.prompt_target import discover_target_capabilities_async\n", "\n", "queried = await discover_target_capabilities_async(target=target)\n", "print(queried)\n", "```\n", "\n", "Below we mock the target's underlying transport (`_send_prompt_to_target_async`) so the notebook\n", "stays self-contained — the result shape is the same as a live run. We mock the protected method\n", "rather than `send_prompt_async` so the probe still exercises the real validation and memory\n", "pipeline." ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "discover_target_capabilities_async result:\n", " supports_multi_turn: True\n", " supports_system_prompt: True\n", " supports_multi_message_pieces: True\n", " supports_json_output: True\n", " supports_json_schema: True\n", " input_modalities: [['image_path'], ['image_path', 'text'], ['text']]\n" ] } ], "source": [ "from unittest.mock import AsyncMock\n", "\n", "from pyrit.models import MessagePiece\n", "from pyrit.prompt_target import discover_target_capabilities_async\n", "\n", "\n", "def _ok_response():\n", " return [\n", " Message(\n", " message_pieces=[\n", " MessagePiece(\n", " role=\"assistant\",\n", " original_value=\"ok\",\n", " original_value_data_type=\"text\",\n", " conversation_id=\"probe\",\n", " response_error=\"none\",\n", " )\n", " ]\n", " )\n", " ]\n", "\n", "\n", "probe_target = OpenAIChatTarget(model_name=\"gpt-4o\", endpoint=\"https://example.invalid/\", api_key=\"sk-not-a-real-key\")\n", "probe_target._send_prompt_to_target_async = AsyncMock(return_value=_ok_response()) # type: ignore[method-assign]\n", "\n", "queried = await discover_target_capabilities_async(target=probe_target, per_probe_timeout_s=5.0) # type: ignore\n", "print(\"discover_target_capabilities_async result:\")\n", "print(f\" supports_multi_turn: {queried.supports_multi_turn}\")\n", "print(f\" supports_system_prompt: {queried.supports_system_prompt}\")\n", "print(f\" supports_multi_message_pieces: {queried.supports_multi_message_pieces}\")\n", "print(f\" supports_json_output: {queried.supports_json_output}\")\n", "print(f\" supports_json_schema: {queried.supports_json_schema}\")\n", "print(f\" input_modalities: {sorted(sorted(m) for m in queried.input_modalities)}\")" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "To narrow the probe to specific capabilities (faster, fewer calls), pass `capabilities=`:\n", "\n", "```python\n", "from pyrit.prompt_target.common.target_capabilities import CapabilityName\n", "\n", "queried = await discover_target_capabilities_async(\n", " target=target,\n", " capabilities=[CapabilityName.JSON_SCHEMA, CapabilityName.SYSTEM_PROMPT],\n", ")\n", "```\n", "\n", "Similarly, narrow the modality probe set with `test_modalities=` and override the\n", "packaged default probe assets with `test_assets=`." ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "### Discovering undeclared modalities\n", "\n", "By default `discover_target_capabilities_async` only probes modality combinations the target already\n", "**declares** in `capabilities.input_modalities`. For an OpenAI-compatible endpoint that\n", "claims text-only but might actually accept images, pass `test_modalities=` explicitly to\n", "probe combinations beyond the declared baseline. Provide `test_assets=` as well if you need\n", "to override the packaged defaults or probe a modality without one:\n", "\n", "```python\n", "queried = await discover_target_capabilities_async(\n", " target=target,\n", " test_modalities={frozenset({\"text\"}), frozenset({\"text\", \"image_path\"})},\n", " test_assets={\"image_path\": \"/path/to/test_image.png\"},\n", ")\n", "```\n", "\n", "Similarly, when narrowing the probe set with `capabilities=`, capabilities NOT in the\n", "narrowed set are copied from the target's declared values rather than being reset to\n", "`False` — narrowing controls *what is re-queried*, not what the returned dataclass\n", "reports. This makes incremental probing safe:\n", "\n", "```python\n", "# Re-query only JSON support; other declared flags pass through unchanged.\n", "queried = await discover_target_capabilities_async(\n", " target=target,\n", " capabilities={CapabilityName.JSON_OUTPUT, CapabilityName.JSON_SCHEMA},\n", ")\n", "```" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "## 8. Applying probed capabilities back onto the target\n", "\n", "`discover_target_capabilities_async` is intentionally pure: it returns a `TargetCapabilities` without\n", "mutating the target. That lets you inspect (or diff against the declared view, log, gate on\n", "the result) before committing. Once you're satisfied, call `target.apply_capabilities(...)`\n", "to install the probed view on the instance. The target's existing\n", "`CapabilityHandlingPolicy` is preserved — policy expresses user intent (ADAPT vs RAISE),\n", "which is independent of what the probe found.\n", "\n", "Why a two-step pattern rather than auto-apply? Probe results are an upper bound\n", "(\"the request was accepted\"); a target that silently ignores a feature still passes its\n", "probe. Keeping discovery separate from application lets callers diff, log, persist, or\n", "reject the result before it affects subsequent sends.\n", "\n", "Below is the end-to-end pattern: construct a target whose declared capabilities are\n", "pessimistic, discover what the endpoint actually accepts, diff the two views, then apply." ] }, { "cell_type": "code", "execution_count": null, "id": "24", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "declared (before probing):\n", " supports_multi_turn: False\n", " supports_system_prompt: False\n", " supports_json_output: False\n", "\n", "probed (returned from discover_target_capabilities_async, target NOT yet updated):\n", " supports_multi_turn: True\n", " supports_system_prompt: True\n", " supports_json_output: True\n", " target.capabilities.supports_multi_turn (still declared): False\n", "\n", "flags probed True that were declared False: ['supports_multi_turn', 'supports_system_prompt', 'supports_multi_message_pieces', 'supports_json_output', 'supports_json_schema']\n", "\n", "after apply_capabilities:\n", " supports_multi_turn: True\n", " supports_system_prompt: True\n", " supports_json_output: True\n", " policy preserved: True\n", "\n", "CHAT_TARGET_REQUIREMENTS.validate now passes against the probed target\n" ] } ], "source": [ "# Start with an instance that declares fewer capabilities than the endpoint actually has,\n", "# e.g. a custom gateway whose support we're unsure about.\n", "pessimistic_config = TargetConfiguration(\n", " capabilities=TargetCapabilities(\n", " supports_multi_turn=False,\n", " supports_system_prompt=False,\n", " supports_multi_message_pieces=False,\n", " supports_json_output=False,\n", " supports_json_schema=False,\n", " # Editable history has no live probe and falls back to the declared value.\n", " # Declare it True here so the probed view inherits it.\n", " supports_editable_history=True,\n", " ),\n", ")\n", "endpoint_target = OpenAIChatTarget(\n", " model_name=\"custom-model\",\n", " endpoint=\"https://example.invalid/\",\n", " api_key=\"sk-not-a-real-key\",\n", " custom_configuration=pessimistic_config,\n", ")\n", "endpoint_target._send_prompt_to_target_async = AsyncMock(return_value=_ok_response()) # type: ignore[method-assign]\n", "\n", "print(\"declared (before probing):\")\n", "print(f\" supports_multi_turn: {endpoint_target.capabilities.supports_multi_turn}\")\n", "print(f\" supports_system_prompt: {endpoint_target.capabilities.supports_system_prompt}\")\n", "print(f\" supports_json_output: {endpoint_target.capabilities.supports_json_output}\")\n", "\n", "# Step 1: discover. No mutation yet — `endpoint_target.capabilities` is unchanged.\n", "probed_caps = await discover_target_capabilities_async(target=endpoint_target, per_probe_timeout_s=5.0) # type: ignore\n", "\n", "print(\"\\nprobed (returned from discover_target_capabilities_async, target NOT yet updated):\")\n", "print(f\" supports_multi_turn: {probed_caps.supports_multi_turn}\")\n", "print(f\" supports_system_prompt: {probed_caps.supports_system_prompt}\")\n", "print(f\" supports_json_output: {probed_caps.supports_json_output}\")\n", "print(f\" target.capabilities.supports_multi_turn (still declared): {endpoint_target.capabilities.supports_multi_turn}\")\n", "\n", "# Step 2: diff — see exactly what the probe upgraded.\n", "declared = pessimistic_config.capabilities\n", "upgraded = [\n", " name\n", " for name in (\n", " \"supports_multi_turn\",\n", " \"supports_system_prompt\",\n", " \"supports_multi_message_pieces\",\n", " \"supports_json_output\",\n", " \"supports_json_schema\",\n", " )\n", " if getattr(probed_caps, name) and not getattr(declared, name)\n", "]\n", "print(f\"\\nflags probed True that were declared False: {upgraded}\")\n", "\n", "# Step 3: apply. Policy is preserved; the normalization pipeline is rebuilt.\n", "original_policy = endpoint_target.configuration.policy\n", "endpoint_target.apply_capabilities(capabilities=probed_caps)\n", "\n", "print(\"\\nafter apply_capabilities:\")\n", "print(f\" supports_multi_turn: {endpoint_target.capabilities.supports_multi_turn}\")\n", "print(f\" supports_system_prompt: {endpoint_target.capabilities.supports_system_prompt}\")\n", "print(f\" supports_json_output: {endpoint_target.capabilities.supports_json_output}\")\n", "print(f\" policy preserved: {endpoint_target.configuration.policy is original_policy}\")\n", "\n", "# Subsequent consumer checks now reflect the probed reality — for example, a chat-style\n", "# requirement that would have failed against the pessimistic declaration now passes.\n", "CHAT_TARGET_REQUIREMENTS.validate(target=endpoint_target)\n", "print(\"\\nCHAT_TARGET_REQUIREMENTS.validate now passes against the probed target\")" ] } ], "metadata": { "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.12" } }, "nbformat": 4, "nbformat_minor": 5 }