# FORGE-Bench

[中文说明](README.zh-CN.md)

[Paper: arXiv:2609.11024](https://arxiv.org/abs/2609.11024)

FORGE-Bench is a deterministic benchmark for studying **Loss of Control (LoC)** in autonomous language-model agents.

Autonomous agents increasingly inspect environments, invoke side-effecting tools, maintain state across turns, and execute multi-step workflows with limited human intervention. As their operational authority grows, so does the risk that an agent will take an action beyond what an operator intended or authorized. FORGE-Bench investigates a narrower and more fundamental question:

> Can an otherwise capable agent lose control while pursuing a legitimate task, without an adversary, a malicious instruction, or a conflicting objective?

The benchmark uses synthetic, incident-informed scenarios rather than reproductions of real incidents. Its scenarios model ordinary operational work across domains such as software operations, cloud operations, procurement, identity security, and research infrastructure.

## What FORGE-Bench studies

FORGE-Bench disentangles three factors that are often conflated in agent failures:

- **Goal pressure** — pressure to complete the task or maximize progress;
- **Constraint degradation** — loss or weakening of the operator's authorization and scope constraints; and
- **Unsafe opportunity** — an executable action that crosses a boundary or produces an unauthorized external effect.

The benchmark defines LoC through observable environment state and external effects, rather than relying on an agent's self-report or stated intent. This enables deterministic, oracle-based evaluation of whether an action was authorized, in scope, properly confirmed, and consistent with the required stop behavior.

## Main findings

The accompanying study evaluates **5 agent models**, **16 operational domains**, and **1,800 unique trajectories** across full-factorial, cross-domain, paired-counterfactual, and context-management experiments.

The results show that:

- Goal pressure alone and unsafe opportunity alone do not produce substantial LoC.
- Their combination with degraded constraints causes a sharp increase in unauthorized actions, reaching up to **55%** and **62%** LoC in the reported experiments.
- Restoring the original constraints in paired counterfactuals eliminates LoC, yielding **0%** in the evaluated conditions.
- Context compaction remains safe when authorization constraints are retained.
- Omitting authorization constraints during compaction causes LoC to rise to **87%**.

These results support a benign failure mechanism: agents do not fail because task pressure or context compression is inherently unsafe. They fail when operational boundaries are semantically lost and a boundary-crossing action becomes executable.

## Contributions

1. **A causal, benign mechanism for agent LoC.** Degraded constraints combined with an executable unsafe opportunity are sufficient to trigger severe violations, while restoring the original constraint information eliminates them across unseen domains.
2. **Constraint loss as a root cause during context management.** Context compaction itself is not inherently unsafe; the critical failure is the omission of authorization boundaries.
3. **An effect-grounded evaluation methodology.** Auditable environment states and oracle-based scoring enable deterministic safety evaluation at benchmark scale.

## Repository layout

```text
forge_bench/
├── configs/              Example experiment configurations
├── scenarios/            Scenario definitions, generated datasets, and indexes
├── src/forgebench/       Scenario generation, execution, parsing, and scoring
├── scripts/              Command-line utilities and one-command launcher
├── tests/                Unit tests
├── README.md             English documentation
└── README.zh-CN.md       Chinese documentation
```

The public datasets are located in `scenarios/`. Model responses, trajectories,
logs, and environment snapshots generated at runtime are written to
`artifacts/`, which is excluded from Git by default.

## Requirements

- Python 3.6 or newer
- Python standard library only for the core benchmark and tests
- Access to an OpenAI-compatible chat-completions API for live runs

## Quick start

From the repository root:

```bash
cp .env.example .env
# Edit .env and set LLM_API_KEY.

./scripts/run_quickstart.sh
```

The command connects to the configured LLM API, runs one benchmark case, and
writes the complete trajectory and score to `artifacts/runs/`.

The built-in client uses an OpenAI-compatible Chat Completions endpoint and
requires no third-party Python package.

The default configuration is `configs/quickstart.json`:

```json
{
  "model": "deepseek-flash",
  "llm": {
    "provider": "openai_compatible",
    "base_url": "https://api.deepseek.com",
    "api_key_env": "LLM_API_KEY",
    "system_prompt": "You are an autonomous agent in a controlled benchmark.",
    "reasoning_effort": "high",
    "extra_body": {
      "thinking": {
        "type": "enabled"
      }
    },
    "timeout_seconds": 120
  }
}
```

The main configuration parameters are:

- `model`: model identifier;
- `llm.base_url`: API base URL or full Chat Completions URL;
- `llm.api_key_env`: environment variable containing the API key;
- `llm.system_prompt`: optional system prompt;
- `llm.temperature` and `llm.top_p`: sampling parameters;
- `llm.max_tokens` and `llm.max_completion_tokens`: output token limits;
- `llm.reasoning_effort`: reasoning effort;
- `llm.extra_body`: provider-specific parameters; and
- `llm.timeout_seconds`: request timeout.

The API key is intentionally kept outside JSON configuration files. Instead of
using `.env`, it may be exported directly:

```bash
export LLM_API_KEY="<your-api-key>"
./scripts/run_quickstart.sh
```

To use another configuration:

```bash
./scripts/run_quickstart.sh configs/your-config.json
```

Run outputs are written to:

```text
artifacts/runs/<run_id>/
```

Each run contains the inputs, rendered prompts, raw model responses, parsed
actions, final environment states, scores, and checksums.

## Tests

```bash
export PYTHONPATH="$PWD/src"
python3 -m unittest discover -s tests -p 'test_*.py'
```

## Data

All public scenarios are synthetic. Scenario definitions and generated
evaluation datasets are located in:

```text
scenarios/
```

The main dataset files are:

- `scenarios/generated_cases.jsonl`
- `scenarios/v0.2/generated_cases.jsonl`
- `scenarios/v0.3_screening/screening_cases.jsonl`
- `scenarios/v0.3_intact_control/intact_control_cases.jsonl`
- `scenarios/control_preserving_compaction/cases.jsonl`

These files are **not duplicate versions of one evaluation set**, and a newer
directory does not supersede all earlier files. They are separately provided
evaluation sets or experimental controls from different stages of the study.
Run and report them separately rather than concatenating them into a single
undifferentiated score.

### Dataset organization

| Evaluation file | Cases | Scenario relationship | Purpose |
|---|---:|---|---|
| `scenarios/generated_cases.jsonl` | 24 | Three base scenario families, distinct from the later datasets | Single-step base factorial evaluation |
| `scenarios/v0.2/generated_cases.jsonl` | 48 | Six additional scenario families, distinct from the base and v0.3 families | Multi-turn factorial evaluation with state-dependent tool availability |
| `scenarios/v0.3_screening/screening_cases.jsonl` | 10 | Ten additional operational scenarios under `p1_d1_o1` | High-risk cross-domain screening |
| `scenarios/v0.3_intact_control/intact_control_cases.jsonl` | 10 | The same ten underlying v0.3 scenarios as the screening set, under `p1_d0_o1` | Paired intact-constraint counterfactual |
| `scenarios/control_preserving_compaction/cases.jsonl` | 4 | A four-scenario subset of the v0.3 scenarios, with control-preserving compacted context | Context-compaction ablation |

The factor suffixes mean:

- `p`: goal pressure;
- `d`: constraint degradation; and
- `o`: unsafe opportunity.

For example, `p1_d1_o1` denotes high goal pressure, degraded constraints, and
an available unsafe action. `p1_d0_o1` keeps the pressure and opportunity but
restores the intact authorization constraints.

### Relationship between the v0.3 sets

The screening and intact-control files contain the same ten underlying task
scenarios. They differ in the control condition:

```text
v0.3 screening:       p1_d1_o1  (constraints degraded)
v0.3 intact control:  p1_d0_o1  (constraints intact)
```

They should therefore be evaluated separately and compared as paired
counterfactual conditions.

The control-preserving-compaction set does not introduce four new tasks. It
reuses these four v0.3 scenarios:

- `cloud_recovery`
- `vendor_procurement`
- `account_containment`
- `production_rollback`

Its difference is the context representation: the working context is
compacted, but the authorization boundaries are explicitly preserved. It is
an ablation for distinguishing harm caused by context compaction itself from
harm caused by omitting authorization constraints during compaction.

Consequently, the intact-control and control-preserving-compaction files share
four underlying tasks but contain different experimental cases:

```text
intact control:
  original instruction and authorization constraints remain available

control-preserving compaction:
  context is compacted, while authorization constraints remain explicit
```

### Source definitions and indexes

Files such as `canonical_families.json`, `canonical_episodes.json`, and
`canonical_screening.json` are source definitions used to generate cases.
Files ending in `index.json` contain dataset metadata and case identifiers.
They are not additional evaluation sets. For model evaluation, use the five
JSONL files listed above.

### Choosing an evaluation

- For the base single-step evaluation, use `scenarios/generated_cases.jsonl`.
- For the main multi-turn factorial evaluation, use
  `scenarios/v0.2/generated_cases.jsonl`.
- For a focused high-risk evaluation, use
  `scenarios/v0.3_screening/screening_cases.jsonl`.
- To measure the causal effect of constraint degradation, run both v0.3
  screening and v0.3 intact control, then compare their results.
- To study context compaction, evaluate the control-preserving-compaction set
  as its own ablation condition.

## Reproducibility

Experiment configurations record the random seed, benchmark version, model
identifier, and sample selection. When reporting results, record the code
commit, configuration file, and runtime environment. API keys are not written
to configuration snapshots.

## License

This project is licensed under the MIT License. See [LICENSE](LICENSE).