Skip to content

MAINT BREAK: Move include_baseline from Scenario constructor to initi… #1700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adrian-gavrila wants to merge 3 commits into
base: main
Choose a base branch
from
Open

MAINT BREAK: Move include_baseline from Scenario constructor to initi… #1700

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a21dbbb
MAINT BREAK: Move include_baseline from Scenario constructor to initi…
May 8, 2026
7b2f327
adding tests for additional classVar
May 9, 2026
fe9cf55
doc updates
May 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 60 additions & 41 deletions doc/code/scenarios/0_scenarios.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,6 @@
" - `version`: Integer version number\n",
" - `strategy_class`: The strategy enum class for this scenario\n",
" - `objective_scorer_identifier`: Identifier dict for the scoring mechanism (optional)\n",
" - `include_default_baseline`: Whether to include a baseline attack (default: True)\n",
" - `scenario_result_id`: Optional ID to resume an existing scenario (optional)\n",
"\n",
"5. **Initialization**: Call `await scenario.initialize_async()` to populate atomic attacks:\n",
Expand All @@ -83,6 +82,8 @@
" - `max_concurrency`: Number of concurrent operations (default: 1)\n",
" - `max_retries`: Number of retry attempts on failure (default: 0)\n",
" - `memory_labels`: Optional labels for tracking (optional)\n",
" - `include_baseline`: Whether to prepend a baseline attack (defaults to the scenario type's\n",
" `DEFAULT_INCLUDE_BASELINE`; most scenarios default it on, `Jailbreak` defaults it off)\n",
"\n",
"### Example Structure\n",
"\n",
Expand All @@ -101,9 +102,15 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
"Loaded environment file: ./.pyrit/.env\n",
"Loaded environment file: ./.pyrit/.env.local\n"
"Found default environment files: ['./.pyrit/.env']\n",
"Loaded environment file: ./.pyrit/.env\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"No new upgrade operations detected.\n"
]
}
],
Expand Down Expand Up @@ -193,34 +200,13 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']\n",
"Loading default configuration file: ./.pyrit/.pyrit_conf\n",
"Found default environment files: ['./.pyrit/.env']\n",
"Loaded environment file: ./.pyrit/.env\n",
"Loaded environment file: ./.pyrit/.env.local\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Available Scenarios:\n",
"================================================================================\n",
"\u001b[1m\u001b[36m\n",
" airt.content_harms\u001b[0m\n",
" Class: ContentHarms\n",
" Description:\n",
" Content Harms Scenario implementation for PyRIT. This scenario contains\n",
" various harm-based checks that you can run to get a quick idea about\n",
" model behavior with respect to certain harm categories.\n",
" Aggregate Strategies:\n",
" - all\n",
" Available Strategies (7):\n",
" hate, fairness, violence, sexual, harassment, misinformation, leakage\n",
" Default Strategy: all\n",
" Default Datasets (7, max 4 per dataset):\n",
" airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,\n",
" airt_misinformation, airt_leakage\n",
"\u001b[1m\u001b[36m\n",
" airt.cyber\u001b[0m\n",
" Class: Cyber\n",
" Description:\n",
Expand All @@ -229,9 +215,9 @@
" Cyber class contains different variations of the malware generation\n",
" techniques.\n",
" Aggregate Strategies:\n",
" - all\n",
" - all, single_turn, multi_turn\n",
" Available Strategies (2):\n",
" single_turn, multi_turn\n",
" prompt_sending, red_teaming\n",
" Default Strategy: all\n",
" Default Datasets (1, max 4 per dataset):\n",
" airt_malware\n",
Expand All @@ -256,14 +242,14 @@
" Description:\n",
" Leakage scenario implementation for PyRIT. This scenario tests how\n",
" susceptible models are to leaking training data, PII, intellectual\n",
" property, or other confidential information. The Leakage class\n",
" contains different attack variations designed to extract sensitive\n",
" information from models.\n",
" property, or other confidential information. Uses the registry/factory\n",
" pattern to construct attack techniques.\n",
" Aggregate Strategies:\n",
" - all, single_turn, multi_turn, ip, sensitive_data\n",
" Available Strategies (4):\n",
" first_letter, image, role_play, crescendo\n",
" Default Strategy: all\n",
" - all, default, single_turn, multi_turn\n",
" Available Strategies (9):\n",
" prompt_sending, role_play, many_shot, tap, crescendo_simulated,\n",
" red_teaming, context_compliance, first_letter, image\n",
" Default Strategy: default\n",
" Default Datasets (1, max 4 per dataset):\n",
" airt_leakage\n",
"\u001b[1m\u001b[36m\n",
Expand Down Expand Up @@ -296,6 +282,21 @@
" Default Datasets (1, max 4 per dataset):\n",
" airt_imminent_crisis\n",
"\u001b[1m\u001b[36m\n",
" airt.rapid_response\u001b[0m\n",
" Class: RapidResponse\n",
" Description:\n",
" Rapid Response scenario for content-harms testing. Tests model behavior\n",
" across multiple harm categories using selectable attack techniques.\n",
" Aggregate Strategies:\n",
" - all, default, single_turn, multi_turn\n",
" Available Strategies (7):\n",
" prompt_sending, role_play, many_shot, tap, crescendo_simulated,\n",
" red_teaming, context_compliance\n",
" Default Strategy: default\n",
" Default Datasets (7, max 4 per dataset):\n",
" airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,\n",
" airt_misinformation, airt_leakage\n",
"\u001b[1m\u001b[36m\n",
" airt.scam\u001b[0m\n",
" Class: Scam\n",
" Description:\n",
Expand All @@ -309,6 +310,21 @@
" Default Strategy: all\n",
" Default Datasets (1, max 4 per dataset):\n",
" airt_scams\n",
" Supported Parameters:\n",
" - max_turns (int) [default: 5]: Maximum conversation turns for the persuasive_rta strategy.\n",
"\u001b[1m\u001b[36m\n",
" benchmark.adversarial\u001b[0m\n",
" Class: AdversarialBenchmark\n",
" Description:\n",
" Benchmarking scenario that compares the attack success rate (ASR) of\n",
" several different adversarial models.\n",
" Aggregate Strategies:\n",
" - all, default, single_turn, multi_turn, light\n",
" Available Strategies (4):\n",
" role_play, tap, red_teaming, context_compliance\n",
" Default Strategy: light\n",
" Default Datasets (1, max 8 per dataset):\n",
" harmbench\n",
"\u001b[1m\u001b[36m\n",
" foundry.red_team_agent\u001b[0m\n",
" Class: RedTeamAgent\n",
Expand Down Expand Up @@ -359,7 +375,7 @@
"\n",
"================================================================================\n",
"\n",
"Total scenarios: 8\n"
"Total scenarios: 9\n"
]
},
{
Expand Down Expand Up @@ -389,10 +405,13 @@
"\n",
"Every scenario can optionally include a **baseline attack** — a `PromptSendingAttack` that sends\n",
"each objective directly to the target without any converters or multi-turn techniques. This is\n",
"controlled by the `include_default_baseline` parameter (default: `True` for most scenarios).\n",
"controlled by the `include_baseline` parameter on `initialize_async`; when omitted, each\n",
"scenario falls back to its own `DEFAULT_INCLUDE_BASELINE` class attribute (most scenarios\n",
"default it on; `Jailbreak` defaults it off).\n",
"\n",
"To run *only* the baseline (no attack strategies), create a `RedTeamAgent` with\n",
"`include_baseline=True` (the default) and pass `scenario_strategies=None`. See\n",
"To run *only* the baseline (no attack strategies), pass `include_baseline=True` and\n",
"`scenario_strategies=None` explicitly to `initialize_async` — that way the example works\n",
"regardless of the scenario's default. See\n",
"[Common Scenario Parameters](./1_common_scenario_parameters.ipynb) for a working example."
]
},
Expand Down Expand Up @@ -436,7 +455,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.15"
"version": "3.12.13"
}
},
"nbformat": 4,
Expand Down
12 changes: 8 additions & 4 deletions doc/code/scenarios/0_scenarios.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@
# - `version`: Integer version number
# - `strategy_class`: The strategy enum class for this scenario
# - `objective_scorer_identifier`: Identifier dict for the scoring mechanism (optional)
# - `include_default_baseline`: Whether to include a baseline attack (default: True)
# - `scenario_result_id`: Optional ID to resume an existing scenario (optional)
#
# 5. **Initialization**: Call `await scenario.initialize_async()` to populate atomic attacks:
Expand All @@ -85,6 +84,8 @@
# - `max_concurrency`: Number of concurrent operations (default: 1)
# - `max_retries`: Number of retry attempts on failure (default: 0)
# - `memory_labels`: Optional labels for tracking (optional)
# - `include_baseline`: Whether to prepend a baseline attack (defaults to the scenario type's
# `DEFAULT_INCLUDE_BASELINE`; most scenarios default it on, `Jailbreak` defaults it off)
#
# ### Example Structure
#
Expand Down Expand Up @@ -174,10 +175,13 @@ def _build_display_group(self, *, technique_name: str, seed_group_name: str) ->
#
# Every scenario can optionally include a **baseline attack** — a `PromptSendingAttack` that sends
# each objective directly to the target without any converters or multi-turn techniques. This is
# controlled by the `include_default_baseline` parameter (default: `True` for most scenarios).
# controlled by the `include_baseline` parameter on `initialize_async`; when omitted, each
# scenario falls back to its own `DEFAULT_INCLUDE_BASELINE` class attribute (most scenarios
# default it on; `Jailbreak` defaults it off).
#
# To run *only* the baseline (no attack strategies), create a `RedTeamAgent` with
# `include_baseline=True` (the default) and pass `scenario_strategies=None`. See
# To run *only* the baseline (no attack strategies), pass `include_baseline=True` and
# `scenario_strategies=None` explicitly to `initialize_async` — that way the example works
# regardless of the scenario's default. See
# [Common Scenario Parameters](./1_common_scenario_parameters.ipynb) for a working example.

# %% [markdown]
Expand Down
Loading
Loading