Skip to content

v0.13.0

Latest
Latest

Choose a tag to compare

View all tags
@adrian-gavrila adrian-gavrila released this 17 Apr 21:08
· 26 commits to main since this release
v0.13.0
6dc8b94

What's changed?

This release continues the push toward a more composable framework. The headline items are the TargetConfiguration redesign (replacing TargetCapabilities), a new AttackTechnique abstraction that standardizes how attacks declare and consume their arguments, and a new Converter Panel in the CoPyRIT GUI. We also landed a large unit-test coverage push, hardened CoPyRIT for deployment, and closed out several security items.

⚠️ Breaking Changes

  • TargetConfiguration replaces TargetCapabilities with per-piece input/output typing (#1573, #1588)
  • New AttackTechnique abstraction with standardized attack args across the framework (#1592, #1608)
  • Removed functionality deprecated for v0.13.0, including the FoundryScenario alias and piece.role in conversation analytics (#1618, #1623)
  • Stricter validation of explicit empty field overrides in the attack executor (#1507)

Please review the deprecation notes and migration guidance before upgrading.

🎯 Targets & Attacks

  • TargetConfiguration and message pieces redesign for richer target modeling (#1573, #1588)
  • New TargetRequirements to express target-level capability requirements (#1582)
  • AttackTechniqueRegistry for discoverable, composable attacks (#1611)
  • supports_system_prompt flag on targets (#1563)
  • Image input enabled by default for OpenAIChatTarget (#1628)
  • Preserve roles when converting messages to seed prompts (#1508)

📚 Datasets

  • Added VisualLeakBench dataset loader (#1531)
  • Added ISO 42001-aligned harm definitions for AI supply chain, transparency, and governance (#1462)
  • Normalize remote dataset file types from URLs (#1486)
  • Handle empty CSV exports in the remote dataset cache (#1481)
  • Include subdirectory jailbreak templates in listings (#1498)

📊 Scoring

  • SelfAskRefusalScorer improvements and updated scorer metrics workflow (#1549)
  • Pre-release scorer evaluation metrics refresh (#1626)
  • Deduplicate message pieces before batch scoring (#1504)

🖥️ CoPyRIT (GUI) & CLI

  • Converter Panel added to the CoPyRIT GUI (#1471)
  • Security hardening and Azure deployment support for CoPyRIT (#1554)
  • Backend now defaults to localhost instead of 0.0.0.0 (#1612)
  • GUI target config now respects the configured model name even when an env var is set (#1590)
  • CLI bug fixes and minor updates (#1559)
  • Preserve quoted shell arguments in run parsing (#1483)

🧩 Framework internals

  • IdentifierFilters to enable generic DB queries on component identifiers (#1557)
  • Standardized AIRTInitializer (#1578)
  • Support relative blob paths in AzureBlobStorageIO (#1478)
  • Respect export type in SQLite conversation exports (#1493)
  • Preserve raw HTTP body whitespace in HTTPTarget (#1495)

🔒 Security

  • Mitigate Jinja2 Server-Side Template Injection (SSTI) vulnerability (#1577, #1587)
  • Resolve code scanning path injection alerts in the media endpoint (#1607)
  • Harden the Jupyter Docker image (#1584)
  • Dependabot security bumps across Python and frontend (#1606, #1614)

🧪 Tests & Tooling

  • Coverage enforcement gate added, with a 78% global floor and 90% on diffs (#1605)
  • Massive unit-test coverage push: auth (#1596), models (#1601), score (#1602), executor (#1603), identifiers (#1597), remote dataset loaders (#1604), converters (#1594), memory models (#1598), common utilities (#1600), converter configuration (#1599)
  • AWS Bedrock partner integration tests for OpenAI-compatible Mantle endpoints (#1575)
  • Partner integration test pipeline YAML (#1543)
  • Windows support for the npm lookup in prepare_package.py (#1569, #1629)

📖 Docs

  • Documentation restructured around three user pathways (framework / CLI / GUI) (#1583)
  • New scenario doc pages for 7 previously undocumented scenarios (#1558)
  • Patch release guide added to the release process doc (#1585)
  • Cookbooks 1, 2, and 4 migrated into proper doc locations; cookbooks directory removed (#1564, #1580)
  • Pre-commit hook to strip tqdm progress bars from notebooks (#1581)
  • Frontend style instructions (#1576)

🐛 Bug Fixes and Improvements

See the full changelog below for everything else.

Full list of changes

  • FIX: Add *.jsonl to MANIFEST.in for scorer metrics packaging by @varunj-msft in #1546
  • MAINT post-0.12.0 release update to 0.12.1.dev0 by @varunj-msft in #1547
  • MAINT: Allow custom Likert system prompt and scale by @jsong468 in #1514
  • [MAINT] Migrate repository references from Azure/PyRIT to microsoft/PyRIT by @spencrr in #1535
  • [BREAKING] FIX: CLI Updates by @jsong468 in #1545
  • [MAINT]: Fold notebook header removal logic to nbstripout by @spencrr in #1550
  • FEAT: Add harm_categories to Babelscape ALERT dataset (#449) by @CheerathAniketh in #1551
  • fix: handle closed logging streams in dispose_engine shutdown by @2byrds in #1526
  • TEST Add partner-integration-tests yml by @hannahwestra25 in #1543
  • FEAT: Updating Scorer Metrics Update Workflow and SelfAskRefusalScorer update by @rlundeen2 in #1549
  • FIX use the same integration test service principal by @hannahwestra25 in #1562
  • MAINT: standardize converter docstrings by @jbolor21 in #1560
  • DOC: Add scenario doc pages for all 7 undocumented scenarios by @varunj-msft in #1558
  • FEAT add supports_system_prompt flag by @hannahwestra25 in #1563
  • MAINT Remove references to the now-deprecated Crucible by @behnam-o in #1568
  • DOC add instructions about partner integration tests by @hannahwestra25 in #1574
  • MAINT Pin node package versions by @behnam-o in #1571
  • DOC update User Guide dropdown nav to match actual TOC sections by @romanlutz in #1572
  • TEST Add AWS Bedrock integration tests for OpenAI-compatible Mantle endpoint by @romanlutz in #1575
  • DOC add style instructions for frontend by @hannahwestra25 in #1576
  • [MAINT]: Fixup remaining Azure/PyRIT refs by @spencrr in #1556
  • FEAT: Add partner integration tests for azure-ai-evaluation red team … by @slister1001 in #1533
  • FEAT: Converter Panel for GUI! by @jbolor21 in #1471
  • DOC: Migrate cookbooks 1, 2, and 4 to proper doc locations - PR #2 by @varunj-msft in #1564
  • FEAT add TargetConfiguration & pieces by @hannahwestra25 in #1573
  • DOC: Delete cookbooks directory and fix cross-references - PR #3 by @varunj-msft in #1580
  • FIX: Audio files content check bug by @jbolor21 in #1579
  • FIX Mitigate Jinja2 Server-Side Template Injection (SSTI) vulnerability by @romanlutz in #1577
  • DOC: Add pre-commit hook to strip tqdm progress bars from notebooks by @rlundeen2 in #1581
  • MAINT: harden jupyter docker by @rlundeen2 in #1584
  • MAINT Bump version to 0.13.0.dev0 by @romanlutz in #1586
  • FIX: Further updates to jinja deserialization by @rlundeen2 in #1587
  • DOC: Restructure documentation around three user pathways by @rlundeen2 in #1583
  • DOC: Add patch release guide to release process by @romanlutz in #1585
  • FEAT add TargetRequirements by @hannahwestra25 in #1582
  • FEAT: Introduce IdentifierFilters to allow generic DB queries on identifier… by @behnam-o in #1557
  • FEAT: Standardizing AIRTInitializer by @ValbuenaVC in #1578
  • FIX: CLI bug fixes and minor updates by @jsong468 in #1559
  • FIX: Windows support for npm lookup by @fdubut in #1569
  • Preserve raw HTTP body whitespace in HTTPTarget by @biefan in #1495
  • Respect export type in SQLite conversation exports by @biefan in #1493
  • TEST: Moving dataset tests to end-to-end by @rlundeen2 in #1589
  • Handle empty CSV exports in remote dataset cache by @biefan in #1481
  • FEAT: add ISO 42001 aligned harm definitions for AI supply chain, transparency and governance by @tejas0077 in #1462
  • MAINT Support relative blob paths in AzureBlobStorageIO by @biefan in #1478
  • FIX Include subdirectory jailbreak templates in listings by @biefan in #1498
  • TEST: Rename converter/ to prompt_converter/ and add 19 converter tests by @romanlutz in #1594
  • TEST: Rename tests/unit/scenarios/ to tests/unit/scenario/ by @romanlutz in #1593
  • FEAT: Security & Azure deployment for CoPyRIT GUI by @adrian-gavrila in #1554
  • Bump dependencies to resolve Dependabot security alerts by @romanlutz in #1606
  • TEST: Rename target/ to prompt_target/ and add target tests by @romanlutz in #1595
  • Preserve roles when converting messages to seed prompts by @biefan in #1508
  • TEST: Add coverage enforcement (fail-under 78%, diff-cover 90%) by @romanlutz in #1605
  • TEST: Add unit tests for remote dataset loaders by @romanlutz in #1604
  • TEST: Add unit tests for pyrit/identifiers/ module by @romanlutz in #1597
  • TEST: Fixing integration tests by @rlundeen2 in #1609
  • MAINT BREAK: Adding Attack Technique Abstraction by @rlundeen2 in #1592
  • MAINT Breaking: Standardizing Attack Args by @rlundeen2 in #1608
  • TEST: Add unit tests for pyrit/executor/ module by @romanlutz in #1603
  • TEST: Add unit tests for pyrit/score/ infrastructure by @romanlutz in #1602
  • TEST: Add unit tests for prompt_converter_configuration.py by @romanlutz in #1599
  • TEST: Add unit tests for pyrit/models/ module by @romanlutz in #1601
  • TEST: Add unit tests for pyrit/auth/ module by @romanlutz in #1596
  • TEST: Adding resiliency to refusal notebook by @rlundeen2 in #1610
  • FEAT migrate to TargetConfiguration from TargetCapabilities by @hannahwestra25 in #1588
  • Fix code scanning path injection alerts in media endpoint by @romanlutz in #1607
  • FEAT Add VisualLeakBench dataset loader (arXiv:2603.13385) by @Copilot in #1531
  • FIX: GUI target config shows wrong model name due to env var override by @romanlutz in #1590
  • FIX Default backend to localhost instead of 0.0.0.0 by @adrian-gavrila in #1612
  • Raise a clear error for empty YAML files by @biefan in #1482
  • FIX Preserve quoted shell arguments in run parsing by @biefan in #1483
  • FIX: update dependencies to address Dependabot security alerts by @romanlutz in #1614
  • TEST: Add unit tests for pyrit/common/ utilities by @romanlutz in #1600
  • TEST: Add unit tests for pyrit/memory/memory_models.py by @romanlutz in #1598
  • MAINT Deduplicate message pieces before batch scoring by @biefan in #1504
  • FEAT: Adding AttackTechniqueRegistry by @rlundeen2 in #1611
  • MAINT Remove all functionality deprecated for v0.13.0 by @romanlutz in #1618
  • MAINT Normalize remote dataset file types from URLs by @biefan in #1486
  • FIX Replace deprecated piece.role with piece.api_role in doc notebooks by @romanlutz in #1620
  • MAINT Remove v0.13.0 deprecated APIs: FoundryScenario alias and piece.role in conversation_analytics by @adrian-gavrila in #1623
  • MAINT: Emit deprecation warning for unused fields being removed from message piece by @behnam-o in #1625
  • Validate explicit empty field overrides in attack executor by @biefan in #1507
  • FIX: remove type suppression comments from MessagePiece.to_message() via TYPE_CHECKING by @Copilot in #1621
  • MAINT: Add pre-release scorer evaluation metrics by @adrian-gavrila in #1626

Full Changelog: v0.12.1...v0.13.0

Contributors

2byrds, romanlutz, and 14 other contributors
Assets 2
Loading