Skip to content

Add ASR reporting by converter type#1350

Draft
Copilot wants to merge 4 commits intomainfrom
copilot/add-asr-by-converter-type
Draft

Add ASR reporting by converter type#1350
Copilot wants to merge 4 commits intomainfrom
copilot/add-asr-by-converter-type

Conversation

Copy link
Contributor

Copilot AI commented Feb 4, 2026

Adds converter type grouping to attack result analytics. Enables comparing effectiveness of different converters (Base64, ROT13, translation, etc.) across attacks.

Changes

  • pyrit/analytics/result_analysis.py: Extended analyze_results() to extract converter class names from AttackResult.last_response.converter_identifiers and compute stats per converter type. Returns new "By_converter_type" dict alongside existing groupings.

  • tests/unit/analytics/test_result_analysis.py: Added 5 tests covering no-converter tracking, single/multiple converters per attack, and undetermined outcomes. Added make_converter() helper to reduce duplication.

Behavior

Attacks without converters tracked as "no_converter". Pipeline attacks with multiple converters contribute to each converter's stats independently.

results = analyze_results(attack_results)

# New grouping available
results["By_converter_type"]["Base64Converter"]
# AttackStats(success_rate=0.75, successes=3, failures=1, undetermined=0, total_decided=4)

Non-breaking change - preserves existing return structure.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • astral.sh
    • Triggering command: /usr/bin/curl curl -LsSf REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>FEAT Reporting: show ASR by converter type</issue_title>
<issue_description>#### Is your feature request related to a problem? Please describe.

Related to #384.

printing ASR by converter type (e.g., translation converter, base64, etc.)</issue_description>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits February 4, 2026 01:49
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
Co-authored-by: romanlutz <10245648+romanlutz@users.noreply.github.com>
Copilot AI changed the title [WIP] Add feature to show ASR by converter type Add ASR reporting by converter type Feb 4, 2026
Copilot AI requested a review from romanlutz February 4, 2026 01:57

# Extract converter types from last_response
converter_types = []
if attack.last_response is not None and attack.last_response.converter_identifiers:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have so many things to filter on, I'm wondering if we want to go this route or something more general. E.g. filtering on any ConverterIdentifier field

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FEAT Reporting: show ASR by converter type

3 participants