Open
Conversation
* Fix top sample data * count top level sample.usage * update version to fix build analyze error on change log
…5527) * Fix XPIA binary_path incompatibility for model targets (#5058420) When the indirect jailbreak (XPIA) strategy creates file-based context prompts with binary_path data type, the callback chat target now reads the file content and converts to text before invoking the callback. This prevents ValueError from targets that don't support binary_path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: extract helper, add error handling and tests - Extract _resolve_content() helper to handle binary_path file reading for both current request AND conversation history pieces - Add try/except with logger.warning for unreadable files, falling back to the file path string - Add comment noting sync file read is intentional for small XPIA files - Add 5 unit tests for binary_path resolution Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply black formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…45528) * Fix content-filter responses showing raw JSON in results (#5058447) When Azure OpenAI content filters block a response, the result processor now detects the raw API payload and replaces it with a human-readable message like "[Response blocked by content filter: self_harm (severity: medium)]" instead of showing raw JSON. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: prefer JSON parsing, fix type annotation, add tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply black formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: fix false positives, generic regex, tighten checks - Fix critical false-positive bug: replace _has_content_filter_keys() (matches on key presence) with _has_finish_reason_content_filter() (requires finish_reason == content_filter). Azure OpenAI always includes content_filter_results even in unfiltered responses. - Replace hardcoded 4-category regex fallback with generic pattern that matches any category with filtered: true. - Tighten Step 3 last-resort check to require finish_reason indicator. - Add 5 new tests covering false-positive passthrough scenarios and non-standard category regex detection. - Replace TestHasContentFilterKeys with TestHasFinishReasonContentFilter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…items (#45722) The Foundry execution path (_build_messages_from_pieces) was not extracting token_usage from piece labels when building JSONL messages, unlike the orchestrator path in formatting_utils.py. This caused missing sample.usage on row-level output_items for agent targets using the Foundry path. Add token_usage extraction from labels for all message roles inside the existing hasattr guard, matching the behavior in formatting_utils.write_pyrit_outputs_to_file(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…feature flag (#45727) * Fix legacy endpoint backwards compatibility for _use_legacy_endpoint feature flag Fix 7 bugs that prevented the _use_legacy_endpoint=True flag from being fully backwards compatible with the pre-sync-migration behavior: 1. Add bidirectional metric name mapping in evaluate_with_rai_service_sync() and evaluate_with_rai_service_sync_multimodal(): legacy endpoint gets hate_fairness, sync endpoint gets hate_unfairness, regardless of caller input. 2. Skip _parse_eval_result() for legacy endpoint in _evaluate_query_response(): legacy returns a pre-parsed dict from parse_response(), return directly. 3. Restore whole-conversation evaluation in _evaluate_conversation() when legacy endpoint: send all messages in a single call (pre-migration behavior) instead of per-turn evaluation. 4. Remove dead effective_metric_name variable in _evaluation_processor.py: metric normalization is now handled at the routing layer. 5. Pass evaluator_name in red team evaluation processor for telemetry. 6. Add use_legacy_endpoint parameter to Foundry RAIServiceScorer and forward it to evaluate_with_rai_service_sync(). Remove redundant manual metric name mapping (now handled by routing layer). 7. Update metric_mapping.py comment to document the routing layer approach. Tests: - 9 new unit tests in test_legacy_endpoint_compat.py covering query/response, conversation, metric enum, and _parse_eval_result paths - 4 new unit tests in test_content_safety_rai_script.py covering routing, metric name mapping for both endpoints - 5 new e2e tests in test_builtin_evaluators.py covering all content safety evaluators with legacy endpoint, key format parity, and conversation mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip new e2e tests in playback mode (no recordings yet) The 5 new legacy endpoint e2e tests require test proxy recordings that don't exist yet. Mark them with pytest.mark.skip so CI passes in playback mode. The tests work correctly in live mode (verified locally). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove local test scripts from tracking Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add e2e test recordings and fix test infrastructure - Record 5 new legacy endpoint e2e tests (pushed to azure-sdk-assets) - Fix PROXY_URL callable check in conftest.py for local recording compat - Fix missing request.getfixturevalue() in test_self_harm_evaluator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove local test scripts that break CI collection These files import azure.ai.evaluation.red_team which requires pyrit, causing ImportError in CI environments without the redteam extra. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add groundedness legacy metric mapping and comprehensive legacy e2e tests - Map groundedness -> generic_groundedness for legacy annotation endpoint - Set metric_display_name to preserve 'groundedness' output keys - Add e2e tests for ALL evaluators with _use_legacy_endpoint=True: GroundednessPro, ProtectedMaterial, CodeVulnerability, IndirectAttack, UngroundedAttributes, ECI Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor metric name mapping to single dict Replace if/elif chains with _SYNC_TO_LEGACY_METRIC_NAMES dict used bidirectionally. Adding new metric mappings is now a one-line change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add XPIA and ECI to legacy metric name mapping The legacy annotation API returns results under 'xpia' and 'eci' keys, not 'indirect_attack' and 'election_critical_information'. Without this mapping, parse_response cannot find the metric key in the response and returns empty dict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix XPIA/ECI legacy response key lookup in parse_response The legacy annotation API returns XPIA results under 'xpia' and ECI under 'eci', but parse_response looked for 'indirect_attack' and 'election_critical_information'. Add _SYNC_TO_LEGACY_RESPONSE_KEYS fallback lookup in both parse_response and _parse_content_harm_response. Split mapping into two dicts: - _SYNC_TO_LEGACY_METRIC_NAMES: metrics where the API request name differs - _SYNC_TO_LEGACY_RESPONSE_KEYS: superset including response key differences Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ECI test assertion to use full metric name prefix ECIEvaluator uses _InternalEvaluationMetrics.ECI = 'election_critical_information' as metric_display_name, so output keys are election_critical_information_label, not eci_label. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * adding recordings * Address PR review comments - Define _LEGACY_TO_SYNC_METRIC_NAMES at module level (avoid rebuilding on every call) - Fix assertion in test to match string type (not enum) - Remove unused @patch decorator and cred_mock parameter - Delete test_legacy_endpoint_compat.py entirely - Fix effective_metric_name NameError in _evaluation_processor.py lookup_names - Route legacy conversation through sync wrapper for metric normalization - Remove unused evaluate_with_rai_service_multimodal import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address nagkumar91 review comments - Extract _normalize_metric_for_endpoint() helper (fixes duplication + ensures metric_display_name is set in both sync and multimodal paths) - Fix legacy conversation path to produce evaluation_per_turn structure by wrapping result through _aggregate_results() - Add comments clarifying response key fallback is inherently legacy-only (parse_response is only called from legacy endpoint functions) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix conversation legacy test + thread metric_display_name in multimodal - Fix conversation legacy test: assert per-turn length == 1 (not 2), since legacy sends entire conversation as single call - Thread metric_display_name through evaluate_with_rai_service_multimodal so legacy multimodal results use correct output key names (e.g. hate_unfairness_* not hate_fairness_*) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix legacy endpoint conversation eval routing through _convert_kwargs_to_eval_input The parent class _convert_kwargs_to_eval_input decomposes text conversations into per-turn {query, response} pairs before _do_eval is called, routing to _evaluate_query_response instead of _evaluate_conversation. This bypasses the legacy single-call logic entirely. Override _convert_kwargs_to_eval_input in RaiServiceEvaluatorBase to pass conversations through intact when _use_legacy_endpoint=True, so _evaluate_conversation is reached and sends all messages in one API call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix validate_conversation for text conversations and re-record E2E tests Move validate_conversation() call after the legacy endpoint check since it requires multimodal (image) content. Text conversations routed through the legacy path don't need this validation. Re-recorded test_content_safety_evaluator_conversation_with_legacy_endpoint in live mode and pushed new recordings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add entries for all 6 changes since 1.16.0 and set release date to 2026-03-18: - Fix top sample data (#45214) - Agentic evaluators accept string inputs (#45159) - Fix XPIA binary_path for model targets (#45527) - Fix content-filter raw JSON display (#45528) - Extract token_usage in Foundry path (#45722) - Fix legacy endpoint backwards compat (#45727) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
#45786) * docs: Backport CHANGELOG entries for azure-ai-evaluation 1.16.1 hotfix Add missing entries for 5 changes merged to main since 1.16.0 that were not reflected in the CHANGELOG, and set release date to 2026-03-18: - Agentic evaluators accept string inputs (#45159) - Fix XPIA binary_path for model targets (#45527) - Fix content-filter raw JSON display (#45528) - Extract token_usage in Foundry path (#45722) - Fix legacy endpoint backwards compat (#45727) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: Add 1.16.2 (Unreleased) section to CHANGELOG Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: resolve analyze check failures for 1.16.1 changelog backport - Update _version.py to 1.16.2 to match unreleased CHANGELOG entry - Remove empty 'Other Changes' section from 1.16.2 unreleased block - Add 'Agentic' to cspell.json allowed words list Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…k strategies (#45776) * Fix UTF-8 encoding for red team JSONL files on Windows Add explicit encoding='utf-8' to all file open() calls in the PyRIT result processing path. Without this, Windows defaults to the system locale encoding (charmap/cp1252), causing UnicodeDecodeError when reading JSONL files containing non-ASCII characters from UnicodeConfusable strategy or CJK languages. Fixes: Tests 1.7 (UnicodeConfusable), 1.16 (Japanese/Chinese) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add encoding regression tests for non-ASCII JSONL round-trip Test CJK characters, Unicode confusables, and mixed scripts to prevent future regressions of the charmap encoding bug on Windows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Format with black Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: test production code paths, consolidate CHANGELOG Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply black formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix adversarial_chat_target using user callback instead of RAI service The Foundry execution path was incorrectly passing the user's callback target as adversarial_chat_target to PyRIT's FoundryScenario. This caused PyRIT's TenseConverter to use the callback as its LLM for prompt rephrasing, resulting in the callback's fixed response leaking into converted_value and appearing as the user message in results. Changes: - Create AzureRAIServiceTarget with strategy-appropriate template key instead of reusing the user's callback target - Add _get_adversarial_template_key() to select the correct RAI service template per attack strategy (crescendo, multi-turn, or tense converter) - Show original_value for user messages in _build_messages_from_pieces() as defense-in-depth against converter output leaking into display - Add 9 regression tests covering template key selection, wiring verification, original_value display, and the exact reported bug - Fix existing test mocks to set original_value on user-role pieces Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: @staticmethod, crescendo_format, test cleanup - Convert _get_adversarial_template_key to @staticmethod - Pass crescendo_format=True when crescendo template is selected - Remove anti-pattern test and CentralMemory singleton leak - Update staticmethod test calls to not pass None as self Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add 15 Foundry red team E2E tests for full RAISvc contract coverage Tests cover: basic execution, XPIA, multiple risk categories, application scenarios, strategy combinations, model_config targets, agent callbacks, agent tool context, ProtectedMaterial/CodeVulnerability/TaskAdherence categories, SensitiveDataLeakage, agent-only risk rejection, multi-turn, and crescendo attacks. Also fixes PROXY_URL() TypeError in conftest.py (PROXY_URL is a str, not callable). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PROXY_URL() call and apply black formatting - Revert PROXY_URL back to PROXY_URL() (it's a function, not a variable) - Apply black formatting to assert statements in test_red_team_foundry.py Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Windows encoding bug in tqdm output and use custom seeds for agent risk categories - Add _safe_tqdm_write() wrapper to handle UnicodeEncodeError on Windows cp1252 terminals - Replace all tqdm.write() calls with _safe_tqdm_write() in _red_team.py - Add custom seed prompt files for agent-only risk categories (task_adherence, sensitive_data_leakage, prohibited_actions) that lack server-side seed data - Update test_foundry_task_adherence_category and test_foundry_agent_sensitive_data_leakage to use custom_attack_seed_prompts, bypassing get_attack_objectives API - Apply black formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Re-record foundry E2E tests after merging upstream/main - Merge upstream/main (7 commits) into foundry-e2e-tests branch - Fix PROXY_URL() call in conftest.py (PROXY_URL is a string, not callable) - Re-record all 15 foundry red team E2E tests with updated source code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PROXY_URL handling for both callable and string variants In CI, devtools_testutils.config.PROXY_URL is a function that must be called. Locally (pip-installed), it's a string constant. Use callable() check to handle both environments. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test_foundry_with_model_config_target recording playback failure Patch random.sample and random.choice to return deterministic (first-N) results for the model config target test. This ensures the same objectives are selected during both recording and playback, preventing test proxy 404 mismatches caused by non-deterministic objective selection. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Azure OpenAI endpoint normalization for PyRIT 0.11+ compatibility Extend /openai/v1 path normalization to all Azure endpoint patterns (*.openai.azure.com, *.cognitiveservices.azure.com, sovereign clouds) not just Foundry endpoints. PyRIT 0.11+ uses AsyncOpenAI(base_url=) which appends /chat/completions directly, requiring the /openai/v1 prefix. Without this fix, model config targets using classic AOAI endpoints get 404 errors because PyRIT sends requests to the bare endpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix hate_unfairness metric name mismatch in RAI scorer RISK_CATEGORY_METRIC_MAP mapped HateUnfairness to HATE_FAIRNESS (legacy name), but the sync eval API returns results under hate_unfairness (canonical name). The scorer's result matching compared against the un-normalized hate_fairness, causing it to never match and silently fall back to score=0 — making ASR always 0% for hate_unfairness regardless of actual model behavior. Changes: - metric_mapping.py: Map HateUnfairness to HATE_UNFAIRNESS (canonical name). The routing layer in evaluate_with_rai_service_sync normalizes to the legacy name when use_legacy_endpoint=True, so both paths work. - _rai_scorer.py: Match results against both canonical and legacy aliases using _SYNC_TO_LEGACY_METRIC_NAMES, so future metric renames don't silently break scoring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update recording for model config target test * Update unit tests for Azure OpenAI endpoint normalization Tests now expect /openai/v1 suffix on all Azure endpoints, matching the updated get_chat_target() behavior needed for PyRIT 0.11+. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix agent seed auth for local SDK usage When target_type=agent and no client_id is provided (local execution, not ACA), fall back to the existing credential to set aml-aca-token header. Previously this header was only set via ACA managed identity, causing 'Authorization failed for seeds' when running agent-target red team scans locally. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "Fix agent seed auth for local SDK usage" This reverts commit abb47c4. * Fix send_prompt_async parameter name for PyRIT 0.11+ and agent seed auth Two fixes: 1. _rai_service_target.py: Accept both 'message' (PyRIT 0.11+) and 'prompt_request' (legacy) parameter names in send_prompt_async(). PyRIT 0.11 changed the interface from prompt_request= to message=, causing TypeError on multi-turn and crescendo attacks. 2. _generated_rai_client.py: Set aml-aca-token header from existing credential for agent-type seed requests when no client_id (ACA managed identity) is available. Enables local SDK testing of agent targets without ACA. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update recordings for foundry E2E tests * Update unit tests * Apply black formatting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #45579 review feedback - Fix list[Message] -> List[Message] type hint for Python 3.8 compat - Guard _fallback_response against None when retry kwargs are malformed - Add CHANGELOG entries for metric fix, PyRIT compat, endpoint normalization, and agent token fallback - Move _AZURE_OPENAI_HOST_SUFFIXES to module-level constant - Use _validate_attack_details shared helper in multi-turn/crescendo tests - Change agent token fallback log level from debug to warning Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: improve logging, error handling, and imports - Upgrade XPIA agent fallback log from debug to warning (_red_team.py) - Upgrade aml-aca-token credential fallback log from debug to warning (_generated_rai_client.py) - Raise RuntimeError instead of returning [] in _fallback_response (_rai_service_target.py) - Move metric name imports to module level (_rai_scorer.py) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove empty Features Added and Breaking Changes sections - Remove duplicate adversarial chat target fix entry from 1.16.1 section Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ry (#45865) * Extract RAI scorer token metrics into Score metadata and save to memory - Extract token usage (prompt_tokens, completion_tokens, total_tokens) from RAI service eval_result via sample.usage or result properties.metrics - Add token_usage to score_metadata dict in RAIServiceScorer - Save scores to PyRIT CentralMemory after creation (fail-safe) - Propagate scorer token_usage through ResultProcessor to output item properties.metrics for downstream aggregation - Add 5 unit tests covering token extraction, memory save, and error handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use metric aliases in _extract_token_usage fallback Match against canonical and legacy metric name aliases when extracting token usage from result-level properties.metrics, consistent with how score extraction already handles aliases via _SYNC_TO_LEGACY_METRIC_NAMES and _LEGACY_TO_SYNC_METRIC_NAMES. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Backport 1.16.2 hotfix CHANGELOG with release date (2026-03-24) - Add missing token metrics entry (PR #45865) to 1.16.2 section - Add 1.16.3 (Unreleased) section with existing extra_headers feature - Bump _version.py to 1.16.3 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…46000) * Fix ASR scoring: use score-based threshold instead of passed field The sync eval API's 'passed' field indicates whether the eval task completed successfully, NOT whether the content passed a safety check. Using 'not passed' to determine attack success always returns False (since passed=True for successful evals), resulting in 0% ASR even for clearly harmful content (e.g. raw_score=5.0 for violence). Fix: Replace passed-based logic with score-based threshold comparison matching _evaluation_processor.py. Use get_default_threshold_for_evaluator() (default=3 for 0-7 scale, 0 for binary) and compare raw_score > threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pass evaluator name to get_harm_severity_level and add binary evaluator test - Fix get_harm_severity_level call to pass evaluator=metric_name_str so non-0-7-scale evaluators (e.g. task_adherence) get correct severity labels - Add test_score_async_binary_evaluator_threshold covering binary evaluator threshold (task_adherence, threshold=0) to verify score>0 logic Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Deduplicate attack success logic, fix severity labels, add changelog - Extract is_attack_successful() helper into _common/utils.py to avoid duplicating threshold comparison logic across _rai_scorer.py and _rai_service_eval_chat_target.py - Fix get_harm_severity_level call in _rai_service_eval_chat_target.py to pass evaluator name for correct pattern-specific severity labels - Add CHANGELOG entry for 1.16.3 describing the ASR scoring fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Recover partial red team results when some objectives fail
When PyRIT's scenario raises ValueError due to incomplete objectives
(e.g., evaluator model refuses to score adversarial content), the
completed results were lost because _scenario_result remained None.
Now retrieves partial results from PyRIT's memory database using the
scenario_result_id. PyRIT saves completed results to memory before
raising, so they can be recovered even when the scenario fails.
Tested: 50 objectives with code_vulnerability, 48/50 completed,
2 refused by content filter. Before: 0 results in JSONL. After:
48 results preserved in JSONL.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix partial results recovery: harden logging and update tests
- Use getattr for attack_results in log message to prevent AttributeError
from masking a successful recovery when stored result has unexpected shape
- Use %s-style logger formatting for consistency with rest of codebase
- Update tests to mock the new _scenario_result_id + get_memory() path
instead of the old _result attribute that is no longer read
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip crescendo test with stale recordings
The test proxy cannot match requests due to Accept-Encoding header
mismatch between live requests and existing recordings. Skip until
recordings are re-captured.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use dict default for attack_results in recovery log
attack_results is a dict, not a list. Use {} default to keep types
consistent with get_attack_results() downstream usage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: null-check stored result, count individual results
- Add 'stored_results[0] is not None' guard per reviewer feedback
- Count individual AttackResult objects across objective groups instead
of just dict keys, for more useful recovery diagnostics
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add changelog entry for partial results recovery fix
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
) * Fix evaluator token metrics not persisted in red teaming results The sync eval API returns token usage keys in camelCase (promptTokens, completionTokens) but _extract_token_usage() only looked for snake_case keys (prompt_tokens, completion_tokens). This caused the extraction to silently return an empty dict, so scorer_token_usage was never set and evaluator token metrics were dropped from red teaming output items. The fix normalises both camelCase and snake_case keys to snake_case in _extract_token_usage(), covering both SDK model objects (snake_case) and raw JSON responses from non-OneDP endpoints (camelCase). Also updated _compute_per_model_usage() in _result_processor.py to accept both key styles when aggregating evaluator token usage, since scorer_token_usage now arrives in snake_case. Added two new tests for camelCase key handling in both sample.usage and result properties.metrics extraction paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: use American English spelling (normalize) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
bot
added
the
Evaluation
Contributor
There was a problem hiding this comment.
Pull request overview
Merges the 1.16.3 hotfix work back into main for azure-ai-evaluation, focusing on red team / Foundry integration correctness (endpoint normalization, scoring semantics, partial-result recovery) and improved evaluator compatibility (legacy routing, agent/tool string inputs), along with expanded unit/e2e coverage and updated test data.
Changes:
- Normalize Azure OpenAI endpoints for PyRIT (
/openai/v1) across public + sovereign host suffixes and update related tests. - Fix red teaming scoring and result processing: score-based ASR semantics, content-filter message cleaning, token-usage propagation/aggregation, and partial-result recovery from PyRIT memory.
- Improve legacy endpoint routing + metric name normalization, expand evaluator handling for string-based agent/tool inputs, and add/adjust tests + seed data.
Reviewed changes
Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_strategy_utils.py | Updates expectations for normalized Azure OpenAI endpoints (/openai/v1). |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_result_processor.py | Adds unit coverage for content-filter cleaning helpers in ResultProcessor. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_rai_service_target.py | Adjusts tests for PyRIT 0.11+ send_prompt_async signature/return type changes. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_rai_service_eval_chat_target.py | Updates metric naming and expected metadata (threshold, removal of passed). |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_foundry.py | Extends Foundry tests for thresholds, memory recovery, token usage, and regressions. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_callback_chat_target.py | Adds tests for resolving binary_path pieces into inline text before callbacks. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_content_safety_rai_script.py | Adds tests for legacy routing and metric-name mapping (incl. multimodal sync wrapper). |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_agent_evaluators.py | Removes a “no tool calls” assertion block from tool-call accuracy tests. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/data/evaluation_util_convert_old_output_test.jsonl | Updates conversion fixture rows to include input sample status + generated sample JSON. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/data/evaluation_util_convert_expected_output.json | Updates expected converted AOAI-format output (sample payload + per-model usage changes). |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_red_team.py | Skips a stale-recording test case. |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_red_team_foundry.py | Adds broader Foundry e2e scenarios (model-config targets, agent targets, new risks, multi-turn). |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/test_builtin_evaluators.py | Adds legacy-endpoint e2e coverage and formatting fixes. |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/data/redteam_seeds/task_adherence_seeds.json | Adds seed prompts for TaskAdherence risk category e2e coverage. |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/data/redteam_seeds/sensitive_data_leakage_seeds.json | Adds seed prompts for SensitiveDataLeakage (agent-only) e2e coverage. |
| sdk/evaluation/azure-ai-evaluation/tests/e2etests/data/redteam_seeds/prohibited_actions_seeds.json | Adds seed prompts for ProhibitedActions risk category e2e coverage. |
| sdk/evaluation/azure-ai-evaluation/tests/conftest.py | Makes test-proxy URL handling compatible with callable/static PROXY_URL configurations. |
| sdk/evaluation/azure-ai-evaluation/cspell.json | Adds “Agentic” to spellchecker allowlist. |
| sdk/evaluation/azure-ai-evaluation/CHANGELOG.md | Adds 1.16.4 section and documents 1.16.3 hotfix contents. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/simulator/_model_tools/_generated_rai_client.py | Adds aml-aca-token fallback for agent-type objective fetch when running locally. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_utils/strategy_utils.py | Normalizes Azure OpenAI endpoints across known host suffixes for PyRIT compatibility. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_utils/metric_mapping.py | Switches HateUnfairness mapping to canonical sync metric name (hate_unfairness). |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_utils/_rai_service_target.py | Supports PyRIT 0.11+ message= arg name and returns list of messages; updates retry fallback. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_utils/_rai_service_eval_chat_target.py | Uses score-based success logic and default thresholds; updates severity labeling call. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_result_processor.py | Adds content-filter cleaning, token-usage propagation, safer metadata parsing, and usage aggregation fixes. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_red_team.py | Adds encoding-safe tqdm output and fixes Foundry adversarial target creation to avoid callback leakage. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_scenario_orchestrator.py | Recovers partial results from PyRIT memory; sets refusal_scorer for crescendo logic. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_rai_scorer.py | Normalizes metric aliases, uses score-based thresholds, extracts token usage, and saves scores to memory. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_foundry/_foundry_result_processor.py | Shows original user prompt (not converted) and propagates token_usage from piece labels. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_evaluation_processor.py | Simplifies metric lookup and passes evaluator name through sync eval calls. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_callback_chat_target.py | Resolves binary_path content by reading files before invoking callback targets. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_version.py | Bumps package version to 1.16.4. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_selection/_tool_selection.py | Allows string tool-call inputs and relaxes list-only assumptions. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_output_utilization/_tool_output_utilization.py | Avoids reformatting when query/response/tool_definitions are strings; refactors formatting flow. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_input_accuracy/_tool_input_accuracy.py | Allows string tool-call inputs and adjusts tool-definition extraction accordingly. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_success/_tool_call_success.py | Supports string responses and improves intermediate-response handling. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy.py | Allows string tool-call inputs and relaxes list-only assumptions. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_groundedness/_groundedness.py | Short-circuits for string responses to skip agent-context extraction. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py | Ensures legacy conversation evaluation is single-call and preserves legacy parsing behavior. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate.py | Parses sample.generated_sample_data from inputs into top-level AOAI sample and updates usage aggregation. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py | Adds is_attack_successful() helper for score-threshold based ASR logic. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/rai_service.py | Adds metric normalization + legacy response-key fallback for sync/legacy routing and multimodal support. |
| sdk/evaluation/azure-ai-evaluation/assets.json | Updates the assets tag reference. |
| criteria_results, sample = _process_criteria_metrics( | ||
| criteria_name, metrics, testing_criteria_metadata, logger, eval_id, eval_run_id | ||
| ) | ||
| run_output_results.extend(criteria_results) |
There was a problem hiding this comment.
The loop calls _process_criteria_metrics(...) and captures sample, but sample is no longer used to populate top_sample. This means rows without inputs.sample.generated_sample_data will now always return an empty top-level sample, even when criteria results include per-metric sample data. Consider restoring a fallback (e.g., set top_sample from the first non-empty sample when top_sample is still empty) to avoid regressions for callers that rely on the top-level sample payload.
Suggested change
| run_output_results.extend(criteria_results) | |
| run_output_results.extend(criteria_results) | |
| # Fallback: if no top-level sample is set from input_data, use the first non-empty sample from criteria results. | |
| if not top_sample and sample: | |
| top_sample = sample |
Comment on lines
+281
to
282
| @pytest.mark.skip(reason="Recordings are stale (Accept-Encoding header mismatch). Re-record separately.") | ||
| @pytest.mark.azuretest |
There was a problem hiding this comment.
This test is now unconditionally skipped due to stale recordings. Skipping an @pytest.mark.azuretest permanently removes coverage in playback and can hide regressions. Prefer re-recording/fixing the recording mismatch (or gating the skip to playback-only conditions) so CI continues exercising the scenario.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.