[Incident][108dev] Questions evaluation fallback: no usable graders / empty Foundry runtime config

[Cataldir]

Summary

Questions evaluation is falling back to baseline because there are no usable graders in the questions flow.

Evidence

• GET /api/questions/graders returns [] (200)
• GET /api/questions/assemblies returns only asm-essay-* entries with agents: []
• Repro call to POST /api/questions/grader/interaction returns baseline with:
  • Assembly 'asm-essay-fund1-001' has no graders
• Runtime env on tutor-questions-108dev:
  • PROJECT_ENDPOINT = empty
  • MODEL_DEPLOYMENT_NAME = empty
• Foundry specialist check:
  • Foundry project exists (.../tutor-108dev-ai-project)
  • model deployments exist (gpt-5, gpt-5-nano)
  • project currently has zero agents

Impact

• Users cannot get real multi-grader evaluation on Questions; only fallback baseline feedback appears.

Proposed remediation

1. Restore/set AI env vars for questions (PROJECT_ENDPOINT, MODEL_DEPLOYMENT_NAME).
2. Re-seed question assemblies/graders so asm-quest-* exists with non-empty agents.
3. Add deploy guardrails to fail when AI env vars are empty.
4. Keep [Data Isolation][108dev] Questions assemblies endpoint includes essay assemblies #145 for cross-domain assembly contamination tracking.

[Cataldir]

Break-glass local remediation executed on 108dev (questions):

Actions taken:

1. Started local Docker engine and built local hotfix image from workspace changes.
2. Pushed image to ACR and updated tutor-questions-108dev container app.
3. Verified rollout health:
   • latestRevision = latestReadyRevision = tutor-questions-108dev--0000017
   • provisioningState = Succeeded
   • runningStatus = Running

Post-fix validations:

• GET /api/questions/assemblies now returns only question assemblies with graders (asm-quest-*), no essay assemblies.
• POST /api/questions/grader/interaction with case_id=asm-quest-fund1 now returns full grader output (not fallback baseline/no-graders).

Root-cause context for earlier symptom:

• Prior UI selection was hitting asm-essay-* with empty agents, producing fallback "has no graders".
• Local hotfix plus reseeded question assemblies/graders restored real evaluation path.

[Cataldir]

Fast update completed for user-requested Agents API V2 migration:

What was changed:

• FoundryAgentService.create_agent now prefers V2 AIProjectClient.agents.create_version(...).
• Local grader creator .tmp/create_question_graders.py now creates graders via V2 and produces versioned IDs (agent_name:version).
• Agent naming is normalized to V2 constraints (kebab-case, <=63 chars).

Live break-glass validation (108dev):

• New questions revision active and healthy.
• Question assemblies now reference V2 IDs, e.g. corretor-fund-i-exatidao:1.
• POST /api/questions/grader/interaction returns full grader evaluation (not baseline fallback).

Note:

• PUT /api/questions/assemblies/{id} currently errors (partition_key kwarg issue). Used delete+post recreate flow as temporary operational workaround.

[Cataldir]

Permanent fix completed (English update):

Root cause confirmed:

• The component runtime environment was using an outdated SDK (azure-ai-projects 1.1.0b4), while the repo lockfiles already target stable V2 (2.0.0).
• That mismatch allowed legacy behavior and inconsistent agent API capabilities.

Permanent corrections applied:

1. Enforced V2 dependency ranges in source manifests:
   • lib/pyproject.toml: azure-ai-projects>=2.0.0,<3.0.0, openai>=2.26.0,<3.0.0
   • apps/questions/pyproject.toml: azure-ai-projects>=2.0.0,<3.0.0, openai>=2.26.0,<3.0.0
2. Added fail-fast runtime guard in FoundryAgentService:
   • Raises explicit error if azure-ai-projects < 2.0.0.
3. Migrated agent creation flow to V2 (create_version) and validated versioned IDs (agent_name:version).

Validation evidence:

• Local env now reports azure-ai-projects 2.0.0 and openai 2.29.0.
• V2 grader creation script succeeds and outputs versioned IDs (e.g., corretor-fund-i-exatidao:1).

Notes:

• Current 108dev evaluation path is functioning with V2-created agent references.
• Separate known issue remains for PUT /api/questions/assemblies/{id} (partition_key kwarg error); workaround currently uses delete+post recreate flow.

[Cataldir]

Permanent fix deployed on 108dev for PUT /api/questions/assemblies/{id} internal error (partition_key argument mismatch).

What changed:

• Patched shared Cosmos CRUD update path in lib/src/tutor_lib/cosmos/crud.py:
  • replaced replace_item(..., partition_key=...) with upsert_item(payload)
  • ensured payload id normalization before upsert
• Built and deployed image:
  • tutor108dev4p86ctacr.azurecr.io/tutor/questions-108dev:hotfix-putfix-20260318152023
• Container App revision:
  • latest revision tutor-questions-108dev--0000021 ready and serving traffic

Validation evidence:

• PUT https://tutor-108dev-apim.azure-api.net/api/questions/assemblies/asm-quest-fund1 -> HTTP/1.1 200 OK
• Response payload confirmed persisted assembly with expected graders (agent_id:version)
• GET /api/questions/assemblies returned healthy question assemblies list

Status: resolved in 108dev (break-glass window). Next standard path remains workflow-based deploy to main.