-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add api_format option to OpenAI/Azure targets (chat vs responses) #896
Copy link
Copy link
Closed
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work
Description
Problem
The Vercel AI SDK v3 (@ai-sdk/openai@3.0.41) defaults to the OpenAI Responses API (/responses), but not all OpenAI-compatible endpoints support it (e.g. GitHub Models returns 404). Currently we use a URL heuristic (baseURL.includes('api.openai.com')) to decide between .chat() and default — this works but is fragile.
Proposal
Add an api_format field to OpenAI and Azure target configs in targets.yaml:
targets:
- name: grader
provider: openai
api_format: chat # "chat" (default) | "responses"
base_url: https://models.github.ai/inference/v1
api_key: ${{ GH_MODELS_TOKEN }}
model: ${{ GH_MODELS_MODEL }}
- name: openai-direct
provider: openai
api_format: responses # opt in to Responses API for actual OpenAI
api_key: ${{ OPENAI_API_KEY }}
model: gpt-5-miniBehavior
api_format |
AI SDK method | Endpoint |
|---|---|---|
chat (default) |
openai.chat(model) |
/chat/completions |
responses |
openai(model) |
/responses |
Precedent
- promptfoo:
openai:chat:modelvsopenai:responses:model - pi-ai: each model has an
apifield (openai-responsesoropenai-completions) in its model registry
Why default to chat
- Universally supported by all OpenAI-compatible endpoints
- Responses API is only fully supported by
api.openai.comand Azure OpenAI - For eval grading (our primary use case), Chat Completions is sufficient
- GitHub Models, local proxies, and other OpenAI-compatible APIs only support
/chat/completions
Implementation
- Add
api_formattoBASE_TARGET_SCHEMAintargets.ts - Pass it through
OpenAIResolvedConfigandAzureResolvedConfig - Use it in
OpenAIProviderandAzureProviderconstructors to select.chat()vs default - Remove the URL heuristic (
isOpenAI = config.baseURL.includes('api.openai.com'))
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
in-progressClaimed by an agent — do not duplicate workClaimed by an agent — do not duplicate work