feat(ai-proxy): add defaults field for fallback model options by sihyeonn · Pull Request #12895 · apache/apisix

sihyeonn · 2026-01-13T04:21:21Z

Summary

Add a defaults field to the ai-proxy and ai-proxy-multi plugins for fallback model parameters that apply only when the client request omits them.

This is complementary to the existing options field:

options: always overrides request values (enforcement)
defaults: applied only when the client does not set the field (fallback)

Priority order: options > client request > defaults

Closes #13149

Changes

Add defaults field to schema for ai-proxy and ai-proxy-multi
Apply defaults in base.lua before stream detection
Include defaults.model in llm_model variable fallback
Add documentation (en/zh) for ai-proxy.md and ai-proxy-multi.md
Add test cases covering precedence rules

Baoyuantop

Supplement Documentation - Update ai-proxy.md and ai-proxy.md, adding documentation for the defaults field
Supplement ai-proxy Plugin Tests - Add test cases to ai-proxy.openai-compatible.t or relevant test files
Link to Related Issues or Supplement Requirement Context - Specify the source of use cases for this feature

sihyeonn · 2026-01-29T23:20:53Z

Thanks for the review! I've addressed your feedback:

Added documentation for the defaults field in both ai-proxy.md and ai-proxy-multi.md (en/zh)
Added test cases to ai-proxy.openai-compatible.t
This feature addresses a common use case where admins want to set fallback values (like max_tokens, temperature) without overriding user preferences. Unlike options which always overrides, defaults only applies when users don't specify values in their requests.

Baoyuantop · 2026-03-12T07:48:14Z

Hi @sihyeonn, since this PR is not associated with any issue and is a new feature, we suggest that we first communicate the detailed solution for this feature in an issue, and then push the code forward after the community agrees.

Copilot

Pull request overview

Adds support for a defaults configuration block to the AI proxy plugins so operators can provide fallback model parameters that apply only when the client request omits them (while still allowing options to always override).

Changes:

Introduces defaults in ai-proxy / ai-proxy-multi plugin schemas and documents the new field (EN/ZH).
Passes defaults through the proxy base into the OpenAI-compatible driver layer.
Adds test coverage for default-vs-user-vs-options precedence.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`apisix/plugins/ai-drivers/openai-base.lua`	Applies `model_defaults` into the outgoing request body before applying `model_options`.
`apisix/plugins/ai-proxy/base.lua`	Plumbs `defaults` into `extra_opts` for drivers.
`apisix/plugins/ai-proxy/schema.lua`	Adds `defaults` field to single and multi instance schemas.
`t/plugin/ai-proxy.openai-compatible.t`	Adds tests validating precedence of defaults vs request vs options.
`t/plugin/ai-proxy-multi.openai-compatible.t`	Adds equivalent precedence tests for ai-proxy-multi.
`docs/en/latest/plugins/ai-proxy.md`	Documents new `defaults` field for ai-proxy.
`docs/en/latest/plugins/ai-proxy-multi.md`	Documents new `instances.defaults` field for ai-proxy-multi.
`docs/zh/latest/plugins/ai-proxy.md`	Chinese documentation update for `defaults`.
`docs/zh/latest/plugins/ai-proxy-multi.md`	Chinese documentation update for `instances.defaults`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-12T07:48:52Z

+test-type: defaults
+--- response_body
+{"max_tokens":512,"model":"server-model","temperature":0.7}
+


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Already switched to response_body_like with regex in the previous commit to handle non-deterministic key ordering.

Copilot · 2026-03-12T07:48:52Z

+test-type: defaults
+--- response_body
+{"max_tokens":100,"model":"server-model","temperature":0.5}
+


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Copilot · 2026-03-12T07:48:52Z

+--- more_headers
+test-type: defaults
+--- response_body
+{"max_tokens":100,"model":"server-model","temperature":0.7}


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Copilot · 2026-03-12T07:48:52Z

+test-type: defaults
+--- response_body
+{"max_tokens":512,"model":"server-model","temperature":0.7}
+


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Copilot · 2026-03-12T07:48:53Z

+--- more_headers
+test-type: defaults
+--- response_body
+{"max_tokens":100,"model":"server-model","temperature":0.5}


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Copilot · 2026-03-12T07:48:53Z

+{ "messages": [ { "role": "user", "content": "hello" } ], "model": "user-model", "max_tokens": 100 }
+--- more_headers
+test-type: defaults
+--- response_body
+{"max_tokens":100,"model":"server-model","temperature":0.7}


These assertions compare the raw JSON string exactly, but the response is produced via cjson.safe.encode on a Lua table with string keys, where key order is not guaranteed. This can make the test flaky across Lua/cjson builds. Prefer asserting via response_body_like/response_body eval (regexes) or decode the JSON and compare fields rather than matching the exact serialized key order.

Copilot · 2026-03-12T07:48:53Z

+    -- defaults: apply only when not set in request
+    if extra_opts.model_defaults then
+        for opt, val in pairs(extra_opts.model_defaults) do
+            if request_table[opt] == nil then
+                request_table[opt] = val
+            end
+        end
+    end


Applying model_defaults here mutates request_table after apisix/plugins/ai-proxy/base.lua has already derived ctx.var.request_type and stream_options from the original request body. If defaults (or options) sets fields that affect control flow (e.g., stream), APISIX can end up sending a streamed request while still treating it as non-streaming for logging/metrics and without setting stream_options.include_usage. Consider applying defaults/options earlier (before the request_body.stream branch in before_proxy) or re-evaluating stream-related ctx vars after merging defaults/options.

Moved defaults application to before the stream detection block so ctx vars are set correctly. This was addressed in the previous commit.

Copilot · 2026-03-12T07:48:53Z

        local extra_opts = {
            name = ai_instance.name,
            endpoint = core.table.try_read_attr(ai_instance, "override", "endpoint"),
            model_options = ai_instance.options,
+            model_defaults = ai_instance.defaults,
            conf = ai_instance.provider_conf or {},


Now that ai_instance.defaults is passed through as extra_opts.model_defaults, fields like defaults.model can affect the effective model sent upstream, but before_proxy currently sets ctx.var.llm_model based only on options.model or request_body.model. This means logging/metrics may miss the actual model when it comes solely from defaults. Consider including ai_instance.defaults.model in the model selection logic (with the intended precedence) so ctx vars reflect the request actually sent.

Added defaults.model as a fallback in the llm_model assignment. Also removed the now-unnecessary model_defaults passthrough in extra_opts since defaults are applied directly to request_body in before_proxy.

moonming

Hi @sihyeonn, thank you for the ai-proxy defaults field proposal!

The concept of having a defaults field for fallback model options is interesting. A few points to discuss:

Naming clarity: defaults vs fallback — the name defaults implies these are baseline values that can be overridden, while fallback implies they're used when something is missing. Could you clarify the intended semantics? If these are values used when the request doesn't specify them, defaults is the right name.
Interaction with options: How does defaults interact with the existing options field? We need clear documentation on the precedence: request values > options (override) > defaults (fallback).
Schema documentation: Please ensure the schema includes clear descriptions for each field in defaults.

Looking forward to the design discussion! Thank you.

sihyeonn · 2026-03-18T06:05:33Z

Addressed all review feedback. Thanks for the reviews!

sihyeonn · 2026-03-19T03:08:11Z

@Baoyuantop Addressed all the review feedback — tests switched to regex matching, defaults applied before stream detection, and llm_model now includes defaults.model fallback. Ready for review.

Baoyuantop · 2026-03-19T06:59:38Z

Hi @sihyeonn, before we push this PR further, I hope you can answer these design questions first #12895 (review)

Baoyuantop · 2026-04-03T03:00:29Z

Hi @sihyeonn, please fix the code lint error and merge the master branch

Separate options and defaults behavior: - options: always override user request values - defaults: apply only when not set in user request This allows more flexible configuration where administrators can enforce certain values (via options) while providing sensible defaults for optional parameters. Priority order: options > client request > defaults Closes apache#13149 Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>

…erty Add explicit `model` property to model_defaults_schema for consistency with model_options_schema, and improve the description to clarify the fallback semantics (only applied when not set in request). Signed-off-by: Sihyeon Jang <sihyeon.jang@navercorp.com>

sihyeonn · 2026-04-16T23:31:53Z

Hi @moonming and @Baoyuantop, apologies for the very late response — I've been on an extended vacation and kept missing the review notifications. Sorry for the repeated delays!

1. Naming: defaults vs fallback

I'd like to keep the name defaults. fallback tends to connote error-recovery semantics, whereas defaults is the established term for "values that apply when not explicitly provided" (e.g., function default parameters, HTTP default headers). Since that's exactly the intent here, I think defaults is the clearer choice — and as you noted in the review, if these are values used when the request doesn't specify them, defaults is the right name.

2. Priority order

I believe the correct precedence is options > client request > defaults, not request values > options as written in the review.

options is an enforcement/override mechanism — it always wins regardless of what the client sends (admin enforces). defaults is the opposite: it only kicks in when the client omits the field (admin suggests). So the full priority chain is:

Priority	Source	Behavior
1 (highest)	`options`	Always overrides — admin enforcement
2	client request	What the client actually sent
3 (lowest)	`defaults`	Applied only when client omitted the field

3. Schema documentation

Good catch — I've added an explicit model property to the defaults schema (consistent with options) and improved the description to clarify the fallback semantics. See the latest commit.

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jan 13, 2026

Baoyuantop reviewed Jan 23, 2026

View reviewed changes

Baoyuantop added the wait for update wait for the author's response in this issue/PR label Jan 26, 2026

github-actions Bot added the user responded label Jan 29, 2026

sihyeonn requested a review from Baoyuantop January 29, 2026 23:31

Baoyuantop requested a review from Copilot March 12, 2026 07:44

Copilot started reviewing on behalf of Baoyuantop March 12, 2026 07:44 View session

Baoyuantop added discuss and removed user responded labels Mar 12, 2026

Copilot AI reviewed Mar 12, 2026

View reviewed changes

moonming requested changes Mar 16, 2026

View reviewed changes

github-actions Bot added the user responded label Mar 18, 2026

Baoyuantop removed wait for update wait for the author's response in this issue/PR user responded labels Mar 19, 2026

sihyeonn mentioned this pull request Apr 2, 2026

feat(ai-proxy): add defaults field for fallback model parameters #13149

Open

sihyeonn force-pushed the feat/sh-defaults branch from a8a92cf to c343d20 Compare April 3, 2026 07:27

sihyeonn force-pushed the feat/sh-defaults branch from cc62f9e to ced79c5 Compare April 16, 2026 22:56

github-actions Bot added the user responded label Apr 16, 2026

Conversation

sihyeonn commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

Baoyuantop left a comment

Choose a reason for hiding this comment

Uh oh!

sihyeonn commented Jan 29, 2026

Uh oh!

Baoyuantop commented Mar 12, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

sihyeonn Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

sihyeonn Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

Uh oh!

sihyeonn Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

moonming left a comment

Choose a reason for hiding this comment

Uh oh!

sihyeonn commented Mar 18, 2026

Uh oh!

sihyeonn commented Mar 19, 2026

Uh oh!

Baoyuantop commented Mar 19, 2026

Uh oh!

Baoyuantop commented Apr 3, 2026

Uh oh!

sihyeonn commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

sihyeonn commented Jan 13, 2026 •

edited

Loading

sihyeonn commented Apr 16, 2026 •

edited

Loading