Skip to content

Conversation

@hannesrudolph
Copy link
Collaborator

Summary

Moves cache breakpoint placement from individual providers to a shared utility called from Task.ts before createMessage(). Messages arrive at providers pre-annotated with providerOptions, and the AI SDK routes the correct options to the active provider automatically.

Architecture

Task.ts message preparation:
  cleanConversationHistory
    → resolveCacheProviderOptions(provider, modelInfo, settings)
    → applyCacheBreakpoints(messages, options, config)
    → api.createMessage(systemPrompt, annotatedMessages, metadata)

The AI SDK's providerOptions namespacing ensures each provider only picks up its own cache hints — anthropic.cacheControl for Anthropic family, bedrock.cachePoint for Bedrock.

New Files

File Purpose
src/api/transform/prompt-cache.ts resolveCacheProviderOptions() — maps provider name → cache config; applyCacheBreakpoints() — annotates last N non-assistant batches
src/api/transform/__tests__/prompt-cache.spec.ts 14 test cases

Provider Changes

Provider Removed Preserved
anthropic.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
anthropic-vertex.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
minimax.ts targeting block + applyCacheControlToAiSdkMessages() systemProviderOptions in streamText()
bedrock.ts targeting block + applyCachePointsToAiSdkMessages() systemProviderOptions, usePromptCache, supportsAwsPromptCache()

Key Improvements

  1. Fixes tool message caching gap — Targets non-assistant batches (role: user + role: tool) instead of only role: user. After PR feat: implement ModelMessage storage layer with AI SDK response messages #11409, tool results are separate role: tool messages that now correctly receive cache breakpoints.

  2. Single source of truth — Cache strategy defined once in prompt-cache.ts with provider adapter mapping.

  3. Provider-specific config preserved — Bedrock: 3 breakpoints + anchor at ~1/3 for 20-block lookback coverage. Anthropic family: 2 breakpoints. Bedrock opt-in via awsUsePromptCache.

Not Changed

  • systemProviderOptions in all 4 providers
  • OpenAI Native promptCacheRetention (provider-level option, not per-message)
  • No settings types or UI changes (separate scope)

Stats

+426 / -216 lines • 14 new tests • 5,491 total tests pass • 0 regressions

Move cache breakpoint logic from individual providers to a shared utility
called from Task.ts before createMessage(). Messages arrive at providers
pre-annotated with providerOptions, and the AI SDK routes the correct
options to the active provider automatically.

New files:
- src/api/transform/prompt-cache.ts: resolveCacheProviderOptions() +
  applyCacheBreakpoints() with provider adapter mapping
- src/api/transform/__tests__/prompt-cache.spec.ts: 14 test cases

Changes per provider:
- anthropic.ts: removed targeting block + applyCacheControlToAiSdkMessages()
- anthropic-vertex.ts: same
- minimax.ts: same
- bedrock.ts: removed targeting block + applyCachePointsToAiSdkMessages()

Key improvements:
- Targets non-assistant batches (user + tool) instead of only role=user.
  After PR #11409, tool results are separate role=tool messages that now
  correctly receive cache breakpoints.
- Single source of truth: cache strategy defined once in prompt-cache.ts
- Provider-specific config preserved: Bedrock gets 3 breakpoints + anchor,
  Anthropic family gets 2 breakpoints

Preserved (untouched):
- systemProviderOptions in all providers' streamText() calls
- OpenAI Native promptCacheRetention (provider-level, not per-message)
- Bedrock usePromptCache opt-in + supportsAwsPromptCache()

5,491 tests pass, 0 regressions.
@dosubot dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Feb 11, 2026
@roomote
Copy link
Contributor

roomote bot commented Feb 11, 2026

Rooviewer Clock   See task

Clean refactor overall. One issue flagged around Bedrock's cache checkpoint budget.

  • Bedrock maxMessageBreakpoints = 3 + anchor can produce 5 total checkpoints (including system), exceeding the documented 4-checkpoint limit

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

applyCacheBreakpoints(
cleanConversationHistory,
cacheOptions,
isBedrock ? 3 : 2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The old bedrock code documented a hard limit of 4 cache checkpoints and budgeted them as 1 system + 2 trailing + 1 anchor = 4. Here, maxMessageBreakpoints = 3 for Bedrock plus useAnchor = true yields up to 3 trailing + 1 anchor = 4 message-level breakpoints, plus the system prompt checkpoint applied in bedrock.ts via systemProviderOptions. That totals 5, which exceeds the documented 4-checkpoint limit and could cause Bedrock API errors once a conversation is long enough to trigger the anchor. Changing the trailing count to 2 (matching the old behavior) would keep the total at 4.

Suggested change
isBedrock ? 3 : 2,
isBedrock ? 2 : 2,

Fix it with Roo Code or mention @roomote and request a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant