Add ChatMessageExtensions with UsePromptCaching #4340
Add ChatMessageExtensions with UsePromptCaching #4340gcarney-ai wants to merge 4 commits intoaws:developmentfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a public convenience extension to enable Bedrock prompt caching on Microsoft.Extensions.AI.ChatMessage instances in the Bedrock MEAI extensions package, along with the required DevConfig entry for release automation.
Changes:
- Add
ChatMessageExtensions.UsePromptCachingto stamp a cache point intoChatMessage.AdditionalProperties. - Add a DevConfig entry marking a minor extension update and providing a changelog message.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| generator/.DevConfigs/a1508acc-bb92-4ee6-a356-9b1a82d506b2.json | Declares a minor extension change and changelog message for release automation. |
| extensions/src/AWSSDK.Extensions.Bedrock.MEAI/ChatMessageExtensions.cs | Introduces the UsePromptCaching extension that adds a cache point marker to chat messages. |
You can also share your feedback on Copilot code review. Take the survey.
| public static ChatMessage UsePromptCaching(this ChatMessage message, CacheTTL? cacheTTL = null) | ||
| { | ||
| cacheTTL ??= CacheTTL.FIVE_MINUTES; |
| cacheTTL ??= CacheTTL.FIVE_MINUTES; | ||
| message.AdditionalProperties ??= []; | ||
|
|
||
| message.AdditionalProperties.Add(nameof(ContentBlock.CachePoint), new CachePointBlock { Type = CachePointType.Default, Ttl = cacheTTL }); |
There was a problem hiding this comment.
Great catch. Converted to indexing directly
| /// <remarks> | ||
| /// This method adds a cache point marker to the message's additional properties. Prompt caching | ||
| /// allows Bedrock to cache the prompt tokens for reuse in subsequent requests, reducing costs | ||
| /// and latency. The model must support prompt caching and the message must be within the minimum token threshold for caching. |
|
@gcarney-ai Other than the testing, can you review Copilot's comments? I think the unresolved ones make sense. |
|
I'm going to close this PR as we haven't had any activity. If you'd like, please feel free to reopen and address Copilot's feedback. |
Description
I have added a new file named
ChatMessageExtensionswith a single extension:UsePromptCaching. This change creates AdditionalProperties on the message if it is not already populated, and adds CachePoint ContentBlock. I used the Prompt caching for faster model inference for reference. TheUsePrompCachingmethod takes an optional parameterCacheTTLwhich allows users to pass in either 5Min or 1Hr...defaulting to 5 min.Motivation and Context
Prompt caching is a crucial piece for both latency and cost optimization, by placing it in a public extension, it enables discovery and reduces duplicate code written by consumers.
Testing
No existing test fixtures for this project. But I manually tested each case.
Screenshots (if appropriate)
Types of changes
Checklist
License