[ISSUE #9926] Optimize memory allocation in parsePublishMessageQueues#10206
Open
daguimu wants to merge 1 commit intoapache:developfrom
Open
[ISSUE #9926] Optimize memory allocation in parsePublishMessageQueues#10206daguimu wants to merge 1 commit intoapache:developfrom
daguimu wants to merge 1 commit intoapache:developfrom
Conversation
…Queues Skip list creation entirely when namespace is empty (most common case), and pre-allocate ArrayList with known capacity when namespace stripping is needed. Also cache the namespace value to avoid repeated method calls. Fixes apache#9926
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under high throughput workloads (e.g., 50K LMQ tps),
parsePublishMessageQueuescauses massive memory allocation due to:ArrayList(with default capacity 10) for every call, even when namespace is emptygetNamespace()calls per queue elementArrayListgrowth/copy when the list size exceeds initial capacityProfiling by the issue reporter showed this method as a significant source of memory allocation overhead.
Root Cause
The method unconditionally creates a new list and copies all queues with namespace stripping, even when no namespace is configured (the most common deployment scenario). The
ArrayListis also created with default capacity instead of being pre-sized.Fix
messageQueueListdirectly, avoiding all allocationnew ArrayList<>(messageQueueList.size())to avoid internal array resizinggetNamespace()call outside the loop to avoid repeated method invocationsTests Added
testParsePublishMessageQueuesReturnsOriginalListWhenNamespaceEmpty— Verifies same list instance is returned when namespace is empty stringtestParsePublishMessageQueuesReturnsOriginalListWhenNamespaceNull— Verifies same list instance is returned when namespace is nulltestParsePublishMessageQueuesStripsNamespace— Verifies namespace is correctly stripped when presenttestParsePublishMessageQueuesPreservesOrderAndSize— Verifies correct behavior with 100 queuesImpact
Fixes #9926