(POC) Add monthly benchmarking results to the changelog#8733
(POC) Add monthly benchmarking results to the changelog#8733
Conversation
|
Newest code from mattermost has been published to preview environment for Git SHA 47d6ac9 |
agarciamontoro
left a comment
There was a problem hiding this comment.
This is great, @amyblais, thank you!
I know I'm not consistent in my wording in the reports, but now that we're adding it to our more visible docs, it makes sense to agree on a convention. I like the wording of the three last reports in the PR. Tweaking a couple of things to be more specific, we could use the following template:
Benchmarking test results showed no significant difference: a x% increase/decrease in the number of supported users for the new release, which lies within the
[-5%, +5%]prediction interval. View the full raw data and methodology in our Performance Reports repository.
I'm open to tweaking that, but consistency is important. Also, I think that starting with "results showed no significant difference" is good to deliver the most important information first.
I'll also adapt my future reports to the template decided here.
We're doing a great job benchmarking our releases and the test results are useful as they show consistent improvement. Proposal is to make the test results more visible for customer trust.