Benchmark Runner QoL Improvements#3004
Merged
lerno merged 6 commits intoc3lang:masterfrom Mar 7, 2026
Merged
Conversation
Fixed merge conflicts of c3lang#2672 ``` From @NotsoanoNimus: I work with `compile-benchmark` quite often - only fair I help with its upkeep. Slapped this together in a couple hours, because I really need more consistent results and a way to visualize them, and the benchmark runner is a bit of a rat's nest. Changes, in no particular order: * Add a MEDIAN metric to the results * The _mean_ gives us throughput information, but it's too heavily skewed by a set's outliers. * The _median_ gives us an idea of performance expectations with a 50% probability. Put another way, the tested function has performed at or better than the median in 50% of samples taken. * Caveat: a sorted set is required. For high-volume iterations, this can get expensive. * Improve the output units and refactor that into `NanoDuration`, so it can be used elsewhere as desired * Provide some pretty colors, oooooo * Added CSV reporting option to get a resultant data-set * Not the most useful thing, but makes it easy to plug benchmark outputs into other software. Could probably be improved upon. and yeah I'm a boomer who likes CSV, sue me * Restructure the benchmark runtime to be more like the tester runtime * Fixed a divide-by-zero crash when the benchmark iteration count was <100 * Rudimentary command-line options to adjust benchmark options ```
Collaborator
|
I would drop the median in favour of standard deviation |
differentiate measure units by color (us, ms, s)
Member
Author
I added standard deviation by default (no need for a flag) |
Member
Author
Collaborator
|
I just fixed this. You should revert your changes. |
Collaborator
|
(And merge with the latest) |
38e9c80 to
4f00ea5
Compare
Collaborator
|
Out of scope for this one, but the benchmarking progress indicator is an old thing and works poorly, and I think it should be removed anyway: |
Collaborator
|
Another point of improvement would be to add a symbol for standard deviation instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixed merge conflicts of #2672
From original PR by @NotsoanoNimus :
I work with
compile-benchmarkquite often - only fair I help with itsupkeep. Slapped this together in a couple hours, because I really need
more consistent results and a way to visualize them, and the benchmark
runner is a bit of a rat's nest.
Changes, in no particular order:
heavily skewed by a set's outliers.
a 50% probability. Put another way, the tested function has performed at
or better than the median in 50% of samples taken.
this can get expensive.
NanoDuration,so it can be used elsewhere as desired
outputs into other software. Could probably be improved upon. and yeah
I'm a boomer who likes CSV, sue me
runtime