Skip to content

Conversation

@jack-berg
Copy link
Member

@jack-berg jack-berg commented Feb 12, 2026

  • Switch delta sync metric storage coordination mechanism from AtomicLong to a striped set of ReentrantLocks.
  • Switch DoubleExplicitBucketHistogramAggregator locking mechanism to use a striped set of cells approach.
  • The extra work needed to select a lock / cell slows down the single threaded / no contention performance, but significantly improves the performance under contention.
  • I tried a bunch of different techniques over weeks and this yielded the best results. For now...
  • The new more thorough stress test in Sync instrument stress test #7986 gives the confidence to try techniques like this without breaking things

Summary view of perf changes to MetricRecordBenchmark ops/s on my machine:

Benchmark Temporality Cardinality Instrument Type Base Score After Score Change %
threads1 DELTA 1 COUNTER_SUM 13268.943 8113.116 -38.9%
threads1 DELTA 1 UP_DOWN_COUNTER_SUM 12117.078 7116.562 -41.3%
threads1 DELTA 1 GAUGE_LAST_VALUE 3413.938 3579.052 +4.8%
threads1 DELTA 1 HISTOGRAM_EXPLICIT 6979.876 5108.527 -26.8%
threads1 DELTA 1 HISTOGRAM_BASE2_EXPONENTIAL 4227.734 3767.116 -10.9%
threads1 DELTA 100 COUNTER_SUM 8385.390 7705.679 -8.1%
threads1 DELTA 100 UP_DOWN_COUNTER_SUM 8977.006 7870.198 -12.3%
threads1 DELTA 100 GAUGE_LAST_VALUE 2795.431 2998.816 +7.3%
threads1 DELTA 100 HISTOGRAM_EXPLICIT 7254.371 4143.646 -42.9%
threads1 DELTA 100 HISTOGRAM_BASE2_EXPONENTIAL 3777.950 3668.988 -2.9%
threads1 CUMULATIVE 1 COUNTER_SUM 14821.424 15211.867 +2.6%
threads1 CUMULATIVE 1 UP_DOWN_COUNTER_SUM 15744.303 15272.696 -3.0%
threads1 CUMULATIVE 1 GAUGE_LAST_VALUE 6177.182 7315.585 +18.4%
threads1 CUMULATIVE 1 HISTOGRAM_EXPLICIT 8580.816 6646.129 -22.5%
threads1 CUMULATIVE 1 HISTOGRAM_BASE2_EXPONENTIAL 4229.879 4202.725 -0.6%
threads1 CUMULATIVE 100 COUNTER_SUM 8983.140 8553.166 -4.8%
threads1 CUMULATIVE 100 UP_DOWN_COUNTER_SUM 9143.542 9148.823 +0.1%
threads1 CUMULATIVE 100 GAUGE_LAST_VALUE 8019.513 8658.896 +8.0%
threads1 CUMULATIVE 100 HISTOGRAM_EXPLICIT 7110.039 5590.956 -21.4%
threads1 CUMULATIVE 100 HISTOGRAM_BASE2_EXPONENTIAL 4061.643 4061.193 -0.0%
threads4 DELTA 1 COUNTER_SUM 1459.197 3642.403 +149.6%
threads4 DELTA 1 UP_DOWN_COUNTER_SUM 1454.658 2202.933 +51.4%
threads4 DELTA 1 GAUGE_LAST_VALUE 1276.315 1886.270 +47.8%
threads4 DELTA 1 HISTOGRAM_EXPLICIT 956.358 2254.040 +135.6%
threads4 DELTA 1 HISTOGRAM_BASE2_EXPONENTIAL 1128.382 1623.778 +43.9%
threads4 DELTA 100 COUNTER_SUM 1099.899 4987.529 +353.5%
threads4 DELTA 100 UP_DOWN_COUNTER_SUM 1032.450 4880.035 +372.7%
threads4 DELTA 100 GAUGE_LAST_VALUE 1095.108 3816.537 +248.5%
threads4 DELTA 100 HISTOGRAM_EXPLICIT 1211.766 5494.981 +353.5%
threads4 DELTA 100 HISTOGRAM_BASE2_EXPONENTIAL 1079.769 3385.470 +213.6%
threads4 CUMULATIVE 1 COUNTER_SUM 3909.910 6061.089 +55.0%
threads4 CUMULATIVE 1 UP_DOWN_COUNTER_SUM 4572.543 6635.251 +45.1%
threads4 CUMULATIVE 1 GAUGE_LAST_VALUE 1419.921 1971.709 +38.9%
threads4 CUMULATIVE 1 HISTOGRAM_EXPLICIT 1281.900 3745.206 +192.1%
threads4 CUMULATIVE 1 HISTOGRAM_BASE2_EXPONENTIAL 1049.875 1818.146 +73.2%
threads4 CUMULATIVE 100 COUNTER_SUM 4534.466 7510.675 +65.7%
threads4 CUMULATIVE 100 UP_DOWN_COUNTER_SUM 5665.878 6670.130 +17.7%
threads4 CUMULATIVE 100 GAUGE_LAST_VALUE 4073.034 6777.530 +66.4%
threads4 CUMULATIVE 100 HISTOGRAM_EXPLICIT 3089.665 9629.410 +211.7%
threads4 CUMULATIVE 100 HISTOGRAM_BASE2_EXPONENTIAL 3083.684 5692.648 +84.6%
Raw benchmark output before and after

Before (on my machine):

Benchmark                                          (aggregationTemporality)  (cardinality)  (instrumentTypeAndAggregation)   Mode  Cnt      Score      Error   Units
MetricRecordBenchmark.threads1                                        DELTA              1                     COUNTER_SUM  thrpt    5  13268.943 ±   65.386   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1             UP_DOWN_COUNTER_SUM  thrpt    5  12117.078 ±  238.182   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1                GAUGE_LAST_VALUE  thrpt    5   3413.938 ±   16.742   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1              HISTOGRAM_EXPLICIT  thrpt    5   6979.876 ±  801.319   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   4227.734 ±   53.597   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100                     COUNTER_SUM  thrpt    5   8385.390 ±  293.904   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100             UP_DOWN_COUNTER_SUM  thrpt    5   8977.006 ±  305.210   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100                GAUGE_LAST_VALUE  thrpt    5   2795.431 ±   36.730   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100              HISTOGRAM_EXPLICIT  thrpt    5   7254.371 ±  696.852   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   3777.950 ±   27.756   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1                     COUNTER_SUM  thrpt    5  14821.424 ± 1005.117   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1             UP_DOWN_COUNTER_SUM  thrpt    5  15744.303 ±  756.500   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1                GAUGE_LAST_VALUE  thrpt    5   6177.182 ± 1165.503   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1              HISTOGRAM_EXPLICIT  thrpt    5   8580.816 ±  248.202   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   4229.879 ±  163.105   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100                     COUNTER_SUM  thrpt    5   8983.140 ±  509.650   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100             UP_DOWN_COUNTER_SUM  thrpt    5   9143.542 ±  290.350   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100                GAUGE_LAST_VALUE  thrpt    5   8019.513 ±  599.203   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100              HISTOGRAM_EXPLICIT  thrpt    5   7110.039 ±  112.877   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   4061.643 ±   85.223   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1                     COUNTER_SUM  thrpt    5   1459.197 ±   79.876   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1             UP_DOWN_COUNTER_SUM  thrpt    5   1454.658 ±  268.251   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1                GAUGE_LAST_VALUE  thrpt    5   1276.315 ±  504.427   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1              HISTOGRAM_EXPLICIT  thrpt    5    956.358 ±   12.636   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   1128.382 ±  156.809   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100                     COUNTER_SUM  thrpt    5   1099.899 ±  269.155   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100             UP_DOWN_COUNTER_SUM  thrpt    5   1032.450 ±  262.733   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100                GAUGE_LAST_VALUE  thrpt    5   1095.108 ±  106.812   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100              HISTOGRAM_EXPLICIT  thrpt    5   1211.766 ±  378.141   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   1079.769 ±  136.818   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1                     COUNTER_SUM  thrpt    5   3909.910 ±  121.044   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1             UP_DOWN_COUNTER_SUM  thrpt    5   4572.543 ±  545.694   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1                GAUGE_LAST_VALUE  thrpt    5   1419.921 ±  148.174   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1              HISTOGRAM_EXPLICIT  thrpt    5   1281.900 ±   50.942   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   1049.875 ±   11.795   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100                     COUNTER_SUM  thrpt    5   4534.466 ± 1372.262   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100             UP_DOWN_COUNTER_SUM  thrpt    5   5665.878 ± 1510.865   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100                GAUGE_LAST_VALUE  thrpt    5   4073.034 ± 1132.285   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100              HISTOGRAM_EXPLICIT  thrpt    5   3089.665 ±  490.095   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   3083.684 ±  530.229   ops/s

After (on my machine):

Benchmark                                          (aggregationTemporality)  (cardinality)  (instrumentTypeAndAggregation)   Mode  Cnt      Score      Error   Units
MetricRecordBenchmark.threads1                                        DELTA              1                     COUNTER_SUM  thrpt    5   8113.116 ±  282.473   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1             UP_DOWN_COUNTER_SUM  thrpt    5   7116.562 ±   91.245   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1                GAUGE_LAST_VALUE  thrpt    5   3579.052 ±  173.461   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1              HISTOGRAM_EXPLICIT  thrpt    5   5108.527 ±  345.142   ops/s
MetricRecordBenchmark.threads1                                        DELTA              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   3767.116 ±  200.985   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100                     COUNTER_SUM  thrpt    5   7705.679 ±  170.463   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100             UP_DOWN_COUNTER_SUM  thrpt    5   7870.198 ±  836.084   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100                GAUGE_LAST_VALUE  thrpt    5   2998.816 ±    2.615   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100              HISTOGRAM_EXPLICIT  thrpt    5   4143.646 ±  160.402   ops/s
MetricRecordBenchmark.threads1                                        DELTA            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   3668.988 ±   98.559   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1                     COUNTER_SUM  thrpt    5  15211.867 ±   67.059   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1             UP_DOWN_COUNTER_SUM  thrpt    5  15272.696 ±  227.497   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1                GAUGE_LAST_VALUE  thrpt    5   7315.585 ±  349.701   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1              HISTOGRAM_EXPLICIT  thrpt    5   6646.129 ±  798.034   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   4202.725 ±  265.219   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100                     COUNTER_SUM  thrpt    5   8553.166 ±  158.919   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100             UP_DOWN_COUNTER_SUM  thrpt    5   9148.823 ±  243.243   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100                GAUGE_LAST_VALUE  thrpt    5   8658.896 ±  176.335   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100              HISTOGRAM_EXPLICIT  thrpt    5   5590.956 ±  102.107   ops/s
MetricRecordBenchmark.threads1                                   CUMULATIVE            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   4061.193 ±   54.127   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1                     COUNTER_SUM  thrpt    5   3642.403 ±  250.120   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1             UP_DOWN_COUNTER_SUM  thrpt    5   2202.933 ±  894.319   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1                GAUGE_LAST_VALUE  thrpt    5   1886.270 ±  557.925   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1              HISTOGRAM_EXPLICIT  thrpt    5   2254.040 ±  172.336   ops/s
MetricRecordBenchmark.threads4                                        DELTA              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   1623.778 ±   77.203   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100                     COUNTER_SUM  thrpt    5   4987.529 ±  777.833   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100             UP_DOWN_COUNTER_SUM  thrpt    5   4880.035 ±  417.418   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100                GAUGE_LAST_VALUE  thrpt    5   3816.537 ±  372.760   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100              HISTOGRAM_EXPLICIT  thrpt    5   5494.981 ± 1226.019   ops/s
MetricRecordBenchmark.threads4                                        DELTA            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   3385.470 ± 1267.804   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1                     COUNTER_SUM  thrpt    5   6061.089 ±  488.477   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1             UP_DOWN_COUNTER_SUM  thrpt    5   6635.251 ±  396.261   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1                GAUGE_LAST_VALUE  thrpt    5   1971.709 ±  271.515   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1              HISTOGRAM_EXPLICIT  thrpt    5   3745.206 ±  463.504   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE              1     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   1818.146 ±   64.783   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100                     COUNTER_SUM  thrpt    5   7510.675 ± 2174.329   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100             UP_DOWN_COUNTER_SUM  thrpt    5   6670.130 ± 3913.191   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100                GAUGE_LAST_VALUE  thrpt    5   6777.530 ± 2874.081   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100              HISTOGRAM_EXPLICIT  thrpt    5   9629.410 ± 1416.613   ops/s
MetricRecordBenchmark.threads4                                   CUMULATIVE            100     HISTOGRAM_BASE2_EXPONENTIAL  thrpt    5   5692.648 ± 1505.061   ops/s

@jack-berg jack-berg requested a review from a team as a code owner February 12, 2026 20:02
// (AggregatorHolder), and so if a recording thread encounters an odd value,
// all it needs to do is release the "read lock" it just obtained (decrementing by 2),
// and then grab and record against the new current interval (AggregatorHolder).
private final AtomicInteger activeRecordingThreads = new AtomicInteger(0);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: update javadoc comments to reflect current design before merging

forThread().unlock();
}

@SuppressWarnings("ThreadPriorityCheck")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: remove leftover annotation from a previous revision

@codecov
Copy link

codecov bot commented Feb 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.21%. Comparing base (0a1a2ce) to head (68238cb).

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #8077      +/-   ##
============================================
+ Coverage     90.20%   90.21%   +0.01%     
- Complexity     7594     7595       +1     
============================================
  Files           841      841              
  Lines         22915    22943      +28     
  Branches       2290     2296       +6     
============================================
+ Hits          20670    20698      +28     
- Misses         1529     1531       +2     
+ Partials        716      714       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant