feat: add capped sampling rate increases#10715
Open
raphaelgavache wants to merge 2 commits intomasterfrom
Open
feat: add capped sampling rate increases#10715raphaelgavache wants to merge 2 commits intomasterfrom
raphaelgavache wants to merge 2 commits intomasterfrom
Conversation
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 60 metrics, 11 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065542
Total [baseline] (10.934 s) : 0, 10934382
Agent [candidate] (1.075 s) : 0, 1074851
Total [candidate] (10.904 s) : 0, 10904200
section appsec
Agent [baseline] (1.242 s) : 0, 1242076
Total [baseline] (10.944 s) : 0, 10944360
Agent [candidate] (1.245 s) : 0, 1244931
Total [candidate] (10.942 s) : 0, 10941886
section iast
Agent [baseline] (1.23 s) : 0, 1230268
Total [baseline] (11.155 s) : 0, 11155303
Agent [candidate] (1.238 s) : 0, 1238438
Total [candidate] (11.166 s) : 0, 11166284
section profiling
Agent [baseline] (1.192 s) : 0, 1192466
Total [baseline] (10.921 s) : 0, 10921327
Agent [candidate] (1.194 s) : 0, 1194406
Total [candidate] (10.956 s) : 0, 10956416
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.2 ms) : 0, 1200
crashtracking [candidate] (1.19 ms) : 0, 1190
BytebuddyAgent [baseline] (627.73 ms) : 0, 627730
BytebuddyAgent [candidate] (632.288 ms) : 0, 632288
AgentMeter [baseline] (29.108 ms) : 0, 29108
AgentMeter [candidate] (29.36 ms) : 0, 29360
GlobalTracer [baseline] (258.184 ms) : 0, 258184
GlobalTracer [candidate] (260.379 ms) : 0, 260379
AppSec [baseline] (32.881 ms) : 0, 32881
AppSec [candidate] (33.214 ms) : 0, 33214
Debugger [baseline] (64.567 ms) : 0, 64567
Debugger [candidate] (66.309 ms) : 0, 66309
Remote Config [baseline] (615.172 µs) : 0, 615
Remote Config [candidate] (621.198 µs) : 0, 621
Telemetry [baseline] (10.594 ms) : 0, 10594
Telemetry [candidate] (10.713 ms) : 0, 10713
Flare Poller [baseline] (4.532 ms) : 0, 4532
Flare Poller [candidate] (4.438 ms) : 0, 4438
section appsec
crashtracking [baseline] (1.19 ms) : 0, 1190
crashtracking [candidate] (1.188 ms) : 0, 1188
BytebuddyAgent [baseline] (659.364 ms) : 0, 659364
BytebuddyAgent [candidate] (661.181 ms) : 0, 661181
AgentMeter [baseline] (12.027 ms) : 0, 12027
AgentMeter [candidate] (12.031 ms) : 0, 12031
GlobalTracer [baseline] (259.087 ms) : 0, 259087
GlobalTracer [candidate] (260.311 ms) : 0, 260311
AppSec [baseline] (168.426 ms) : 0, 168426
AppSec [candidate] (168.024 ms) : 0, 168024
Debugger [baseline] (66.768 ms) : 0, 66768
Debugger [candidate] (67.029 ms) : 0, 67029
Remote Config [baseline] (677.568 µs) : 0, 678
Remote Config [candidate] (649.786 µs) : 0, 650
Telemetry [baseline] (9.403 ms) : 0, 9403
Telemetry [candidate] (9.162 ms) : 0, 9162
Flare Poller [baseline] (3.757 ms) : 0, 3757
Flare Poller [candidate] (3.647 ms) : 0, 3647
IAST [baseline] (25.275 ms) : 0, 25275
IAST [candidate] (25.621 ms) : 0, 25621
section iast
crashtracking [baseline] (1.201 ms) : 0, 1201
crashtracking [candidate] (1.229 ms) : 0, 1229
BytebuddyAgent [baseline] (794.113 ms) : 0, 794113
BytebuddyAgent [candidate] (800.406 ms) : 0, 800406
AgentMeter [baseline] (11.278 ms) : 0, 11278
AgentMeter [candidate] (11.31 ms) : 0, 11310
GlobalTracer [baseline] (247.397 ms) : 0, 247397
GlobalTracer [candidate] (248.384 ms) : 0, 248384
AppSec [baseline] (32.301 ms) : 0, 32301
AppSec [candidate] (33.423 ms) : 0, 33423
Debugger [baseline] (68.277 ms) : 0, 68277
Debugger [candidate] (67.699 ms) : 0, 67699
Remote Config [baseline] (535.046 µs) : 0, 535
Remote Config [candidate] (553.596 µs) : 0, 554
Telemetry [baseline] (8.683 ms) : 0, 8683
Telemetry [candidate] (8.635 ms) : 0, 8635
Flare Poller [baseline] (3.424 ms) : 0, 3424
Flare Poller [candidate] (3.49 ms) : 0, 3490
IAST [baseline] (27.168 ms) : 0, 27168
IAST [candidate] (27.107 ms) : 0, 27107
section profiling
crashtracking [baseline] (1.163 ms) : 0, 1163
crashtracking [candidate] (1.164 ms) : 0, 1164
BytebuddyAgent [baseline] (683.57 ms) : 0, 683570
BytebuddyAgent [candidate] (684.238 ms) : 0, 684238
AgentMeter [baseline] (8.579 ms) : 0, 8579
AgentMeter [candidate] (8.625 ms) : 0, 8625
GlobalTracer [baseline] (216.365 ms) : 0, 216365
GlobalTracer [candidate] (218.406 ms) : 0, 218406
AppSec [baseline] (32.461 ms) : 0, 32461
AppSec [candidate] (32.695 ms) : 0, 32695
Debugger [baseline] (67.074 ms) : 0, 67074
Debugger [candidate] (67.104 ms) : 0, 67104
Remote Config [baseline] (623.312 µs) : 0, 623
Remote Config [candidate] (604.96 µs) : 0, 605
Telemetry [baseline] (8.91 ms) : 0, 8910
Telemetry [candidate] (8.839 ms) : 0, 8839
Flare Poller [baseline] (3.773 ms) : 0, 3773
Flare Poller [candidate] (3.75 ms) : 0, 3750
ProfilingAgent [baseline] (99.147 ms) : 0, 99147
ProfilingAgent [candidate] (98.036 ms) : 0, 98036
Profiling [baseline] (99.729 ms) : 0, 99729
Profiling [candidate] (98.612 ms) : 0, 98612
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.069 s) : 0, 1069431
Total [baseline] (8.731 s) : 0, 8730665
Agent [candidate] (1.066 s) : 0, 1065833
Total [candidate] (8.774 s) : 0, 8773860
section iast
Agent [baseline] (1.23 s) : 0, 1230321
Total [baseline] (9.35 s) : 0, 9350099
Agent [candidate] (1.235 s) : 0, 1234574
Total [candidate] (9.378 s) : 0, 9377964
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.194 ms) : 0, 1194
crashtracking [candidate] (1.206 ms) : 0, 1206
BytebuddyAgent [baseline] (632.007 ms) : 0, 632007
BytebuddyAgent [candidate] (628.102 ms) : 0, 628102
AgentMeter [baseline] (29.248 ms) : 0, 29248
AgentMeter [candidate] (29.097 ms) : 0, 29097
GlobalTracer [baseline] (258.598 ms) : 0, 258598
GlobalTracer [candidate] (258.876 ms) : 0, 258876
AppSec [baseline] (33.359 ms) : 0, 33359
AppSec [candidate] (33.086 ms) : 0, 33086
Debugger [baseline] (63.574 ms) : 0, 63574
Debugger [candidate] (64.991 ms) : 0, 64991
Remote Config [baseline] (629.476 µs) : 0, 629
Remote Config [candidate] (615.755 µs) : 0, 616
Telemetry [baseline] (9.949 ms) : 0, 9949
Telemetry [candidate] (9.201 ms) : 0, 9201
Flare Poller [baseline] (4.666 ms) : 0, 4666
Flare Poller [candidate] (4.5 ms) : 0, 4500
section iast
crashtracking [baseline] (1.196 ms) : 0, 1196
crashtracking [candidate] (1.189 ms) : 0, 1189
BytebuddyAgent [baseline] (795.205 ms) : 0, 795205
BytebuddyAgent [candidate] (797.283 ms) : 0, 797283
AgentMeter [baseline] (11.297 ms) : 0, 11297
AgentMeter [candidate] (11.322 ms) : 0, 11322
GlobalTracer [baseline] (247.656 ms) : 0, 247656
GlobalTracer [candidate] (248.992 ms) : 0, 248992
AppSec [baseline] (33.045 ms) : 0, 33045
AppSec [candidate] (33.158 ms) : 0, 33158
Debugger [baseline] (66.453 ms) : 0, 66453
Debugger [candidate] (66.823 ms) : 0, 66823
Remote Config [baseline] (543.972 µs) : 0, 544
Remote Config [candidate] (547.02 µs) : 0, 547
Telemetry [baseline] (8.564 ms) : 0, 8564
Telemetry [candidate] (8.612 ms) : 0, 8612
Flare Poller [baseline] (3.455 ms) : 0, 3455
Flare Poller [candidate] (3.475 ms) : 0, 3475
IAST [baseline] (26.9 ms) : 0, 26900
IAST [candidate] (27.132 ms) : 0, 27132
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 19 metrics, 17 unstable metrics. Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (17.278 ms) : 17106, 17449
. : milestone, 17278,
appsec (18.692 ms) : 18503, 18880
. : milestone, 18692,
code_origins (17.814 ms) : 17635, 17992
. : milestone, 17814,
iast (17.742 ms) : 17566, 17917
. : milestone, 17742,
profiling (19.224 ms) : 19029, 19420
. : milestone, 19224,
tracing (17.699 ms) : 17524, 17874
. : milestone, 17699,
section candidate
no_agent (18.072 ms) : 17888, 18257
. : milestone, 18072,
appsec (18.638 ms) : 18446, 18830
. : milestone, 18638,
code_origins (17.986 ms) : 17806, 18166
. : milestone, 17986,
iast (18.046 ms) : 17864, 18227
. : milestone, 18046,
profiling (18.62 ms) : 18435, 18805
. : milestone, 18620,
tracing (17.518 ms) : 17343, 17692
. : milestone, 17518,
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.182 ms) : 1170, 1193
. : milestone, 1182,
iast (3.213 ms) : 3170, 3256
. : milestone, 3213,
iast_FULL (5.852 ms) : 5792, 5911
. : milestone, 5852,
iast_GLOBAL (3.657 ms) : 3591, 3722
. : milestone, 3657,
profiling (2.199 ms) : 2179, 2219
. : milestone, 2199,
tracing (1.754 ms) : 1740, 1768
. : milestone, 1754,
section candidate
no_agent (1.176 ms) : 1164, 1188
. : milestone, 1176,
iast (3.214 ms) : 3172, 3255
. : milestone, 3214,
iast_FULL (5.927 ms) : 5867, 5987
. : milestone, 5927,
iast_GLOBAL (3.693 ms) : 3629, 3757
. : milestone, 3693,
profiling (2.096 ms) : 2077, 2115
. : milestone, 2096,
tracing (1.778 ms) : 1762, 1794
. : milestone, 1778,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (15.089 s) : 15089000, 15089000
. : milestone, 15089000,
appsec (14.945 s) : 14945000, 14945000
. : milestone, 14945000,
iast (18.438 s) : 18438000, 18438000
. : milestone, 18438000,
iast_GLOBAL (17.825 s) : 17825000, 17825000
. : milestone, 17825000,
profiling (14.863 s) : 14863000, 14863000
. : milestone, 14863000,
tracing (14.767 s) : 14767000, 14767000
. : milestone, 14767000,
section candidate
no_agent (15.517 s) : 15517000, 15517000
. : milestone, 15517000,
appsec (15.127 s) : 15127000, 15127000
. : milestone, 15127000,
iast (17.638 s) : 17638000, 17638000
. : milestone, 17638000,
iast_GLOBAL (17.781 s) : 17781000, 17781000
. : milestone, 17781000,
profiling (14.966 s) : 14966000, 14966000
. : milestone, 14966000,
tracing (14.627 s) : 14627000, 14627000
. : milestone, 14627000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~866193c877, baseline=1.60.0-SNAPSHOT~1c44c6f9a7
dateFormat X
axisFormat %s
section baseline
no_agent (1.476 ms) : 1464, 1487
. : milestone, 1476,
appsec (3.793 ms) : 3570, 4017
. : milestone, 3793,
iast (2.257 ms) : 2188, 2327
. : milestone, 2257,
iast_GLOBAL (2.297 ms) : 2227, 2366
. : milestone, 2297,
profiling (2.073 ms) : 2018, 2128
. : milestone, 2073,
tracing (2.076 ms) : 2022, 2130
. : milestone, 2076,
section candidate
no_agent (1.482 ms) : 1470, 1494
. : milestone, 1482,
appsec (3.7 ms) : 3483, 3916
. : milestone, 3700,
iast (2.246 ms) : 2178, 2315
. : milestone, 2246,
iast_GLOBAL (2.3 ms) : 2231, 2370
. : milestone, 2300,
profiling (2.11 ms) : 2053, 2166
. : milestone, 2110,
tracing (2.056 ms) : 2003, 2110
. : milestone, 2056,
|
5 tasks
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the trace-agent is restarted, a rate of 100% is initially provided by the trace-agent, increasing dramatically the number of traces sampled. A rate could go suddenly from 0.1% to 100% and back to 0.1% when the trace-agent eventually computes the new sampling rate.
Passing system tests with link to RFC
In particular it is observed that when the agent restarts, the payload buffering that waits for new container tags breaches its memory limit and we send spans without container tags.
This PR applies a limit of sampling rate increases of x2 every 1s resulting in a x10 completed every 3-4s
1->100% takes 7s
0.1 -> 100% takes 10s
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.