Skip to content

perf: add benchmarks for sim_round#242

Draft
Fraser999 wants to merge 1 commit intodylan/load-test-bundle-simfrom
fraser/eng-1956/benchmarks
Draft

perf: add benchmarks for sim_round#242
Fraser999 wants to merge 1 commit intodylan/load-test-bundle-simfrom
fraser/eng-1956/benchmarks

Conversation

@Fraser999
Copy link
Contributor

@Fraser999 Fraser999 commented Mar 12, 2026

Description

  • Adds Criterion benchmarks for sim_round() covering bundles (varying count, DB latency, concurrency, sender distribution), standalone transactions, complex bundle shapes (multi-tx, host txs), and failure paths (preflight and EVM revert).
  • Introduces LatencyDb wrapper around the in-memory test DB that sleeps on every DatabaseRef read, simulating real RPC round-trip costs. TestDb is now CacheDB<LatencyDb>, mirroring the production CacheDB<AlloyDB> layering, with existing tests using the LatencyDb with latency disabled to match existing behaviour.
  • Extends TestDbBuilder with with_contract() and with_latency() helpers.
Example benchmark output

The command:

cargo bench --bench sim

This was run on a Ryzen 9 7900X × 24, and ran for ~11m 15s:

sim_varying_simple_bundle_counts_no_db_latency/bundle_count/1
                        time:   [234.55 µs 235.87 µs 236.86 µs]
sim_varying_simple_bundle_counts_no_db_latency/bundle_count/10
                        time:   [3.2674 ms 3.2835 ms 3.3035 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
sim_varying_simple_bundle_counts_no_db_latency/bundle_count/100
                        time:   [33.786 ms 33.863 ms 33.987 ms]
sim_varying_simple_bundle_counts_no_db_latency/bundle_count/1000
                        time:   [343.52 ms 345.19 ms 346.62 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

sim_10_simple_bundles_varying_db_latency/latency/0ms
                        time:   [3.2407 ms 3.2535 ms 3.2647 ms]
Benchmarking sim_10_simple_bundles_varying_db_latency/latency/50ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 23.6s.
sim_10_simple_bundles_varying_db_latency/latency/50ms
                        time:   [2.3594 s 2.3597 s 2.3600 s]
Benchmarking sim_10_simple_bundles_varying_db_latency/latency/200ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 94.1s.
sim_10_simple_bundles_varying_db_latency/latency/200ms
                        time:   [9.4096 s 9.4099 s 9.4102 s]

sim_100_simple_bundles_no_db_latency_varying_concurrency/thread_count/1
                        time:   [25.504 ms 25.540 ms 25.563 ms]
sim_100_simple_bundles_no_db_latency_varying_concurrency/thread_count/2
                        time:   [28.179 ms 28.720 ms 29.240 ms]
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high severe
sim_100_simple_bundles_no_db_latency_varying_concurrency/thread_count/4
                        time:   [33.793 ms 33.951 ms 34.096 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
sim_100_simple_bundles_no_db_latency_varying_concurrency/thread_count/8
                        time:   [45.229 ms 45.770 ms 46.057 ms]
sim_100_simple_bundles_no_db_latency_varying_concurrency/thread_count/16
                        time:   [77.430 ms 77.688 ms 78.221 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

sim_100_simple_bundles_no_db_latency_varying_sender_count/sender_count/1
                        time:   [25.908 ms 26.031 ms 26.129 ms]
sim_100_simple_bundles_no_db_latency_varying_sender_count/sender_count/5
                        time:   [31.306 ms 31.576 ms 31.929 ms]
sim_100_simple_bundles_no_db_latency_varying_sender_count/sender_count/20
                        time:   [33.422 ms 33.503 ms 33.601 ms]
sim_100_simple_bundles_no_db_latency_varying_sender_count/sender_count/100
                        time:   [34.148 ms 34.665 ms 35.716 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

sim_100_bundles_no_db_latency_varying_txs_per_bundle/txs_per_bundle/1
                        time:   [33.863 ms 34.031 ms 34.157 ms]
sim_100_bundles_no_db_latency_varying_txs_per_bundle/txs_per_bundle/3
                        time:   [63.696 ms 63.886 ms 64.097 ms]
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high mild
Benchmarking sim_100_bundles_no_db_latency_varying_txs_per_bundle/txs_per_bundle/5: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 7.0s or enable flat sampling.
sim_100_bundles_no_db_latency_varying_txs_per_bundle/txs_per_bundle/5
                        time:   [92.419 ms 93.891 ms 95.434 ms]
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) high mild
  1 (10.00%) high severe
sim_100_bundles_no_db_latency_varying_txs_per_bundle/txs_per_bundle/10
                        time:   [166.92 ms 168.18 ms 169.37 ms]

sim_100_bundles_no_db_latency_varying_host_txs_per_bundle/host_txs_per_bundle/0
                        time:   [33.835 ms 33.965 ms 34.063 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
sim_100_bundles_no_db_latency_varying_host_txs_per_bundle/host_txs_per_bundle/1
                        time:   [21.841 µs 23.201 µs 25.496 µs]
sim_100_bundles_no_db_latency_varying_host_txs_per_bundle/host_txs_per_bundle/3
                        time:   [22.594 µs 24.331 µs 25.891 µs]
sim_100_bundles_no_db_latency_varying_host_txs_per_bundle/host_txs_per_bundle/5
                        time:   [25.124 µs 27.394 µs 29.828 µs]

sim_varying_tx_counts_with_no_db_latency/tx_count/1
                        time:   [265.82 µs 266.57 µs 267.39 µs]
sim_varying_tx_counts_with_no_db_latency/tx_count/10
                        time:   [3.5487 ms 3.5803 ms 3.6148 ms]
sim_varying_tx_counts_with_no_db_latency/tx_count/100
                        time:   [36.645 ms 36.836 ms 37.062 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
sim_varying_tx_counts_with_no_db_latency/tx_count/1000
                        time:   [385.71 ms 388.59 ms 390.86 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

sim_10_txs_with_varying_db_latency/latency/0ms
                        time:   [13.908 ms 15.909 ms 17.080 ms]
Benchmarking sim_10_txs_with_varying_db_latency/latency/50ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 23.9s.
sim_10_txs_with_varying_db_latency/latency/50ms
                        time:   [2.3710 s 2.3771 s 2.3831 s]
Found 3 outliers among 10 measurements (30.00%)
  2 (20.00%) low mild
  1 (10.00%) high mild
Benchmarking sim_10_txs_with_varying_db_latency/latency/200ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 94.4s.
sim_10_txs_with_varying_db_latency/latency/200ms
                        time:   [9.4087 s 9.4115 s 9.4164 s]
Found 2 outliers among 10 measurements (20.00%)
  1 (10.00%) high mild
  1 (10.00%) high severe

sim_10_simple_bundles_all_failing_preflight/latency/0ms
                        time:   [2.2787 µs 2.3106 µs 2.3489 µs]
Benchmarking sim_10_simple_bundles_all_failing_preflight/latency/50ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 5.0s.
sim_10_simple_bundles_all_failing_preflight/latency/50ms
                        time:   [500.62 ms 500.66 ms 500.73 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe
Benchmarking sim_10_simple_bundles_all_failing_preflight/latency/200ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 20.0s.
sim_10_simple_bundles_all_failing_preflight/latency/200ms
                        time:   [2.0006 s 2.0007 s 2.0007 s]

sim_10_simple_bundles_all_failing_execution/latency/0ms
                        time:   [343.26 µs 346.63 µs 350.19 µs]
sim_10_simple_bundles_all_failing_execution/latency/50ms
                        time:   [401.17 ms 401.22 ms 401.27 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Benchmarking sim_10_simple_bundles_all_failing_execution/latency/200ms: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 16.0s.
sim_10_simple_bundles_all_failing_execution/latency/200ms
                        time:   [1.6012 s 1.6013 s 1.6015 s]
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) high mild

Related Issue

Closes ENG-1956.

Testing

  • make fmt passes
  • make clippy passes
  • make test passes
  • New tests added (if applicable)

Copy link
Contributor Author

Fraser999 commented Mar 12, 2026

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant