fix(mempool): tune for larger mempool size (lowered minrelaytxfee)#179
fix(mempool): tune for larger mempool size (lowered minrelaytxfee)#179
Conversation
There was a problem hiding this comment.
Pull request overview
This PR optimizes mempool performance for larger mempool sizes resulting from lowered minrelaytxfee. The main issue was that mempool evictions required O(n²) work scanning all scripthash histories, which blocked wallet operations and slowed wallet syncs.
Changes:
- Added per-transaction scripthash tracking to enable O(k) complexity pruning during evictions (where k is the number of scripts a transaction touches)
- Refactored backlog stats calculation to use a cloned slice instead of direct HashMap access for more efficient updates
- Made
TxFeeInfocloneable to support the backlog stats optimization
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/util/fees.rs | Added Clone derive to TxFeeInfo to support efficient backlog stats updates |
| src/new_index/mempool.rs | Added tx_scripthashes tracking map, implemented targeted pruning with prune_history_entries, and refactored backlog stats to use from_feeinfo_slice |
| flake.nix | Added libclang to build dependencies for rocksdb-sys/bindgen support |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
shesek
left a comment
shesek
left a comment
There was a problem hiding this comment.
Looks good! Added some minor comments.
DeviaVir
force-pushed
the
mempool-increase-patch
branch
2 times, most recently
from
f66f640 to
8a8a46c
Compare
|
@shesek I think I implemented all your feedback (thank you!) - mind taking a fresh look? |
|
If we want to further optimize for larger mempools, we could consider reviving the ZMQ-based syncer approach that I started working on a long time ago but ended up getting de-prioritized. The optimization for mempool evictions is actually something that I've been meaning to implement in the ZMQ syncer branch, it becomes much more crucial with ZMQ because mempool transactions are evicted individually based on the |
|
All looks good apart than Edit: hah I guess I should be careful about @-mentioning copilot, that PR he created seems pretty useless and not what I wanted >.< |
DeviaVir
force-pushed
the
mempool-increase-patch
branch
from
8a8a46c to
884a102
Compare
I saw "While the change appears valid (helping rocksdb-sys/bindgen find libclang), it seems unrelated to the mempool tuning described in the PR title and description. Consider whether this should be a separate PR or update the PR description to mention this build dependency fix." and yeah it is maybe not wrong but I needed that to build successfully so opted to leave it in. |
|
I think that the I guess there's no harm in keeping Other than that everything look great, thanks for the quick back-and-forth :-) |
The 0.00000100 BTC/kB minrelaytxfee keeps a much larger mempool. In electrs every mempool eviction/update takes a write lock and scans all scripthash histories to drop removed txids. With a larger mempool that became
O(n²)work and blocked walletget_history/listunspentreaders, showing up as ~2× slower wallet syncs.Code changes to cut the lock time:
mempool.rs: track, per tx, the exact scripthashes it touches (tx_scripthashes) and addprune_history_entriesso evictions prune only those buckets instead of scanning the whole history map. Complexity drops to O(k) where k is the tx’s touched scripts.mempool.rs: backlog stats now rebuild from a collected fee-info slice (from_feeinfo_slice), preparing for cheaper refresh and avoiding extra hash lookups.fees.rs: makeTxFeeInfo Cloneto support the above.These changes shorten write-lock sections during mempool updates and should directly reduce wallet sync latency under large mempools.