feat(storage): Write-Optimized Merge (WOM) Tree — v1.8.0 by Copilot · Pull Request #4204 · makr-code/ThemisDB

Copilot · 2026-03-13T19:40:55Z

Implements WomTree, a Bε-tree-style write-optimized structure targeting 2–5× write amplification (vs. 10–30× for LSM), with full review-comment resolution and ASAN-clean test suite.

Description

Core (`include/storage/wom_tree.h` + `src/storage/wom_tree.cpp`)

Bε-tree design: internal nodes hold lazy write buffers; mutations flush downward only when buffer_size_bytes is exceeded. Key config: buffer_size_bytes (64 KiB default), fanout (16), leaf_capacity (256), max_buffered_entries, lazy_deletes.

WomTree::Config cfg;
cfg.buffer_size_bytes = 256 * 1024;  // ~3× write amp
cfg.fanout = 16;
WomTree tree(cfg);
tree.put("user:42", payload);
auto s = tree.stats();
// s.writeAmplification() ≥ 1.0, targets 2–5×

Review fixes applied:

std::shared_mutex: shared lock for get/contains/stats/scan/scanRange; exclusive for mutations — concurrent readers no longer block each other
lazy_deletes implemented: false → directRemove() descends to leaf and calls clearBufferedOpsForKey() on the path to prevent buffer resurrection after flush; true (default) → tombstone buffered lazily
writeAmplification() always ≥ 1.0: single-leaf fast path now counts bytes in stat_internal_bytes via applyOpToLeaf; stat doc updated
size() exact: doGet() checked before buffering each PUT; REMOVE always decrements (existence pre-verified); applyOpToLeaf intentionally does not touch stat_live_entries to avoid double-counting during flush propagation
fanout enforced: doOneInternalSplit + fixAllInternalOverflows recursively split overfull internal nodes after every doInsertOp, including root growth
scan()/scanRange() lock scope: snapshot under shared lock, invoke callback outside — deadlock-free when writers are called from within the callback
flushNode OOB fix: child_ops (sized to original child count) was indexed past bounds after maybeSplitChild inserted right-leaf siblings; fixed with splits_so_far offset

Tests (`tests/test_wom_tree.cpp` — 40 tests, `WomTreeFocusedTests`)

All 5 acceptance criteria covered plus dedicated tests for each review fix:

Group	Tests
AC-1: write amplification	`AC1_WriteAmplification_BelowLSMBaseline`, `WriteAmplification_AtLeastOneAfterSinglePut`
AC-2: update-heavy	`UpdateHeavy_ManyKeys_CorrectAfterOverwrite`
AC-3: lazy flush / compaction	`AC3_LazyFlush_*`, `AC3_Compact_ForcesAllBuffersToLeaves`
AC-4: space amplification	`AC4_StatsExpose_InternalBufferBytes`
AC-5: point-read trade-off	`AC5_PointRead_MultiLevelTraversal_StillCorrect`
`lazy_deletes=false`	`LazyDeletesFalse_*` (3 tests)
`size()` accuracy	`Size_AccurateWithBufferedOps`, `Size_AccurateAfterBufferedRemove`
fanout enforcement	`FanoutEnforced_InternalNodesSplitCorrectly`, `FanoutEnforced_TreeHeightBounded`
scan lock scope	`Scan_CallbackInvokedOutsideLock_NoDeadlock`
concurrent reads	`ConcurrentReaders_NoBlockingEachOther`

All 40 pass under AddressSanitizer + UBSanitizer, zero warnings.

Build & CI

cmake/CMakeLists.txt: wom_tree.cpp → THEMIS_CORE_SOURCES
tests/CMakeLists.txt: test_wom_tree_focused target
.github/workflows/wom-tree-ci.yml: path-scoped CI on GCC-12/Clang-15/GCC-13 using standard composite actions

Docs

src/storage/ROADMAP.md: WOM Tree marked [x]
src/storage/FUTURE_ENHANCEMENTS.md: status → ✅ Implemented (v1.8.0)

Type of Change

Testing

Unit tests added/updated
Integration tests added/updated
Manual testing performed

📚 Research & Knowledge (wenn applicable)

Diese PR basiert auf wissenschaftlichen Paper(s) oder Best Practices?
- Falls JA: Research-Dateien in /docs/research/ angelegt?
- Falls JA: Im Modul-README unter "Wissenschaftliche Grundlagen" verlinkt?
- Falls JA: In /docs/research/implementation_influence/ eingetragen?

Relevante Quellen:

Paper: M. A. Bender et al., "An Introduction to Bε-Trees and Write-Optimization," ;login: USENIX Magazine, 40(5):22–28, 2015 — basis for buffered-propagation design and WA bound O(log_B(N)/ε)
Best Practice:
Architecture Decision:

Checklist

Code follows project style guidelines
Self-review completed
Documentation updated (if needed)
No new warnings introduced

Original prompt

This section details on the original issue you should resolve

<issue_title>Write-Optimized Merge (WOM) Tree</issue_title>
<issue_description>### Context

This issue implements the roadmap item 'Write-Optimized Merge (WOM) Tree' for the storage domain. It is sourced from the consolidated roadmap under 🟢 Low Priority — Future (v1.9.0+) and targets milestone v1.8.0.

Primary detail section: Write-Optimized Merge (WOM) Tree

Goal

Deliver the scoped changes for Write-Optimized Merge (WOM) Tree in src/storage/ and complete the linked detail section in a release-ready state for v1.8.0.

Detailed Scope

Write-Optimized Merge (WOM) Tree

Priority: Low
Target Version: v1.8.0

Alternative to LSM-tree for write-heavy workloads.

Advantages:

Lower write amplification (2-5x vs 10-30x for LSM)

Better for update-heavy workloads

Reduced compaction overhead

Trade-offs:

Higher space amplification

Slower point reads

Acceptance Criteria

Lower write amplification (2-5x vs 10-30x for LSM)

Better for update-heavy workloads

Reduced compaction overhead

Higher space amplification

Slower point reads

Relationships

Roadmap row: Add comprehensive investor-grade monetary valuation analysis for ThemisDB #260 (🟢 Low Priority — Future (v1.9.0+))

Depends on: none identified during generation.

Part of: consolidated roadmap delivery tracking.

References

src/ROADMAP.md

src/storage/FUTURE_ENHANCEMENTS.md#write-optimized-merge-wom-tree

Source key: roadmap:260:storage:v1.8.0:write-optimized-merge-wom-tree

Generated from the consolidated source roadmap. Keep the roadmap and issue in sync when scope changes.

</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes Write-Optimized Merge (WOM) Tree #4090

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

…sue #260) Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

Copilot

Pull request overview

Adds a new in-memory storage primitive, WomTree (Write-Optimized Merge Tree), intended as a Bε-tree-style write-optimized alternative to LSM-based structures, along with focused unit tests and a dedicated CI workflow.

Changes:

Introduces WomTree public API + implementation (put/get/remove/contains/scan/scanRange/compact/flushOnce, stats, config validation).
Adds focused GTest suite and CMake target for WOM tree.
Adds path-scoped GitHub Actions workflow and updates storage roadmap/docs to mark WOM tree implemented.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`include/storage/wom_tree.h`	Public API, config + stats definitions, and module documentation.
`src/storage/wom_tree.cpp`	WOM tree core logic: buffering, flushing, leaf splits, scans, stats wiring.
`tests/test_wom_tree.cpp`	Focused unit tests covering core operations, stats, and concurrency scenarios.
`tests/CMakeLists.txt`	Adds `test_wom_tree_focused` executable and registers `WomTreeFocusedTests`.
`cmake/CMakeLists.txt`	Adds `src/storage/wom_tree.cpp` to core build sources.
`.github/workflows/wom-tree-ci.yml`	New WOM-tree-specific CI job to build/run focused tests across compilers/OSes.
`src/storage/ROADMAP.md`	Marks WOM Tree phase as completed in v1.8.0.
`src/storage/FUTURE_ENHANCEMENTS.md`	Updates WOM tree entry to “Implemented” and points to implementation files.

You can also share your feedback on Copilot code review. Take the survey.

include/storage/wom_tree.h

+ *
+ * ## Thread safety
+ *
+ * All public methods are thread-safe via an internal reader-writer mutex.


include/storage/wom_tree.h

+         * If true, deleted keys are tombstoned in-memory and purged
+         * asynchronously during the next flush/compact pass.
+         * If false, the delete takes effect immediately (higher read path
+         * cost for large buffers).


include/storage/wom_tree.h

+        /**
+         * @brief Estimated write-amplification factor.
+         *
+         * Ratio of bytes written internally (buffer propagation + leaf
+         * writes) to bytes written by the user.  A value close to 1.0
+         * means minimal write overhead; the WOM tree targets 2–5×
+         * compared to 10–30× for typical LSM trees.
+         *
+         * Returns 0.0 if no user bytes have been written yet.
+         */
+        double writeAmplification() const noexcept {
+            if (user_bytes_written == 0) return 0.0;
+            return static_cast<double>(internal_bytes_written) /
+                   static_cast<double>(user_bytes_written);
+        }


src/storage/wom_tree.cpp

+    void doInsertOp(Op op) {
+        stat_user_bytes.fetch_add(op.byteSize(), std::memory_order_relaxed);
+
+        if (root->is_leaf) {
+            // Single-leaf fast path.
+            if (op.type == OpType::PUT) {
+                bool was_present = (root->leafFind(op.key) != root->data.end());
+                root->leafApply(op);
+                if (!was_present) {
+                    stat_live_entries.fetch_add(1, std::memory_order_relaxed);
+                }
+            } else {
+                bool was_present = (root->leafFind(op.key) != root->data.end());
+                root->leafApply(op);
+                if (was_present) {
+                    stat_live_entries.fetch_sub(1, std::memory_order_relaxed);
+                }
+            }
+            maybeSplitRootLeaf();
+            return;
+        }
+
+        // Append to root buffer.
+        root->buffer_bytes += op.byteSize();
+        root->buffer.push_back(std::move(op));
+
+        // Flush root buffer if it exceeds the threshold.
+        if (root->buffer_bytes > config.buffer_size_bytes) {
+            flushNode(*root, 1);
+        }
+
+        // Global buffer pressure check.
+        if (config.max_buffered_entries > 0) {
+            size_t total_buf = countBufferedEntries(*root);
+            if (total_buf > config.max_buffered_entries) {
+                flushAll(*root, 1);
+            }
+        }
+    }


src/storage/wom_tree.cpp

+        parent.children.insert(parent.children.begin() + static_cast<ptrdiff_t>(ci + 1),
+                                std::move(right));
+
+        // If parent is now overfull, it will be handled by the parent's caller.


src/storage/wom_tree.cpp

+void WomTree::scan(
+    const std::function<bool(std::string_view, std::string_view)>& callback) const {
+    std::lock_guard<std::mutex> lk(impl_->mu);
+    std::map<std::string, std::string> materialized;
+    impl_->collectAllEntries(materialized);
+    for (const auto& [k, v] : materialized) {
+        if (!callback(k, v)) break;
+    }
+}


makr-code · 2026-03-14T16:28:47Z

@copilot apply changes based on the comments in this thread

Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

…ble-counting; add scan assertion Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

Copilot · 2026-03-14T18:01:29Z

@copilot apply changes based on the comments in this thread

All 6 review comments are addressed in commits 74d75d5 and e6d7da0:

Thread-safety: switched to std::shared_mutex; get/contains/stats/scan/scanRange use shared_lock, mutating ops use exclusive lock_guard.
lazy_deletes implemented: false → directRemove descends to the leaf and calls clearBufferedOpsForKey to prevent buffer resurrection after the next flush.
writeAmplification() fixed: single-leaf fast-path writes are now counted in stat_internal_bytes so WA ≥ 1.0 always; doc updated.
size() accurate: doGet is called before buffering each PUT to check key existence; REMOVE always decrements (existence pre-verified); doc updated.
fanout enforced: doOneInternalSplit + fixAllInternalOverflows split overfull internal nodes after every doInsertOp, including root growth.
scan() lock scope: tree is snapshotted under shared lock; callback is invoked outside the lock — confirmed deadlock-free by the new Scan_CallbackInvokedOutsideLock_NoDeadlock test that writes inside the callback.

Also fixed a heap-buffer-overflow in flushNode where child_ops (sized to the original child count) was indexed past its bounds after leaf splits inserted new right-leaf children — now uses a splits_so_far offset. Extracted applyOpToLeaf to deduplicate leaf-write logic, and corrected a latent stat_live_entries double-count it would have caused. All 40 tests pass cleanly under AddressSanitizer + UBSanitizer.

Initial plan

5e922e4

Copilot AI assigned Copilot and makr-code Mar 13, 2026

Copilot started work on behalf of makr-code March 13, 2026 19:59 View session

Copilot AI and others added 2 commits March 13, 2026 20:24

feat(storage): implement Write-Optimized Merge (WOM) Tree (v1.8.0, is…

d617efc

…sue #260) Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

fix(tests): restore missing StorageEngineDI status message in CMakeLists

33ce080

Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Implement Write-Optimized Merge (WOM) Tree for storage~~ feat(storage): Write-Optimized Merge (WOM) Tree — v1.8.0 Mar 13, 2026

Copilot AI requested a review from makr-code March 13, 2026 20:28

Copilot finished work on behalf of makr-code March 13, 2026 20:28

makr-code marked this pull request as ready for review March 14, 2026 06:13

makr-code requested a review from Copilot March 14, 2026 06:14

Copilot started reviewing on behalf of makr-code March 14, 2026 06:14 View session

Copilot AI reviewed Mar 14, 2026

View reviewed changes

Copilot started work on behalf of makr-code March 14, 2026 17:16 View session

Copilot AI and others added 2 commits March 14, 2026 17:55

fix(storage): address all 6 review comments on WOM Tree implementation

74d75d5

Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

refactor(storage): extract applyOpToLeaf helper; fix live_entries dou…

e6d7da0

…ble-counting; add scan assertion Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>

Copilot finished work on behalf of makr-code March 14, 2026 18:02

Copilot started work on behalf of makr-code March 14, 2026 18:18 View session

Copilot finished work on behalf of makr-code March 14, 2026 18:55

makr-code merged commit 3de3f39 into develop Mar 14, 2026
12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(storage): Write-Optimized Merge (WOM) Tree — v1.8.0#4204

feat(storage): Write-Optimized Merge (WOM) Tree — v1.8.0#4204
makr-code merged 5 commits intodevelopfrom
copilot/implement-write-optimized-merge-tree

Copilot AI commented Mar 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

makr-code commented Mar 14, 2026

Uh oh!

Copilot AI commented Mar 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Core (include/storage/wom_tree.h + src/storage/wom_tree.cpp)

Tests (tests/test_wom_tree.cpp — 40 tests, WomTreeFocusedTests)

Build & CI

Docs

Type of Change

Testing

📚 Research & Knowledge (wenn applicable)

Checklist

Goal

Detailed Scope

Write-Optimized Merge (WOM) Tree

Acceptance Criteria

Relationships

References

Comments on the Issue (you are @copilot in this section)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

makr-code commented Mar 14, 2026

Uh oh!

Copilot AI commented Mar 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 13, 2026 •

edited

Loading

Core (`include/storage/wom_tree.h` + `src/storage/wom_tree.cpp`)

Tests (`tests/test_wom_tree.cpp` — 40 tests, `WomTreeFocusedTests`)