Skip to content

feat(storage): BlobRedundancyManager — implement RocksDB EventListener for SST deletion#4201

Open
Copilot wants to merge 2 commits intodevelopfrom
copilot/implement-rocksdb-event-listener
Open

feat(storage): BlobRedundancyManager — implement RocksDB EventListener for SST deletion#4201
Copilot wants to merge 2 commits intodevelopfrom
copilot/implement-rocksdb-event-listener

Conversation

Copy link
Contributor

Copilot AI commented Mar 13, 2026

createRocksDBListener() unconditionally returned "RocksDB listener not implemented" and OnTableFileDeleted() was a no-op, leaving the redundancy manager blind to compaction-driven SST deletions and unable to trigger replication for blobs that lost their backing file.

Changes

Core implementation

  • createRocksDBListener(): returns a real RocksDBBlobListener instead of an error
  • notifySSTFileDeleted(file_path) (new public method): write-locks blobs_, marks every BlobLocation whose path matches as is_healthy = false, collects affected blob IDs, and drains them into repair_queue_ with repair_cv_.notify_all()
  • RocksDBBlobListener::OnTableFileDeleted(): delegates to notifySSTFileDeleted(info.file_path) — was previously a logging stub

Listener registration

  • RocksDBWrapper::addEventListener(shared_ptr<EventListener>) (new): pushes into options_->listeners before open() so callers can wire the blob listener at DB-open time
  • Added #include <rocksdb/listener.h> and class EventListener forward-declaration to rocksdb_wrapper

Tests (tests/test_raid_redundancy.cpp)

  • CreateRocksDBListenerSucceeds — factory returns non-null
  • NotifySSTFileDeletedMarksLocationUnhealthy — primary location transitions to unhealthy
  • NotifySSTFileDeletedUnknownPathIsNoOp — unknown path does not crash or mutate state
  • NotifySSTFileDeletedOnlyAffectsMatchingBlobs — sibling blob stays healthy
  • RocksDBListenerOnTableFileDeletedTriggersReplication — end-to-end: OnTableFileDeleted via factory listener marks location unhealthy

Build / CI

  • tests/CMakeLists.txt: test_blob_redundancy_event_listener_focused target running BlobRedundancyManagerTest.*
  • .github/workflows/blob-redundancy-event-listener-ci.yml: matrix (gcc-12, clang-15, gcc-13) scoped to the touched files

Type of Change

  • Bug fix
  • New feature
  • Refactoring
  • Documentation
  • Other:

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed

📚 Research & Knowledge (wenn applicable)

  • Diese PR basiert auf wissenschaftlichen Paper(s) oder Best Practices?
    • Falls JA: Research-Dateien in /docs/research/ angelegt?
    • Falls JA: Im Modul-README unter "Wissenschaftliche Grundlagen" verlinkt?
    • Falls JA: In /docs/research/implementation_influence/ eingetragen?

Relevante Quellen:

  • Paper:
  • Best Practice:
  • Architecture Decision:

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Documentation updated (if needed)
  • No new warnings introduced
Original prompt

This section details on the original issue you should resolve

<issue_title>BlobRedundancyManager: Implement RocksDB Event Listener</issue_title>
<issue_description>### Context

This issue implements the roadmap item 'BlobRedundancyManager: Implement RocksDB Event Listener' for the storage domain. It is sourced from the consolidated roadmap under 🟢 Low Priority — Future (v1.9.0+) and targets milestone v1.8.0.

Primary detail section: BlobRedundancyManager: Implement RocksDB Event Listener

Goal

Deliver the scoped changes for BlobRedundancyManager: Implement RocksDB Event Listener in src/storage/ and complete the linked detail section in a release-ready state for v1.8.0.

Detailed Scope

BlobRedundancyManager: Implement RocksDB Event Listener

Priority: Low
Target Version: v1.8.0

blob_redundancy_manager.cpp line 768: "RocksDB listener not implemented" — the redundancy manager cannot react to RocksDB compaction events (SST file deletions) to trigger re-replication of blobs that lose their storage backing.

Implementation Notes:

  • [ ] Implement a BlobRedundancyEventListener subclassing rocksdb::EventListener; override OnTableFileDeleted to trigger re-replication of any blobs whose backing SST was deleted.
  • [ ] Register the listener via rocksdb::Options::listeners at database open time.

Priority: High
Target Version: v1.7.0

Raft-based distributed transactions across multiple nodes.

Features:

  • Two-phase commit (2PC) protocol
  • Raft consensus for transaction coordination
  • Cross-shard atomic operations
  • Automatic deadlock detection

API:

DistributedTransactionManager dtx_manager(nodes);
auto tx = dtx_manager.beginDistributedTransaction();

// Write to multiple shards
tx->put("shard1:key1", "value1");
tx->put("shard2:key2", "value2");

// Commit atomically across shards
tx->commit();  // 2PC protocol

Use Cases:

  • Multi-tenant data isolation
  • Geographic data distribution
  • Horizontal scaling

Acceptance Criteria

  • Implement a BlobRedundancyEventListener subclassing rocksdb::EventListener; override OnTableFileDeleted to trigger re-replication of any blobs whose backing SST was deleted.
  • Register the listener via rocksdb::Options::listeners at database open time.
  • Two-phase commit (2PC) protocol
  • Raft consensus for transaction coordination
  • Cross-shard atomic operations
  • Automatic deadlock detection
  • Multi-tenant data isolation
  • Geographic data distribution
  • Horizontal scaling

Relationships

References

  • src/ROADMAP.md
  • src/storage/FUTURE_ENHANCEMENTS.md#blobredundancymanager-implement-rocksdb-event-listener
  • Source key: roadmap:259:storage:v1.8.0:blobredundancymanager-implement-rocksdb-event-listener

Generated from the consolidated source roadmap. Keep the roadmap and issue in sync when scope changes.

</issue_description>

Comments on the Issue (you are @copilot in this section)


📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.

…deletion events

- Fix createRocksDBListener() to return a real RocksDBBlobListener instead of an error
- Add BlobRedundancyManager::notifySSTFileDeleted() to mark locations unhealthy and queue repair
- Implement RocksDBBlobListener::OnTableFileDeleted() delegating to notifySSTFileDeleted()
- Add RocksDBWrapper::addEventListener() to register listeners before DB open
- Add 5 BlobRedundancyManagerTest cases covering listener creation and SST deletion handling
- Add test_blob_redundancy_event_listener_focused CMake target
- Add blob-redundancy-event-listener-ci.yml CI workflow

Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement RocksDB event listener in BlobRedundancyManager feat(storage): BlobRedundancyManager — implement RocksDB EventListener for SST deletion Mar 13, 2026
Copilot AI requested a review from makr-code March 13, 2026 20:14
@makr-code makr-code marked this pull request as ready for review March 14, 2026 06:14
@makr-code makr-code requested a review from Copilot March 14, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BlobRedundancyManager: Implement RocksDB Event Listener

2 participants