Skip to content

Replace python-rocksdb with Rocksdict#967

Merged
qzhodl merged 2 commits intomasterfrom
rocksdict
Mar 24, 2026
Merged

Replace python-rocksdb with Rocksdict#967
qzhodl merged 2 commits intomasterfrom
rocksdict

Conversation

@qzhodl
Copy link
Copy Markdown
Contributor

@qzhodl qzhodl commented Mar 5, 2026

Problem: python-rocksdb blocks upgrading to Python 3.13

python-rocksdb is a Cython wrapper around the RocksDB C++ library. It is effectively broken on upgrading to Python 3.13 for two reasons:

  • Removed C API call (PyEval_InitThreads)
    python-rocksdb calls PyEval_InitThreads() from its .pyx source. This API was deprecated in Python 3.9, became a no-op, and was removed in Python 3.12.
    Fix: delete the call (trivial).
  • Cython 3.x migration is non-trivial
    Python 3.13 support requires Cython 3.x, but python-rocksdb was written with Cython 0.29.x semantics, and those assumptions break under Cython 3.x. Migrating would require non-trivial .pyx refactors.

Is migrating python-rocksdb to Cython 3.x viable?

A serious attempt already exists: faust-streaming/python-rocksdb migrated the codebase to Cython 3.x and removed the PyEval_InitThreads() call.

However, it has two major issues:

  • Still not caught up to Python 3.13
    Even after migrating to Cython 3.x, the fork only reaches Python 3.12. Python 3.13 introduced additional C API changes beyond 3.12, and the fork never added/validated 3.13 support.
  • Abandoned + recommends moving away
    The fork is effectively unmaintained and explicitly points users to rocksdict as the recommended replacement.

Why rocksdict

  • Rust + PyO3 instead of Cython
    PyO3 abstracts CPython internals, so Python version churn is largely handled by upgrading the PyO3 crate rather than patching our wrapper code. This is the main reason rocksdict already supports Python 3.7–3.14.
  • Prebuilt wheels across platforms (Linux/macOS/Windows)
    No system RocksDB dependency, no Cython, no local compilation step — significantly lowers install friction and CI variability.
  • Actively maintained
    The project is actively maintained and has comparable stars to python-rocksdb.

Validation/tests completed

I’ve validated the change with the following tests:

  • Mining test: ran a mining node in my local devnet.
  • Sync test: ran a full node with this change and synced to the latest devnet block.
  • Compatibility test (rolling upgrade):
    • synced to latest devnet block on the previous version
    • switched to the version with this change
    • continued syncing successfully

Next step

I’m planning to run a mainnet full node with this change and validate end-to-end sync behavior under real network conditions.

@qzhodl qzhodl requested review from ping-ke, qizhou and syntrust March 5, 2026 04:22
@syntrust
Copy link
Copy Markdown

syntrust commented Mar 5, 2026

How do we upgrade an older quarkchain node to this version? Will the data be impacted?

@qzhodl
Copy link
Copy Markdown
Contributor Author

qzhodl commented Mar 6, 2026

How do we upgrade an older quarkchain node to this version? Will the data be impacted?

No. Full nodes running older versions only need to upgrade the Docker image. Existing data will remain intact.

Copy link
Copy Markdown
Contributor

@qizhou qizhou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qzhodl qzhodl merged commit ba790a9 into master Mar 24, 2026
4 checks passed
@qzhodl qzhodl deleted the rocksdict branch March 24, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants