Skip to content

Porting PyFDB to Pybind 11#193

Merged
danovaro merged 104 commits intodevelopfrom
feature/pyfdb-integration
Feb 26, 2026
Merged

Porting PyFDB to Pybind 11#193
danovaro merged 104 commits intodevelopfrom
feature/pyfdb-integration

Conversation

@tbkr
Copy link
Copy Markdown
Contributor

@tbkr tbkr commented Nov 7, 2025

Description

I'm aware that this is a somewhat mid-sized PR 😉 It's also a complete rewrite with tests and documentation.

Porting FDB to PyBind11, making the Python-API more user-friendly and introducing tests for the API layer which are testing core functionality of the FDB in an ephemeral FDB setup.

Documentation is added to the code base as well as the sphinx generated side.

The coverage was tested and is close to 100% testing all provided use-cases and functionalities.

For now I keep this PR in WIP status and I'm open to discuss Implementation details.

Notes for reviewers:

  • Think about the current API design and raise issues, esp. if you are aware of use-cases which aren't currently implemented
  • Read the documentation carefully and check whether the written functionalities are correctly mirrored from the FDB.
  • Feel free to raise potential issues I'm not aware of at this point in time.

Contributor Declaration

By opening this pull request, I affirm the following:

  • All authors agree to the Contributor License Agreement.
  • The code follows the project's coding standards.
  • I have performed self-review and added comments where needed.
  • I have added or updated tests to verify that my changes are effective and functional.
  • I have run all existing tests and confirmed they pass.

🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/dev-section/fdb/pull-requests/PR-193

@tbkr tbkr changed the title Feature/pyfdb_integration Porting PyFDB to Pybind 11 WIP: Feature/pyfdb_integration Porting PyFDB to Pybind 11 Nov 7, 2025
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Nov 7, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.18%. Comparing base (7129b45) to head (a661fcb).

Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #193   +/-   ##
========================================
  Coverage    71.18%   71.18%           
========================================
  Files          376      376           
  Lines        23709    23709           
  Branches      2476     2476           
========================================
  Hits         16877    16877           
  Misses        6832     6832           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tbkr tbkr force-pushed the feature/pyfdb-integration branch 28 times, most recently from 91dfeff to 2be9698 Compare November 14, 2025 10:24
tbkr added 23 commits February 25, 2026 16:03
Works now with shutil and there is the option to zero-copy read into a
buffer.
The individual objects now return proper python objects. It's possible
to compare ControlElements with StatusElements.

Adjusted the tests accordingly.
Now checking the actual content of what is moved and also checks whether
the cleanup was successful.
Now comparable with a Dict[str, Collection[str]] and constructable, as
well.
To streamline with the other pyfdb API, I made the identifier also
accept python dicts.
Also, the internal representation is now a list of tuples, as for the
identifier there is only a list of key and singular value pairs.
I decoupled the external and internal representation so that this can be
changed quickly.
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR ports PyFDB from SWIG to PyBind11, creating a more Pythonic interface with comprehensive test coverage and documentation. The changes introduce a complete rewrite of the Python API with new test infrastructure and documentation generation setup.

Changes:

  • Replaces SWIG-based bindings with PyBind11 implementation in src/pyfdb_bindings/bindings.cc
  • Adds comprehensive Python API layer with type hints and helper classes in src/pyfdb/
  • Introduces extensive test suite covering all API functionality in tests/pyfdb/integration/
  • Adds Sphinx-based documentation with examples and API references in docs/pyfdb/
  • Updates build system to support PyFDB alongside existing Z3FDB

Reviewed changes

Copilot reviewed 66 out of 84 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/pyfdb_bindings/bindings.cc New PyBind11 C++ bindings for FDB core functionality
src/pyfdb/pyfdb.py Main FDB Python class with all public API methods
src/pyfdb/pyfdb_type.py Type definitions and helper classes (DataHandle, URI, etc.)
src/pyfdb/pyfdb_iterator.py Iterator wrapper classes for various FDB operations
tests/pyfdb/integration/*.py Comprehensive integration tests for all API functions
tests/pyfdb/conftest.py Test fixtures and FDB setup helpers
docs/pyfdb/*.rst Documentation pages for installation, examples, and API
CMakeLists.txt Build configuration for PyFDB feature flag
src/CMakeLists.txt PyFDB package staging and wheel build setup
tests/CMakeLists.txt Test registration for PyFDB
.github/workflows/pyfdb.yml CI workflow for PyFDB builds and tests
Comments suppressed due to low confidence (3)

tests/pyfdb/integration/test_uri.py:1

  • The assertion is comparing a string with a URI object, which will always be True. Based on the test name and context, this should compare test_uri with test_uri_section3 instead.
    src/z3fdb/simple_store_builder.py:1
  • Corrected spelling of 'spefified' to 'specified'.
    src/pyfdb/pyfdb_type.py:1
  • Corrected spelling of 'singluar' to 'singular'.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

@simondsmart simondsmart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really good, except for the execute() method still persisting in the move() api.

Comment thread docs/pyfdb/examples.rst Outdated
Comment thread docs/pyfdb/examples.rst Outdated
Comment thread docs/pyfdb/examples.rst Outdated
Comment thread tests/pyfdb/integration/test_control.py
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 64 out of 82 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Also modified wrong python executable env var in CMake
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants