Skip to content

HPC-friendly Python build #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Grufoony wants to merge 3 commits into from
Closed

HPC-friendly Python build #429

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b34040c
HPC-friendly Python build
Grufoony Mar 23, 2026
690c1b8
Version 5.3.2
Grufoony Mar 23, 2026
fd92882
Fix
Grufoony Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 51 additions & 1 deletion .github/workflows/pypi.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,56 @@ jobs:
name: wheel-linux-${{ matrix.python-version }}
path: wheelhouse/*.whl

build-wheels-linux-hpc:
name: Build wheel on Ubuntu HPC profile (Python 3.12)
needs: [check-version]
runs-on: ubuntu-latest
if: needs.check-version.outputs.should_build == 'true'

steps:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: '3.12'

- name: Install system dependencies
run: |
sudo apt update
sudo apt install -y cmake build-essential doxygen libtbb-dev

- name: Install build dependencies
run: |
python -m pip install --upgrade pip
python -m pip install build wheel setuptools pybind11-stubgen

- name: Build Ubuntu HPC wheel (-O3 only)
env:
CMAKE_ARGS: "-DDSF_HPC_RELEASE=ON -DDSF_OPTIMIZE_ARCH=OFF"
DSF_HPC_BUILD: "ON"
DSF_PACKAGE_VERSION: ${{ needs.check-version.outputs.publish_version }}
run: |
rm -rf dist wheelhouse
python -m build --wheel

Comment on lines +158 to +166
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This job sets CMAKE_ARGS in the environment, but setup.py doesn’t currently read/append CMAKE_ARGS at all, so these flags won’t affect the CMake configure step. Either wire CMAKE_ARGS through in setup.py (preferred, since other jobs also set it) or remove CMAKE_ARGS here and rely solely on the DSF_HPC_BUILD toggle.

Copilot uses AI. Check for mistakes.
- name: Prepare wheel artifact
run: |
mkdir -p wheelhouse
cp dist/*.whl wheelhouse/

- name: Test wheel installation
run: |
python -m pip install wheelhouse/*.whl
python -c "import dsf; print('DSF Ubuntu HPC wheel installation successful')"

- name: Upload Ubuntu HPC wheel as artifact
uses: actions/upload-artifact@v4
with:
name: wheel-linux-hpc-3.12
path: wheelhouse/*.whl

build-wheels-macos:
name: Build wheels on macOS (Python ${{ matrix.python-version }})
needs: [check-version]
Expand Down Expand Up @@ -263,7 +313,7 @@ jobs:

publish:
name: Publish to PyPI
needs: [check-version, build-wheels-linux, build-wheels-macos, build-wheels-windows, build-sdist]
needs: [check-version, build-wheels-linux, build-wheels-linux-hpc, build-wheels-macos, build-wheels-windows, build-sdist]
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new HPC wheel job produces an unrepaired Ubuntu-built wheel (no auditwheel repair) and the publish job now depends on it and uploads all collected *.whl to PyPI/TestPyPI. This risks publishing a non-manylinux, distro-specific wheel with unexpected glibc/system-library requirements, and it may be ignored or cause install/runtime issues for users. Consider either (a) running auditwheel repair for the HPC wheel as well, or (b) keeping the HPC wheel as a CI artifact only and excluding it from the PyPI upload set (and from publish.needs).

Copilot uses AI. Check for mistakes.
runs-on: ubuntu-latest
if: needs.check-version.outputs.should_build == 'true'

Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ authors:
repository-code: 'https://github.com/physycom/DynamicalSystemFramework'
url: 'https://physycom.github.io/DynamicalSystemFramework/'
license: CC-BY-NC-SA-4.0
version: 5.3.1
version: 5.3.2
date-released: '2026-03-06'
25 changes: 19 additions & 6 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ option(DSF_BENCHMARKS "Build DSF benchmarks" OFF)
option(DSF_BUILD_PIC "Build DSF with position-independent code" OFF)
option(BUILD_PYTHON_BINDINGS "Build Python bindings" OFF)
option(DSF_OPTIMIZE_ARCH "Optimize for native architecture" ON)
option(DSF_HPC_RELEASE "Build release flags suitable for HPC infrastructure" OFF)

# If CMAKE_BUILD_TYPE not set, default to Debug
if(NOT CMAKE_BUILD_TYPE)
Expand All @@ -47,21 +48,33 @@ set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)

if(DSF_HPC_RELEASE)
set(DSF_OPTIMIZE_ARCH OFF)
Copy link

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set(DSF_OPTIMIZE_ARCH OFF) when DSF_HPC_RELEASE is enabled will shadow the cache variable created by option(DSF_OPTIMIZE_ARCH ...), which can leave tooling (ccmake/GUI) showing DSF_OPTIMIZE_ARCH=ON while the build actually behaves as OFF. To avoid this confusing split-brain state, set the cache value explicitly (e.g., set(DSF_OPTIMIZE_ARCH OFF CACHE BOOL "..." FORCE)) or otherwise ensure the cache and effective value stay in sync.

Suggested change
set(DSF_OPTIMIZE_ARCH OFF)
set(DSF_OPTIMIZE_ARCH
OFF
CACHE BOOL "Optimize for native architecture" FORCE)

Copilot uses AI. Check for mistakes.
endif()

# Ensure optimization flags are applied only in Release mode
if(CMAKE_BUILD_TYPE MATCHES "Release")
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Ofast -flto=auto")
if(DSF_OPTIMIZE_ARCH)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
if(DSF_HPC_RELEASE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O3")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Ofast -flto=auto")
if(DSF_OPTIMIZE_ARCH)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
endif()
endif()
elseif(CMAKE_CXX_COMPILER_ID MATCHES "MSVC")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /O2")
endif()
elseif(CMAKE_BUILD_TYPE MATCHES "Profile")
if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Ofast -flto=auto -pg")
if(DSF_OPTIMIZE_ARCH)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
if(DSF_HPC_RELEASE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -O3 -pg")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Ofast -flto=auto -pg")
if(DSF_OPTIMIZE_ARCH)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
endif()
endif()
endif()
elseif(CMAKE_BUILD_TYPE MATCHES "Coverage")
Expand Down
26 changes: 26 additions & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from pathlib import Path
import platform
import re
import shlex
import shutil
import subprocess
import sys
Expand Down Expand Up @@ -78,6 +79,26 @@
"-DBUILD_PYTHON_BINDINGS=ON",
]

# Forward generic CMake args from environment (e.g. CI overrides).
env_cmake_args = os.environ.get("CMAKE_ARGS", "").strip()
if env_cmake_args:
cmake_args.extend(shlex.split(env_cmake_args))

# Optional HPC release profile for source builds/wheels.
if os.environ.get("DSF_HPC_BUILD", "").lower() in {
"1",

Check warning

Code scanning / Pylint (reported by Codacy)

Wrong hanging indentation before block (add 4 spaces). Warning

Wrong hanging indentation before block (add 4 spaces).
"on",

Check warning

Code scanning / Pylint (reported by Codacy)

Wrong hanging indentation before block (add 4 spaces). Warning

Wrong hanging indentation before block (add 4 spaces).
"true",

Check warning

Code scanning / Pylint (reported by Codacy)

Wrong hanging indentation before block (add 4 spaces). Warning

Wrong hanging indentation before block (add 4 spaces).
"yes",

Check warning

Code scanning / Pylint (reported by Codacy)

Wrong hanging indentation before block (add 4 spaces). Warning

Wrong hanging indentation before block (add 4 spaces).
}:
cmake_args.extend(
[
"-DDSF_HPC_RELEASE=ON",
"-DDSF_OPTIMIZE_ARCH=OFF",
"-DCMAKE_POSITION_INDEPENDENT_CODE=ON",
]
)

if platform.system() == "Windows":
cmake_args += [f"-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{cfg.upper()}={extdir}"]
if "CMAKE_TOOLCHAIN_FILE" in os.environ:
Expand Down Expand Up @@ -553,4 +574,9 @@
"shapely",
"folium",
],
extras_require={
# Reserved marker extra for HPC-oriented deployments.
# Build-time behavior is driven through DSF_HPC_BUILD env/CMAKE args.
"hpc": [],
},
)
2 changes: 1 addition & 1 deletion src/dsf/dsf.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

static constexpr uint8_t DSF_VERSION_MAJOR = 5;
static constexpr uint8_t DSF_VERSION_MINOR = 3;
static constexpr uint8_t DSF_VERSION_PATCH = 1;
static constexpr uint8_t DSF_VERSION_PATCH = 2;

static auto const DSF_VERSION =
std::format("{}.{}.{}", DSF_VERSION_MAJOR, DSF_VERSION_MINOR, DSF_VERSION_PATCH);
Expand Down
Loading